Job Title: Data Engineer / Databricks Developer
Location: New York (Onsite – 3 days/week)
Employment Type: Full-time
Note: Only local candidates will be considered. In-person interview is mandatory.
Job Overview
We are seeking an experienced Data Engineer / Databricks Developer with a strong background in the Investment Banking domain to join our team in New York. The ideal candidate will be responsible for designing, developing, and optimizing scalable data solutions using Azure-based technologies, with a focus on Databricks and modern data architecture.
Key Responsibilities
- Design, develop, and implement scalable data pipelines using Azure Databricks and Azure Data Factory.
- Build and maintain robust ETL/ELT workflows using Apache Spark and PySpark DataFrames.
- Develop high-performance data transformation processes using Spark and SQL.
- Optimize data pipelines for efficient ingestion, processing, and transformation of large datasets.
- Implement and maintain data governance frameworks, including data access, security, compliance, and lifecycle management.
- Ensure data quality, consistency, and lineage across enterprise data platforms.
- Design and support modern Lakehouse architecture and distributed data processing systems.
- Apply Master Data Management (MDM) principles to maintain data integrity and standardization.
- Collaborate with cross-functional teams to deliver scalable, reliable, enterprise-grade data solutions.
Required Skills & Qualifications
- Strong hands-on experience with Azure Databricks
- Proficiency in Apache Spark / PySpark
- Experience with Azure Data Factory
- Advanced knowledge of SQL
- Expertise in building data pipelines and ETL/ELT processes
- Experience with data governance, data lineage, and data quality frameworks
- Strong understanding of Lakehouse architecture
- Experience working in Investment Banking or Financial Services domain (Mandatory)
Preferred Qualifications
- Experience with large-scale distributed data systems
- Familiarity with cloud-based data platforms and enterprise data architecture
- Strong problem-solving and performance optimization skills