Title: Big Data Engineer - Intermediate
Location: Manhattan West, NY - Onsite
Duration: Right to hire - W2 (USC)
Job Summary
We are seeking a skilled Data Engineer to design, build, and manage scalable ETL pipelines supporting a centralized data lake and Snowflake data warehouse. The role focuses on automating data ingestion, transformation, and aggregation workflows to enable reliable analytics and data-driven decision-making. Key Responsibilities
- Design, develop, and maintain robust ETL pipelines for ingesting data into the enterprise data lake and Snowflake environment.
- Automate data processing, aggregation, and analytical workflows to improve data availability and performance.
- Implement and manage orchestration and scheduling of data pipelines using Control$B!>(BM and Apache Airflow.
- Develop scalable data transformation logic using PySpark and Apache Spark (Java).
- Work with large, structured and semi-structured datasets on AWS infrastructure.
- Ensure data quality, integrity, and reliability across data pipelines.
- Optimize data pipelines for performance, cost, and scalability.
- Collaborate with analytics, data science, and business teams to understand data requirements.
- Monitor, troubleshoot, and resolve pipeline failures and performance bottlenecks.
- Follow best practices for data engineering, security, and documentation.
Required Skills & Qualifications
- Strong experience with data lake architectures and large-scale data processing.
- Hands-on experience with AWS services (e.g., S3, EC2, EMR, Glue, or related).
- Proven expertise in building ETL pipelines for analytics and reporting use cases.
- Solid working knowledge of Snowflake, including data loading, transformations, and performance optimization.
- Experience with workflow automation and scheduling tools such as Control$B!>(BM and Apache Airflow.
- Proficiency in PySpark for distributed data processing.
- Strong programming experience with Apache Spark using Java.
- Good understanding of data modeling, partitioning, and performance tuning concepts.