Role: Data Engineer
Location: Remote (Candidate should be comfortable following PST Time Zone)
Job Type: Full Time
Active LinkedIn ID is a must and must match with the resume.
Must Have Skills
- Azure Databricks, Apache Spark, Snowflake, Data Governance, Unity Catalog, dBT, Microsoft Purview, Kafka, SQL, Python
Experience And Qualifications
- Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.
- 10+ years of experience in data engineering / data architecture, with deep hands‑on expertise in Microsoft Azure.
- Expert‑level experience with Azure Databricks and Apache Spark (PySpark, Spark SQL).
- Delta Lake, Unity Catalog, Job orchestration and performance tuning.
- Strong experience with Snowflake on Azure, including:
- Schema and data model design
- Performance and cost optimization
- RBAC, Streams, and Tasks
- DBT (Core or Cloud) expertise:
- Project structure and best practices
- Tests, exposures, macros
- Strong fundamentals in SQL and Python.
- Proven experience building ETL/ELT pipelines using Azure Data Factory (ADF) and Azure Synapse / Microsoft Fabric pipelines.
- Familiarity with streaming platforms such as Event Hubs and/or Kafka.
- Solid understanding of Lakehouse and data warehousing architectures, dimensional modeling, medallion architecture patterns, and data quality frameworks.
- Strong knowledge of security and governance, including Microsoft Purview (catalog, lineage), data masking and PII handling, managed identities, and private networking.
- Excellent communication and documentation skills, with the ability to create architecture and design documents and present to both technical and business stakeholders.
- Consulting / agency experience, with the ability to lead multiple concurrent projects.
- Willingness to travel as required.
Nice To Have
- Experience with Azure ML, feature stores, or serving ML pipelines from Databricks or Snowflake.
- Infrastructure as Code (IaC) using Terraform or Bicep.
- Containerization experience with Docker.
- Observability and data quality tooling, such as Great Expectations / Delta Expectations, Databricks Quality Flows, Monte Carlo, Datadog.
Preferred Certifications
- Databricks Certified Data Engineer Professional or Databricks Architect.
- Microsoft Azure: DP-203 – Azure Data Engineer, Azure Solutions Architect Expert, Azure Security Engineer Associate.
- Snowflake: SnowPro Core, SnowPro Advanced.
Job Description / Responsibilities
- Partner with client stakeholders to translate business objectives into scalable Azure data architectures and delivery roadmaps.
- Architect and implement lakehouse and data warehouse solutions using Azure Databricks (Delta Lake, Unity Catalog), Databricks Bundle, Snowflake on Azure, dbt (Core / Cloud), ADLS Gen2, and medallion architecture patterns.
- Design and build robust ingestion and transformation pipelines using Azure Data Factory (ADF) and Microsoft Fabric / Synapse pipelines.
- Orchestrate ELT workflows using Databricks Jobs, Delta Live Tables, and dbt models.
- Establish best practices for performance and cost optimization, including cluster sizing and autoscaling, Photon and SQL Warehouses, Snowflake virtual warehouses, caching, file layout optimization, and Z‑ordering.
- Drive observability and monitoring using Azure Monitor, Log Analytics, and Databricks metrics.
- Implement robust data governance and security using Azure Authentication, Unity Catalog, RBAC / ABAC, managed identities, private endpoints and VNet injection, and Azure Key Vault–backed secrets.
- Lead code and design reviews, setting standards for PySpark, SQL, and dbt, unit and integration testing, and data quality checks (expectations and constraints).
- Implement CI/CD pipelines using Azure DevOps or GitHub Actions.
- Guide multi‑domain programs (Healthcare, Retail, BFSI, or similar).
- Mentor data engineers and ensure high‑quality, production‑ready deliverables.
- Evangelize solutions through clear documentation, reference architectures, and executive and stakeholder presentations.
#J-18808-Ljbffr