Senior Data Engineer (Databricks)
Client Corporation is seeking a Senior Data Engineer who enjoys collaborating
across the organization to deliver reliable, scalable data solutions. Join a skilled team
focused on modern data platforms, where you will help shape and operate our Databricks-
based on Lakehouse and streaming analytics capabilities.
This role blends strong business and analytical judgment with hands-on engineering:
you will partner with stakeholders to understand problems, translate them into clear
requirements, design pragmatic architectures, and deliver production-grade ingestion
and transformation pipelines - with ownership through deployment and steady-state
operations.
You will work with teams across the company to clarify data and BI needs, perform
structured requirements discovery alongside Data Modelers, align with BI
development on scope and estimates, and help guide delivery through completion while
promoting engineering quality and consistency.
Responsibilities
- Define data requirements; discover, integrate, and wrangle large volumes of structured and semi/unstructured data; validate outcomes using appropriate tooling in our data environment.
- Support standardized datasets and ad hoc analysis needs; build mechanisms to ingest, validate, normalize, cleanse, and enrich data for downstream consumption.
- Contribute to data policies and technical controls where needed (access patterns, retention considerations, and synthesizing/anonymizing sensitive attributes in line with enterprise standards).
- Implement rigorous data quality approaches for new and evolving sources; iterate with analytics partners to improve sourcing, preparation, and trust in datasets used for insights and modeling.
- Champion data engineering best practices (testing, observability, operational readiness) and contribute practical guidance on analytics preparation and visualization-friendly dataset design where applicable.
- Partner closely with data science and business intelligence teams to deliver data models and pipelines that support reporting, research, and machine learning workflows.
- Build pipelines that clean, transform, aggregate, and publish data from disparate systems into curated layers suitable for BI and advanced analytics.
- Use Databricks, SQL, Spark, scripting, and AWS services to integrate systems reliably and efficiently.
- Apply solid knowledge of data architecture principles and help lead initiatives from requirements through implementation—balancing correctness, performance, cost, and maintainability.
Qualifications Required
- 5+ years of professional experience in data engineering (or equivalent depth), including manipulating, processing, and extracting value from large datasets in production settings.
- Databricks (must have): hands-on delivery on the Databricks platform, including developing and operating Databricks Jobs/Workflows, notebooks, and production pipelines using Apache Spark (Spark SQL and/or PySpark) on Databricks.
- Delta Lake on Databricks (must have): building and maintaining Delta Lake tables (for example: incremental processing, merges/upserts, schema evolution, and performance patterns aligned with Delta best practices).
- Delta Live Tables / DLT (must have): designing, implementing, and operating DLT pipelines (Python or SQL as used in your standards), including expectations around pipeline dependencies, incremental processing, and operational monitoring as supported by DLT.
- Databricks governance (must have): practical experience with Unity Catalog (or equivalent Databricks governance patterns strongly aligned to UC), including secured sharing concepts and catalog/table governance as implemented in your environment.
- 4+ years hands-on Apache Spark engineering for analytics workloads (batch and/or streaming), with ability to implement reliable transformations and troubleshoot performance issues.
- 3+ years working with Apache Kafka (or managed equivalents such as Confluent Kafka) for high-volume event/stream processing, including operational considerations (throughput, lag, scaling, replay scenarios).
- 3+ years building and operating AWS capabilities supporting analytics platforms (for example: S3, IAM-aligned access patterns, integration with compute/storage patterns common to lakehouse architectures; familiarity with services such as Glue/Lambda/MSK as applicable).
- Strong SQL: ability to write and optimize intermediate-to-advanced queries and translate business logic into reliable datasets.
- Demonstrated experience delivering ETL/ELT pipelines in a Databricks lakehouse environment; comfort with incremental loads, schema evolution, and data quality checks.
- Experience working in Agile delivery (Scrum/Kanban/SAFe or similar): backlog refinement, iterative delivery, dependency coordination.
Preferred (nice to have)
- Experience with Snowflake , or other large-scale analytical databases (helpful for hybrid/platform comparisons). NoSQL exposure (Cassan