Job Title: Enterprise Data Engineer (Databricks)
Location: Hybrid – Austin, TX
Duration: 5 Months
Pay Rate: $65/hr on 1099
Role Summary
The Enterprise Data Engineer will design, build, and operate scalable data pipelines within an Azure-based Databricks Lakehouse architecture. The primary focus is to deliver and maintain a software-driven data model for analytics and data consumption.
This is a hands-on, execution-focused role responsible for engineering reliable data ingestion from multiple data sources, performing transformations, implementing data quality checks, and delivering curated datasets integrated with ServiceNow (ITSM/ITSLM) and ApptioOne (ITFM).
The role involves close collaboration with data architects, platform teams, providers, and stakeholders to translate architectural designs into scalable, governed, and production-ready data solutions. The work follows Agile software engineering practices, including GitHub-based workflows and CI/CD-driven SDLC processes.
Key Responsibilities
- Design, build, and maintain data models supporting data consumption, integration, semantic analytics, reporting, and executive dashboards.
- Develop scalable data ingestion and transformation pipelines using Azure PaaS services, Databricks, Delta Lake, Python, and Spark SQL.
- Implement integrations for ServiceNow operational data (SLA, incidents, CMDB) and ApptioOne financial and cost allocation data.
- Develop and enforce data quality checks, validation rules, and monitoring mechanisms for end-to-end pipeline reliability.
- Apply Unity Catalog governance including data access control, lineage management, and schema enforcement as per architectural standards.
- Optimize Databricks Lakehouse performance including pipeline efficiency, storage layout, and query optimization.
- Support CI/CD pipelines and DevOps automation for data engineering workflows using Azure DevOps and GitHub Actions.
- Collaborate with architects, stakeholders, Capgemini teams, and service providers to deliver reporting and analytics solutions.
- Troubleshoot production data issues and ensure operational stability of analytics and reporting systems.
- Maintain documentation, runbooks, and operational standards for Databricks data pipelines.
Required Skills & Experience
- 5+ years of experience in Data Engineering or Analytics Engineering roles.
- Hands-on experience with Databricks, Delta Lake, and Spark-based data pipelines.
- Strong understanding of Medallion Architecture, especially Gold/Platinum layer implementation.
- Proficiency in Python, SQL, and Spark (PySpark or Spark SQL).
- Experience integrating enterprise systems such as ServiceNow (SLA, incident, CMDB data).
- Experience working with financial or cost management platforms (e.g., ApptioOne or similar ITFM tools).
- Strong understanding of data modeling techniques and methodologies.
- Familiarity with Unity Catalog for data governance and access control.
- Experience with Power BI or similar BI tools consuming Lakehouse datasets.
- Experience with Azure data services (e.g., ADLS Gen2, orchestration tools, integration patterns), Azure DevOps, and GitHub-based CI/CD pipelines.
Preferred Qualifications
- Experience supporting public sector data initiatives.
- Familiarity with ITIL 4 / ITIL 5 frameworks and SLA-based reporting.
- Experience with financial systems, SLA analytics, operational KPIs, or cost transparency dashboards.
- Exposure to MLflow, Feature Store, or AI/ML pipelines (implementation support role, not architecture ownership).