Strategic Data Leader - Databricks

Strategic Data Leader Databricks | Large-Scale Data Platform Leadership

Remote Role

Duration: 12+ months

Job Summary:

Client is seeking a highly experienced Strategic Data Leader with deep expertise in Databricks, enterprise-scale data architecture, and large-volume data transformation. This role requires a hands-on leader who can drive data strategy, modernize data platforms, and lead large-scale data engineering initiatives across complex business environments. The ideal candidate should have strong experience handling large data volumes (Large Data Hunger), designing scalable cloud-native architectures, and leading cross-functional teams to deliver high-performance analytics and AI-ready data ecosystems.

Key Responsibilities:

Strategic Leadership

• Define and drive enterprise-wide data strategy, architecture roadmap, and modernization initiatives.
• Lead large-scale data platform transformation programs using Databricks, Snowflake, Azure/AWS/GCP cloud ecosystems.
• Partner with business leaders, analytics teams, and executive stakeholders to align data strategy with business goals.
• Build and scale data governance, data quality, metadata management, and data security frameworks.

Databricks Platform Ownership • Architect and manage enterprise-wide Databricks Lakehouse Platform.

• Lead implementation of Delta Lake, Unity Catalog, ML pipelines, and real-time data processing frameworks.
• Optimize Databricks workloads for performance, scalability, and cost efficiency.
• Establish best practices for cluster management, notebook standards, CI/CD, and deployment automation.

Large Data Engineering & Architecture • Design and implement high-volume batch and streaming data pipelines using Spark, PySpark, SQL, and Databricks workflows.

• Handle petabyte-scale structured and unstructured data from multiple enterprise sources.
• Build robust ETL/ELT frameworks supporting advanced analytics, AI/ML, and BI reporting.
• Drive migration from legacy platforms to modern lakehouse architecture.

Team Leadership • Lead and mentor data engineers, architects, analysts, and platform teams.

• Provide architectural guidance and hands-on technical leadership.
• Establish engineering standards, delivery governance, and operational excellence.
• Drive agile delivery across multiple strategic programs.

Required Skills & Qualifications:

Technical Expertise • 12+ years of experience in Data Engineering, Data Architecture, and Enterprise Data Strategy

• 5+ years of strong hands-on experience with Databricks
• Expert knowledge of: • Apache Spark / PySpark
• Delta Lake
• SQL / Advanced SQL Optimization
• Data Warehousing concepts
• Data Modeling (Dimensional + Lakehouse)
• ETL / ELT Frameworks
• Cloud platforms: Azure / AWS / GCP
• CI/CD for data platforms
• Data Governance & Security

Leadership Experience • Proven experience leading enterprise-wide data transformation initiatives

• Experience managing large distributed technical teams
• Strong executive stakeholder management skills
• Ability to translate business strategy into technical execution

Preferred Qualifications • Experience with Snowflake, Kafka, Airflow, dbt, Synapse, Fabric

• Exposure to AI/ML platform integration
• Experience with Data Mesh / Data Fabric architecture
• Databricks Certifications preferred
• Strong consulting or client-facing leadership experience