Job Title: Lead Data Engineer/Architect with Palantir Foundry
Location: Dallas, TX/Seattle, WA/Atlanta, GA/NY, NJ
FULLTIME
We are seeking a highly experienced Lead Data Engineer to architect, develop, and optimize scalable data solutions using Palantir Foundry. This senior role is ideal for a data engineering expert who thrives in complex, ontology-aware environments and can lead the design of robust data pipelines and applications that drive actionable insights. You will play a pivotal role in shaping our data strategy, collaborating with stakeholders across the organization, and ensuring the integrity, performance, and cost-efficiency of our data ecosystem. While familiarity with Generative AI (GenAI) is a plus, your core expertise should lie in data architecture, pipeline development, and Foundry platform mastery.
Key Responsibilities
Data Architecture & Pipeline Engineering
- Design, build, and maintain end-to-end ETL/ELT pipelines using PySpark, Spark SQL, and Python.
- Architect scalable data frameworks that support high-performance analytics and operational workflows.
- Conduct data cleansing, transformation, and validation to ensure data quality and consistency.
Palantir Foundry Platform Expertise
- Configure and manage data connections, Pipeline Builder, and Contour for data exploration and visualization.
- Develop and maintain ontology objects, ensuring semantic consistency and reusability across applications.
- Manage code repositories and version control within Foundry, promoting modular and maintainable engineering practices.
Ontology-Aware Application Development
- Collaborate with data owners and domain experts to ingest, transform, and model data into ontology-driven applications such as Workshop and Quiver.
- Maintain clarity on the end-to-end architecture and data flow, ensuring seamless integration from ingestion to ontology.
Cross-Functional Collaboration & Leadership
- Partner with data scientists, analysts, and business stakeholders to translate requirements into scalable data solutions.
- Engage with senior leadership to maintain a feedback loop for Foundry program improvements.
- Provide mentorship and technical guidance to junior engineers and analysts.
Operational Excellence & Cost Management
- Establish and maintain cost monitoring and reporting for Foundry-related operations.
- Lead efforts in troubleshooting, performance tuning, and platform stability.
Continuous Improvement & Innovation
- Stay current with Foundry platform updates, emerging features, and best practices.
- Identify opportunities for automation, optimization, and innovation in data workflows.
- Explore and evaluate GenAI capabilities for potential integration into data processes.