Senior Staff Engineer, Big Data Infrastructure

Stripe
San Francsico, US
On-site

Who this role is best for

A natural match if you have experience with large-scale distributed data systems and leading cross-functional engineering teams.

Best fit for

  • Candidates with 12+ years of experience leading large-scale infrastructure migrations and distributed systems
    — “ideally 12+ years of professional experience in a software development role”
  • Individuals with a background in big data platforms like Spark, Airflow, and Iceberg
    — “Experience with big data infrastructure at scale, such as Spark, Iceberg, S3, Airflow, YARN, and Celeborn”
  • Candidates who have driven AI-led transformations in data systems and agentic workflows
    — “Experience driving AI-led architectural transformations and integrating agentic workflows into large-scale data systems”

Things to consider

  • The role demands leadership in complex, high-availability data systems with minimal downtime
    — “Experience leading projects on a cross-team and company-wide scale, including scaling and migrating systems with little to no downtime”
  • A strong focus on reducing operational overhead through platform-driven solutions
    — “Reduce operational burden and maintenance overhead (KTLO) from over 40% to under 10%”

How to stand out

  • Highlight experience with multi-region and multi-cloud data strategies in your resume and interviews
    — “Lead the implementation of a multi-region and multi-cloud data strategy”
  • Emphasize achievements in optimizing reliability and security of distributed systems
    — “Experience developing, maintaining, and debugging distributed systems while optimizing their reliability and security”
  • Showcase your ability to modernize legacy systems such as YARN to Kubernetes
    — “Drive the architecture and delivery of critical modernization bets, including YARN to Kubernetes migrations”
Pace · Fast PacedCollaboration · HighAutonomy · HighDecision Impact · Team

Derived from job-description analysis by Serendipath's career intelligence engine.

What success looks like

  • Scaled foundational data systems
  • Executed modernization bets
  • Reduced operational burden
Typical background
12+ years of professional experience in software developmentStrong engineering background in distributed infrastructure and platform services

Skills & requirements

Required

Distributed SystemsBig Data InfrastructurePlatform ServicesData OrchestrationBatch ComputeData Lake

Preferred

Ai-led Architectural TransformationsAgentic WorkflowsKubernetesDatabase Internals

Stack & domain

Distributed SystemsPlatform ServicesStorageDatabasesComputeKubernetesSparkIcebergHiveS3Azure Blob StorageAirflowYARNCelebornLeadershipCommunicationProblem-solvingBig DataInfrastructure

About the role

Original posting from Stripe

Who we are

About Stripe

Stripe is a financial infrastructure platform for businesses. Millions of companies - from the world’s largest enterprises to the most ambitious startups - use Stripe to accept payments, grow their revenue, and accelerate new business opportunities. Our mission is to increase the GDP of the internet, and we have a staggering amount of work ahead. That means you have an unprecedented opportunity to put the global economy within everyone’s reach while doing the most important work of your career.

About the team

In this role, you’d be joining the Big Data Infrastructure team. Our team builds and maintains the platforms that power data work across Stripe, managing three core layers:

Data Orchestration (Airflow)

Batch Compute (Spark, Celeborn)

Data Lake (Iceberg, Hive, S3, Azure Blob Storage)

We support over 2,000 engineers and 600+ internal teams, processing workloads that organically grow more than 30% year over year. Our infrastructure handles massive scale, managing over 800 PB of data, running 325,000 applications a day, and accounting for hundreds of millions of dollars in annual spend, directly powering critical pipelines like inbound payments, network reconciliation, and capital underwriting.

What you’ll do

As the technical leader for the Big Data Infrastructure organization, you will guide a 40+ person engineering team to scale our foundational data systems. You will partner closely with major internal customers like Data Foundations, Payins Engineering, and AI/ML Foundations to modernize our tech stack and execute Stripe's transition to the "Agentic Era" - building "invisible infrastructure" where AI agents operate data pipelines end-to-end. 

Responsibilities

Serve as the technical leader for the Big Data Infrastructure organization, overseeing distributed systems that manage 20% of Stripe's total AWS spend

Execute Stripe's transition to the "Agentic Era" by building invisible infrastructure that empowers AI agents to author, validate, ship, and operate data pipelines

Drive the architecture and delivery of critical modernization bets, including YARN to Kubernetes migrations, native vectorization (Gluten/Velox), and Airflow and Data Catalog modernization

Lead the implementation of a multi-region and multi-cloud data strategy for high availability, disaster recovery, and Azure Blob Storage compliance

Reduce operational burden and maintenance overhead (KTLO) from over 40% to under 10% by shifting fixes into the platform instead of relying on runbooks

Foster a culture of operational excellence and mentor engineers at various stages in their careers, up to Staff+

Who you are

We’re looking for someone who meets the minimum requirements to be considered for the role. If you meet these requirements, you are encouraged to apply. The preferred qualifications are a bonus, not a requirement.

Minimum requirements

BS or MS in Computer Science or a related field

This is a Senior Staff role, so ideally 12+ years of professional experience in a software development role

A strong engineering background in distributed infrastructure and/or platform services, including one or more of the following: storage, databases, or compute

Experience leading projects on a cross-team and company-wide scale, including scaling and migrating systems with little to no downtime

Experience developing, maintaining, and debugging distributed systems while optimizing their reliability and security

Experience driving a culture of operational excellence, with a deep understanding of metrics, alerts, and dashboards

Preferred qualifications

Experience driving AI-led architectural transformations and integrating agentic workflows into large-scale data systems

Experience in a development team at AWS, Azure, OCI, or Google

Experience with big data infrastructure at scale, such as Spark, Iceberg, S3, Airflow, YARN, and Celeborn

Experience with database internals and/or with Kubernetes

Source: Stripe careers

Similar roles

More roles at Stripe