Research Scientist / Engineer – Training Infrastructure

Luma

Redwood City +1 more

Hybrid

Who this role is best for

Aimed at senior-level engineers with deep experience in distributed PyTorch and foundation model training, who can work remotely or in-office in the US and have a track record of solving complex scaling problems.

Best fit for

Senior engineers with experience in FSDP and multi-node training for foundation models
— “If you haven't worked at the level of FSDP and multi-node training, this is the wrong depth.”
Candidates who have built or optimized training systems across 100+ GPU clusters
— “Experience managing training runs across 100+ GPUs.”
Individuals with a strong background in distributed systems and GPU cluster optimization
— “Deep understanding of GPU clusters, networking, and storage systems.”

Things to consider

This role demands advanced knowledge in PyTorch and distributed training techniques.
— “This is hard PyTorch, CUDA, and distributed-systems work”
The candidate must be comfortable with large-scale infrastructure and cluster management.
— “Design, implement, and optimize efficient distributed training systems for models across thousands of GPUs.”

How to stand out

Highlight your experience with FSDP, Tensor Parallel, and multi-node training in your resume.
— “Research and implement advanced parallelization (FSDP, Tensor Parallel, Pipeline Parallel, Expert Parallel).”
Showcase your work on training stability and convergence in large-scale systems.
— “Optimize training stability, convergence, and resource utilization across massive clusters.”
Demonstrate your ability to build tools for monitoring and debugging complex training runs.
— “Build monitoring, visualization, and debugging tools for large-scale training runs.”

Pace · Fast PacedCollaboration · MediumAutonomy · HighDecision Impact · Team

Derived from job-description analysis by Serendipath's career intelligence engine.

What success looks like

design and optimize distributed training systems
implement advanced parallelization
build monitoring and debugging tools

Typical background

distributed systemsPyTorchCUDA

Skills & requirements

Required

Distributed SystemsPyTorchCUDADistributed TrainingParallelismTraining StabilityResource Utilization

Preferred

Linux Systems AdministrationContainerizationOrchestrationCloud Infrastructure

About the role

Original posting from Luma via Ashby

You'll build the distributed systems that train Luma's large-scale multimodal models across thousands of GPUs, so researchers can focus on innovation on top of reliable, efficient, scalable infrastructure.
This is hard PyTorch, CUDA, and distributed-systems work — advanced parallelism, training stability, and utilization across massive clusters. It fits an engineer who's solved real problems training foundation models at scale. If you haven't worked at the level of FSDP and multi-node training, this is the wrong depth.
What You'll Own
Design, implement, and optimize efficient distributed training systems for models across thousands of GPUs.
Research and implement advanced parallelization (FSDP, Tensor Parallel, Pipeline Parallel, Expert Parallel).
Build monitoring, visualization, and debugging tools for large-scale training runs.
Optimize training stability, convergence, and resource utilization across massive clusters.
First 90 Days
One way the first 90 could unfold.
Days 1–30 — Immerse & Diagnose: Learn the current training stack and where stability and utilization hurt at scale.
Days 30–60 — Ship & Validate: Land a parallelization or stability improvement that measurably helps a real training run.
Days 60–90 — Scale & Systemize: Build the monitoring and tooling that keeps large runs reliable and efficient.
What You Bring
Extensive distributed PyTorch training and parallelisms in foundation-model training.
Deep understanding of GPU clusters, networking, and storage systems.
Familiarity with communication libraries (NCCL, MPI) and distributed-system optimization.
Nice to Have
Strong Linux systems administration and scripting.
Experience managing training runs across 100+ GPUs.
Experience with containerization, orchestration, and cloud infrastructure.
About Luma: Luma's mission is to build unified general intelligence that can generate, understand, and operate in the physical world. We believe multimodality is critical for intelligence — the next step beyond language models comes from vision. Luma is an equal opportunity employer.

Source: Luma careers (Ashby)