Operations Engineer, Fleet Reliability

Fal
Remote
Remote

Who this role is best for

Geared toward mid-level engineers comfortable with troubleshooting GPU hardware issues in a remote on-call environment.

Best fit for

  • Linux system administrators with GPU node troubleshooting experience
    — “Administered Linux Systems in the critical path before
  • Engineers who automate problems rather than repeatedly fixing them
    — “You'd rather automate a problem than fix it twice
  • Candidates unfazed by on-call duties and undefined troubleshooting scenarios
    — “On-call doesn't scare you

Things to consider

  • On-call responsibilities require immediate response to hardware failures
    — “You'll be on-call
  • Half the troubleshooting scenarios lack existing documentation
    — “The runbook doesn't exist yet for half of what you'll do

How to stand out

  • Showcase specific instances where you diagnosed GPU-related hardware failures
    — “Troubleshooted GPU node issues: NVLink, NCCL, IB, driver and firmware bugs
  • Demonstrate automation projects that reduced repetitive operational tasks
    — “Scripted your way out of repetitive work
  • Highlight experience with infrastructure monitoring systems beyond basic alerts
    — “Has experience in observability systems like Grafana and Prometheus
Pace · Fast PacedCollaboration · MediumAutonomy · MediumDecision Impact · TeamLevel · Mid

Derived from job-description analysis by Serendipath's career intelligence engine.

What success looks like

  • Provision, validate, and triage GPU nodes
  • Troubleshoot hardware and software issues
  • Monitor fleet health and take remediation action
Typical background
Linux system administrationGPU node troubleshootingObservability systems experience

Skills & requirements

Required

Linux AdministrationGPU TroubleshootingObservability SystemsScripting

Preferred

Bash ScriptingPythonGo

Stack & domain

Linux Systems AdministrationGpu TroubleshootingObservability Systems (grafana, Prometheus)Scripting (bash, Python, Go)CuriosityComfort With AmbiguityOn-call ExperienceAutomationInfrastructureToolsModel AccessGenerative MediaAi Products

About the role

Original posting from Fal via Greenhouse

fal is the generative media ecosystem powering the next generation of AI products. We build the infrastructure, tools, and model access that teams need to move from idea to production, and do it at scale without compromise. For developers and enterprises, fal is the foundation that makes generative media not just possible, but practical: a unified platform where high-performance inference, orchestration, and observability come together to unlock new categories of AI-native products.

As generative media reshapes industries across a market projected to grow by hundreds of billions over the next decade, fal is becoming the ecosystem that ambitious teams build on.

About the role

As we bring up owned clusters alongside our cloud capacity, we're hiring Operations Engineers to keep the fleet alive. This is a hands-on role. You're first in line when nodes go bad, GPUs throw ECC errors, IB links flap, or a rack stops responding. You'll provision new nodes, validate them, ship them to production, and troubleshoot whatever entropy throws at them. You'll be on-call. You'll be in the weeds.

You're a fit if you've:

Administered Linux Systems in the critical path before

Troubleshooted GPU node issues: NVLink, NCCL, IB, driver and firmware bugs

Has experience in observability systems like Grafana and Prometheus

Scripted your way out of repetitive work (bash, python, go, whatever)

Who you are:

Curious. You don't accept "it's flaky" as a root cause

Comfortable with ambiguity. The runbook doesn't exist yet for half of what you'll do

On-call doesn't scare you

You'd rather automate a problem than fix it twice

Responsibilities:

Provision, validate, and triage GPU nodes across B300, H200, and H100 clusters

Troubleshoot hardware and software issues across compute, network, and storage

Monitor fleet health, take remediation action, push fixes upstream when needed

Write the runbooks. Improve the ones that exist. Delete the ones that don't work

 

Source: Fal careers (Greenhouse)

Similar roles