Operations Engineer, HPC Networking

Fal

Remote

Who this role is best for

Geared toward mid-level engineers comfortable with hands-on HPC networking operations and debugging complex fabric issues in a remote setting.

Best fit for

Engineers who thrive on solving obscure hardware and firmware issues at scale.
— “chase down the weird stuff: link flaps, congestion, NCCL stalls, firmware bugs”
Candidates with production InfiniBand experience who prioritize root cause analysis.
— “Operated InfiniBand fabrics in production: subnet manager, routing, partitioning, monitoring”
Scripting-savvy professionals automating repetitive network operations tasks.
— “Scripted your way through repetitive operational work (bash, python, go, whatever)”

Things to consider

On-call responsiveness required for fabric incidents during critical customer events.
— “A fabric incident during a customer training run doesn't rattle you”
Expect hands-on work from cable installation to firmware debugging.
— “Brought up new fabrics from cable pull through validation”

How to stand out

Document past instances where you diagnosed and resolved rare fabric anomalies.
— “firmware bugs that only show up at scale”
Showcase automation scripts that improved network operational efficiency.
— “Improve the tooling and runbooks so the next incident resolves faster”
Highlight cross-functional collaboration experience with data center teams.
— “Partner with cluster ops on cross-domain incidents where the line between compute and network is blurry”

Pace · Fast PacedCollaboration · MediumAutonomy · MediumDecision Impact · TeamLevel · Senior

Derived from job-description analysis by Serendipath's career intelligence engine.

What success looks like

healthy network fabrics
incident resolution
tooling improvement

Typical background

network operationsfabric management

Skills & requirements

Required

Infiniband OperationEthernet RoceNetwork MonitoringFabric DebuggingScripting

Preferred

Spectrum-xLarge-scale GPU Cluster Networking

Stack & domain

InfinibandEthernet RoceSpectrum-xGpu Cluster NetworkingDetail-orientedCalm Under PressureRoot Cause AnalysisHpcNetworking

About the role

Original posting from Fal via Greenhouse

fal is the generative media ecosystem powering the next generation of AI products. We build the infrastructure, tools, and model access that teams need to move from idea to production, and do it at scale without compromise. For developers and enterprises, fal is the foundation that makes generative media not just possible, but practical: a unified platform where high-performance inference, orchestration, and observability come together to unlock new categories of AI-native products.
As generative media reshapes industries across a market projected to grow by hundreds of billions over the next decade, fal is becoming the ecosystem that ambitious teams build on.
About the role
We're hiring an Operations Engineer for HPC Networking to keep our InfiniBand and Ethernet fabrics healthy as we scale.
This is a hands-on role. You'll bring up new fabrics alongside DC ops, monitor the ones in production, and chase down the weird stuff: link flaps, congestion, NCCL stalls, firmware bugs that only show up at scale. 
You're a fit if you've:
Operated InfiniBand fabrics in production: subnet manager, routing, partitioning, monitoring.
Debugged the full stack: cables, transceivers, switch firmware, HCAs, drivers, NCCL.
Brought up new fabrics from cable pull through validation.
Scripted your way through repetitive operational work (bash, python, go, whatever).
Nice to have: Ethernet RoCE, Spectrum-X, or large-scale GPU cluster networking.
Who you are:
Detail-oriented. Cable plant hygiene is a personality trait.
Calm under fire. A fabric incident during a customer training run doesn't rattle you.
You read vendor release notes for fun, or at least out of self-defense.
You'd rather find the root cause than reboot the switch.
Responsibilities:
Monitor health and performance of InfiniBand and Ethernet fabrics: switches, HCAs, transceivers, links.
Investigate and resolve fabric issues: connectivity, congestion, performance regressions.
Support fabric bring-up alongside DC ops and customer-facing teams.
Run maintenance and upgrades on switches and control plane components.
Partner with cluster ops on cross-domain incidents where the line between compute and network is blurry.
Improve the tooling and runbooks so the next incident resolves faster than the last.
 

Source: Fal careers (Greenhouse)