Staff Research Engineer – RRI Harness

Sequen AI
United States
Remote

Who this role is best for

Best suited to experienced Python engineers with agentic LLM expertise working in enterprise AI infrastructure and distributed systems.

Best fit for

  • Experienced Python engineers with a focus on distributed systems and agentic LLMs
    — “Proven track record: Bring 7+ years of experience building production software, with strong Python and a history of distributed or long-running systems.”
  • Individuals who have designed evaluation infrastructure for ML systems
    — “Build eval infrastructure: Deliver a fast, fixed eval set, a scorecard with measured noise bands that runs on every harness change, and component ablations.”

Things to consider

  • The role demands end-to-end ownership of a critical system with high reliability expectations
    — “Extreme ownership: Take absolute accountability for a system end to end, with a deep care for reliability and observability.”
  • Candidates must be comfortable with long-running systems and GPU environments
    — “The harness is the runtime at the heart of RRI. It runs the agent team for hours or days, on GPUs inside our clients' own clouds or on Sequen-hosted instances.”

How to stand out

  • Highlight experience with multi-agent runtime systems and GPU failure handling in your resume
    — “Engineer for resilience: Make multi-day sessions survive GPU failures, node replacement, rolling upgrades, provider errors, and credit limits without a human restart.”
  • Showcase your ability to work with LLM agents in agentic loops and scale them
    — “Agentic LLM experience: Have built with LLMs in agentic loops — tool calling, prompt and context management, streaming, retries — and know how they fail at scale.”
Pace · Fast PacedCollaboration · HighAutonomy · HighDecision Impact · Team

Derived from job-description analysis by Serendipath's career intelligence engine.

What success looks like

  • Own the harness
  • Contribute to applied research
  • Build eval infrastructure
  • Engineer for resilience
  • Deliver observability
Typical background
7+ years of experience building production softwareStrong PythonHistory of distributed or long-running systems

Skills & requirements

Required

PythonLLMDistributed SystemsMachine LearningAgentic LoopsDebugging

Preferred

GoKubernetesGPU

Stack & domain

PythonLlmsDistributed SystemsKubernetesGPU NodesLeadershipProblem-solvingTeamworkAIMLRanking ModelsProduction Systems

About the role

Original posting from Sequen AI via Ashby

STAFF RESEARCH ENGINEER — RRI HARNESS & APPLIED RESEARCH

ABOUT US

Building the ranking intelligence layer for the internet.

Sequen builds Recursive Ranking Intelligence (RRI): an autonomous research engine in which a team of AI agents does the work of an ML research team, autonomously or together with ML researchers, in our clients' own clouds or on Sequen-hosted instances. The models RRI produces serve production traffic for the world's largest retailers, marketplaces and travel platforms, alongside Sequen's ranking platform, which runs frontier ranking models in production at sub-25ms latency and enterprise scale. Each gain compounds into revenue and margin lift measured in hundreds of millions of dollars per customer.

We are a small, highly technical, early-stage team turning recent advances in AI into production systems that operate under unforgiving real-world constraints.

ABOUT THE ROLE

The harness is the runtime at the heart of RRI. It runs the agent team for hours or days, on GPUs inside our clients' own clouds or on Sequen-hosted instances.

We're looking for a Staff Research Engineer to own the harness and contribute to applied research on how RRI's agents work. This is a research engineering role: you will form hypotheses about agent behaviour, run experiments to test them, and ship what works. You will make RRI reliable, safe and performant across long agent sessions, and extend what it can do. Along the way you will build the eval infrastructure that scores every change to the agents before it ships. You will work side by side with our Recursive Self-Improvement Lead, often on the same projects; your profile adds depth in agent engineering and infrastructure.

KEY RESPONSIBILITIES

  • Own the harness: Build and evolve the Python multi-agent runtime — orchestration, tool contracts, context and memory management, compaction, and model adapters across LLM providers.
  • Contribute to applied research: Design and run experiments on how the agents work — prompting, tool design, context and memory strategies, and model choice — and ship the changes that measurably improve RRI's results.
  • Build eval infrastructure: Deliver a fast, fixed eval set, a scorecard with measured noise bands that runs on every harness change, and component ablations.
  • Engineer for resilience: Make multi-day sessions survive GPU failures, node replacement, rolling upgrades, provider errors, and credit limits without a human restart.
  • Deliver observability: Build event streams, cross-session analysis, per-agent cost and token accounting, and admin tooling over MCP that let us see inside every session.
  • Cut waste: Find where agents idle, loop, or burn tokens, and fix it in code.
  • Design clean contracts: Work with the owners of our Go control plane and GPU proxy on the interfaces between components.
  • Support clients: Partner with applied scientists and forward-deployed engineers to diagnose and fix issues clients hit in production.

ABOUT YOU

  • Proven track record: Bring 7+ years of experience building production software, with strong Python and a history of distributed or long-running systems.
  • Agentic LLM experience: Have built with LLMs in agentic loops — tool calling, prompt and context management, streaming, retries — and know how they fail at scale.
  • System-wide comfort: Debug confidently across process, container, and network boundaries, on Kubernetes and GPU nodes.
  • ML fluency: Read a training script, understand a ranking metric, and tell a real regression from noise.
  • Extreme ownership: Take absolute accountability for a system end to end, with a deep care for reliability and observability.

STRONG CANDIDATES MAY ALSO BRING

  • Applied research experience: Have run ML or LLM experiments end to end, with baselines and ablations, and turned the results into product changes.
  • Evaluation infrastructure: Have built test harnesses, benchmarks, experiment tracking, or CI for ML systems.
  • Polyglot capabilities: Working knowledge of Go for contributions to the control plane and proxy.
  • Customer-controlled deployments: Experience shipping software into on-prem, BYOC, or air-gapped environments and their security constraints.
  • ML platform depth: Experience with PyTorch training at scale, GPU scheduling, or ML platforms.
  • Open-source footprint: Contributions to agent frameworks, eval harnesses, or ML tooling.

WHAT WE VALUE

  • Rigorous systems thinking: You design for the failure case first, because in a multi-day session every rare failure eventually happens.
  • Measure, then change: You would rather ship a smaller change with an eval score than a larger one on intuition.
  • Pragmatic speed: You move quickly without accumulating debilitating technical debt.

WHAT WE OFFER

  • High-impact influence: A staff-level role that owns the core runtime of Sequen's autonomous research engine.
  • Pioneering systems: The chance to build production infrastructure for long-horizon AI agents that train models serving enterprise traffic.
  • Complete flexibility: Unlimited paid time off, flexible hybrid/remote configurations, and a highly collaborative, world-class engineering culture.

Source: Sequen AI careers (Ashby)

Similar roles