Member of Technical Staff (Index, Search Crawler)

Perplexity
London +2 more
Remote

Who this role is best for

Strong fit for engineers with deep systems expertise who design and operate large-scale data pipelines, particularly those with experience in Go, C++, or Rust and a focus on distributed systems.

Best fit for

  • Engineers with 5+ years of building production backend or data infrastructure systems
    — “5+ years of experience building production backend or data infrastructure systems.”
  • Candidates experienced in high-throughput data processing and storage optimization
    — “Experience with high-throughput data processing systems and very large datasets.”
  • Individuals who have operated databases or indexing pipelines in production environments
    — “Experience building or operating databases, storage systems, data planes, or indexing pipelines.”
  • Engineers with strong systems-thinking skills and a focus on end-to-end data flows
    — “A systems-thinking mindset and the ability to reason about end-to-end data flows.”

Things to consider

  • The role demands high availability and fault tolerance in distributed systems
    — “Strong understanding of distributed systems, fault tolerance, consistency, and scalability.”
  • Candidates must be prepared to handle operational incidents in production systems
    — “Experience running production systems and handling operational incidents.”

How to stand out

  • Highlight experience in optimizing indexing strategies for performance and freshness
    — “Develop and optimize indexing strategies that balance performance, freshness, and resource efficiency.”
  • Emphasize contributions to observability in data pipelines with structured logs and metrics
    — “Define and implement observability across offline and near line pipelines, including structured logs, metrics, and data quality signals.”
  • Showcase work on storage formats and update mechanisms for large datasets
    — “Work on storage formats, compaction, and update mechanisms that keep data accessible and current.”
  • Demonstrate ability to build reliable, high-throughput systems with clear interfaces
    — “Build well-tested components with clear responsibilities and interface contracts that stay flexible as the system evolves.”
  • Position yourself as someone who collaborates across teams to align indexing with retrieval needs
    — “Partner with retrieval, ranking, and ML teams to make sure indexing outputs meet their requirements.”
Pace · Fast PacedCollaboration · HighAutonomy · MediumDecision Impact · Team

Derived from job-description analysis by Serendipath's career intelligence engine.

What success looks like

  • Designed and implemented indexing systems
  • Optimized indexing strategies
Typical background
5+ years of experience in backend or data infrastructure

Skills & requirements

Required

Large-scale Indexing SystemsData Processing PipelinesDistributed SystemsHigh-throughput Data Processing

Preferred

Machine LearningCloud Services

Stack & domain

GoC++RustDistributed SystemsFault ToleranceConsistencyScalabilityData ProcessingSearch Infrastructure

About the role

Original posting from Perplexity via Ashby

AI answers are only as trustworthy as the information behind them. Perplexity grounds its answers in fresh, real-world sources, and the index is what makes that possible. We're looking for an engineer to work on the indexing and data processing layer of Perplexity's Search Crawler. You'll build systems that ingest, process, and organize large volumes of crawled content into efficient, queryable structures, working primarily on offline and nearline pipelines that keep data fresh, complete, and accessible to downstream retrieval. Throughput, scalability, and correctness are critical across datasets that change continuously.

Responsibilities

  • Design, implement, and operate large-scale indexing systems and data pipelines at the core of Perplexity's search infrastructure.
  • Develop and optimize indexing strategies that balance performance, freshness, and resource efficiency.
  • Work on storage formats, compaction, and update mechanisms that keep data accessible and current.
  • Keep pipelines reliable and predictable under high-throughput conditions.
  • Build well-tested components with clear responsibilities and interface contracts that stay flexible as the system evolves.
  • Define and implement observability across offline and near line pipelines, including structured logs, metrics, and data quality signals.
  • Monitor throughput, resource usage, and cost, and drive optimisations as business needs change.
  • Partner with retrieval, ranking, and ML teams to make sure indexing outputs meet their requirements.
  • Enable safe experimentation on indexing strategies and processing logic through controlled rollouts and clearly defined quality signals.

Qualifications

  • 5+ years of experience building production backend or data infrastructure systems.
  • Strong programming experience in Go, C++, or Rust.
  • Experience with high-throughput data processing systems and very large datasets.
  • Experience building or operating databases, storage systems, data planes, or indexing pipelines.
  • Strong understanding of distributed systems, fault tolerance, consistency, and scalability.
  • Experience running production systems and handling operational incidents.
  • A systems-thinking mindset and the ability to reason about end-to-end data flows.

Source: Perplexity careers (Ashby)

Similar roles

More roles at Perplexity