Member of Technical Staff (Web Access Engineer, Crawler)

Perplexity
London, United Kingdom
Hybrid

Who this role is best for

Aimed at mid-to-senior level software engineers with web infrastructure experience who can independently debug and implement complex network and rendering systems.

Best fit for

  • Candidates with 5+ years in web infrastructure or crawling and a strong grasp of HTTP/2 and HTTP/3 protocols
    — “5+ years of software engineering experience in web infrastructure, browsers, CDNs, or crawling”
  • Individuals who have experience debugging JavaScript and working with Chrome DevTools and CDP
    — “experience debugging JavaScript and fluency with Chrome DevTools, CDP, HAR files, curl, and packet captures”
  • Professionals who can independently define and prioritize technical metrics for system improvements
    — “Define metrics for fetching reliability, content completeness, and error classification, and use them to prioritize improvements”

Things to consider

  • The role requires a deep understanding of client-side rendering and dynamic page handling
    — “Improve page loading, completion detection, and content capture to preserve relevant content and handle dynamic pages reliably”
  • Candidates must be comfortable working with distributed systems and managing large-scale crawlers
    — “Experience operating crawlers, headless browsers, or distributed content-acquisition systems at scale”

How to stand out

  • Highlight experience in building tools for failure reproduction and regression testing
    — “Build tools to reproduce failures, compare crawler output against expected page content, and turn findings into regression tests”
  • Emphasize your ability to trace technical issues across network protocols and client-side execution
    — “trace problems from network protocols through client-side execution”
  • Demonstrate your fluency with HTTP/1.1, HTTP/2, HTTP/3, and related concepts like caching and redirects
    — “Deep understanding of HTTP/1.1, HTTP/2, and HTTP/3, including status semantics, caching, conditional requests, content encoding, and redirects”
Pace · Fast PacedCollaboration · HighAutonomy · MediumDecision Impact · Team

Derived from job-description analysis by Serendipath's career intelligence engine.

What success looks like

  • Developed reliable fetching and rendering systems
  • Improved content quality and efficiency
Typical background
Web InfrastructureCrawling SystemsCDN Development

Skills & requirements

Required

GoPythonRustC++Http/1.1Http/2Http/3TLSDNSCDNCachingWeb PerformanceJavaScriptChrome DevtoolsCDPHAR FilesCurlPacket Captures

Preferred

CDN DevelopmentWeb CrawlingDistributed SystemsBrowser Engine Development

Stack & domain

GoPythonRustC++Http/1.1Http/2Http/3TLSDNSCookiesSessionsCDN CachingContent DeliveryHTTP Error HandlingRetry BackoffServer-defined Rate LimitsPage LifecycleNetwork IdleService WorkersClient-side RenderingShadow DOMJavascript DebuggingChrome DevtoolsCDPHAR FilesCurlPacket Captures

About the role

Original posting from Perplexity via Ashby

Perplexity is hiring a Web Access Engineer for the Crawler team to build reliable fetching and rendering systems for AI-powered search. You'll work on freely available public web content, within site permissions and approved access policies, and develop the infrastructure that supplies content to downstream indexing and retrieval systems.

Your focus will be diagnosing network errors, page-loading issues, and incomplete rendering. You'll trace problems from network protocols through client-side execution, distinguish technical failures from intentional access restrictions, and turn findings into production fixes that improve content quality and efficiency at scale.

Responsibilities

  • Own fetching and rendering improvements end to end, from investigation through implementation and production rollout, partnering with crawler engineers and SREs.
  • Investigate fetch and rendering failures across DNS, TLS, HTTP, caching, redirects, and client-side rendering, and classify root causes.
  • Improve page loading, completion detection, and content capture to preserve relevant content and handle dynamic pages reliably.
  • Ensure the crawler respects site-imposed access restrictions and rate limits, identifies itself clearly, and retries only transient technical errors.
  • Build tools to reproduce failures, compare crawler output against expected page content, and turn findings into regression tests.
  • Define metrics for fetching reliability, content completeness, and error classification, and use them to prioritize improvements.

Qualifications

  • 5+ years of software engineering experience in web infrastructure, browsers, CDNs, or crawling, with strong programming skills in at least one of Go, Python, Rust, or C++.
  • Deep understanding of HTTP/1.1, HTTP/2, and HTTP/3, including status semantics, caching, conditional requests, content encoding, and redirects; strong knowledge of TLS, DNS, cookies, and sessions.
  • Understanding of CDN caching and content delivery, HTTP error handling, retry backoff, and server-defined rate limits.
  • Knowledge of page lifecycle, network idle, service workers, client-side rendering, and shadow DOM; experience debugging JavaScript and fluency with Chrome DevTools, CDP, HAR files, curl, and packet captures.
  • Comfort working independently when the path forward is not fully defined.

Nice to have

  • Experience developing CDN, caching, or web performance systems.
  • Experience operating crawlers, headless browsers, or distributed content-acquisition systems at scale.
  • Contributions to a browser engine or JavaScript runtime.

Source: Perplexity careers (Ashby)

Similar roles

More roles at Perplexity