[Hiring] Staff Applied Scientist - AI Evaluation & Trust @Sayari

Role Description

Sayari builds AI systems for high-consequence analytical work where being "wrong" carries real-world weight. We are looking for a Staff or Principal Applied Scientist to join our AI Innovation Group as the trusted expert on AI Evaluation and Trust. You will own the "Judgment Layer" of our system:

•

Building the specialized judge models, statistical benchmarks, and multi-turn frameworks that ensure our agents act with the high bar of trustworthiness required by our national security and enterprise customers.

Qualifications

•

10+ years of Machine Learning experience with a focus on Deep Neural Network activities, evaluating model performance & trust.

•

1-2+ years’ experience focused on post-training activities.

•

1+ year experience creating benchmarks to evaluate LLMs.

Requirements

•

Deep expertise in LLM-as-judge architectures, multi-turn evaluation, and Reinforcement Learning (RL/RLHF/RLAIF).

•

Mastery of statistics and experimental design, including significance testing, distribution analysis, and inter-rater reliability.

•

Experience with Mixture-of-Experts (MoE) systems, routing behavior, and expert specialization.

•

Proven ability to own the path from data collection to production deployment; we are a small team and every role is "hands-on."

•

Understanding of Graph RAG and the unique challenges of evaluating non-deterministic, agentic workflows.

Preferred

•

Experience building, fine-tuning (LoRA, etc.), or pre-training models specifically for judgment, preference modeling, or classification tasks.

•

Background in cognitive science, intelligence community tradecraft, or research literature on expert judgment under uncertainty.

•

Experience building or managing large-scale annotation infrastructure and quality assurance protocols.

•

A record of published research or recognized work in preference modeling or AI alignment.

Benefits

•

100% fully paid medical, vision, and dental for employees and their dependents.

•

Generous time off; we observe all US federal holidays, close our office for a winter break (12/24-12/31), in addition to granting 18 PTO days and 10 sick days.

•

Outstanding compensation package; competitive commissions for revenue roles and bonuses for non-revenue positions.

•

A strong commitment to diversity, equity, and inclusion.

•

Eligibility to participate in additional benefits such as 401k match up to 5%, 100% paid life insurance (up to $100,000 coverage), and parental leave.

•

A collaborative and positive culture - your team will be as smart and driven as you.

•

Limitless growth and learning opportunities.

Pay Range

$195,000 — $205,000 USD