Member of Technical Staff, Pretraining Evaluations

Hark
San Jose, CA
On-site

Who this role is best for

Strong fit for candidates who build evaluation systems and analyze model performance across modalities, with experience in statistical rigor and Python engineering.

Best fit for

  • Candidates with experience in model evaluation and statistical analysis of training runs
    — “evaluate Hark's models during pretraining and mid-training”
  • Individuals who have developed benchmarks and evaluation tools for language models
    — “A record of developing useful benchmarks, evaluation methods, or tools that inform research and engineering decisions”
  • Candidates who can work closely with researchers and understand their needs for model comparisons
    — “work with researchers to detect training-data contamination”

Things to consider

  • Compensation range is wide, potentially reflecting varying levels of experience or skills
    — “The US base salary range for this full-time position is between $180,000 - $450,000 annually”
  • The role demands version control of datasets and evaluation settings for reproducibility
    — “Version datasets, prompts, graders, and inference settings so results can be reproduced”

How to stand out

  • Highlight experience in creating reproducible evaluation pipelines for multiple model checkpoints
    — “building reliable evaluation pipelines that run efficiently across many checkpoints”
  • Showcase work on multimodal benchmarks or cross-modal evaluation tools
    — “Extend evaluation tools to image, video, speech, and audio tasks”
  • Emphasize statistical analysis skills in model evaluation and uncertainty quantification
    — “quantify uncertainty in results, and work with data researchers”
Pace · SteadyCollaboration · HighAutonomy · MediumDecision Impact · Team

Derived from job-description analysis by Serendipath's career intelligence engine.

What success looks like

  • accurate model evaluations
  • reproducible results
  • useful benchmarks
Typical background
language model evaluationmultimodal modelsAI research

Skills & requirements

Required

Model EvaluationHeld-out LossPerformance MeasurementReproducible ComparisonsStatistical JudgmentPython EngineeringBenchmark Development

Preferred

Human RatingsModel-based GradersCapability ScalingCode ExecutionMultimodal Tasks

Stack & domain

Model Evaluation SystemsExperimental ToolsReproducible ComparisonsLanguage Model Evaluation MethodsLog-likelihoodGenerative ScoringPrompt SensitivityFew-shot VarianceFormatting EffectsStatistical JudgmentUncertainty EstimationSample-size PlanningDistinguishing Statistical SignificanceBenchmark ContaminationSaturationPython EngineeringBuilding Reliable Evaluation PipelinesCode ExecutionImageVideoSpeechAudioProblem-solvingDecision-makingTeamworkCommunicationTime ManagementAttention To DetailCuriosityCreativityArtificial IntelligenceMultimodal ModelsNext-generation AI HardwareProactiveMultimodalPersistent MemoryAgentic SystemsNatural InteractionSeamlessReal-time Multimodal IntelligenceIntuitiveImmersive User Experiences

About the role

Original posting from Hark via Greenhouse

About Hark

Hark is an artificial intelligence company building advanced, personalized intelligence. One that is proactive, multimodal, and capable of interacting with the world through speech, text, vision, and persistent memory.

We're pairing that intelligence with next-generation hardware to create a universal interface between humans and machines. While today's AI largely operates through chat boxes and decade-old devices, Hark is focused on what comes next: agentic systems that interact naturally with people and the real world.

To get there, we're developing multimodal models and next-generation AI hardware together - designed from the ground up as a single, unified interface for a new era of intelligent systems.

About the Role 

The Omni team at Hark is building the next generation of AI experiences beyond text, enabling models to understand and generate content across multiple modalities, including text, audio, and beyond. Our goal is to create seamless, real-time multimodal intelligence that powers intuitive and immersive user experiences.

As part of the Omni team, you will build the multimodal foundation models at the core of this effort. You'll build the systems used to evaluate Hark's models during pretraining and mid-training. You'll track held-out loss, measure capabilities across text and multimodal tasks, and make comparisons between training runs reproducible. Your analysis will help researchers choose training recipes, select checkpoints, and identify regressions.

Responsibilities

Build and maintain an evaluation harness for held-out loss, performance across data domains, and capabilities in text, code, math, and reasoning.

Design evaluations for early checkpoints and smaller models using log-likelihood scoring, few-shot prompts, and generated completions.

Assess benchmark coverage and saturation, quantify uncertainty in results, and work with data researchers to detect training-data contamination.

Version datasets, prompts, graders, and inference settings so results can be reproduced and compared across checkpoints and training runs.

Analyze experiments and work with researchers to identify useful recipe changes, select checkpoints, and prioritize capability improvements.

Develop new benchmarks for capabilities that existing evaluations do not adequately measure.

Extend evaluation tools to image, video, speech, and audio tasks in collaboration with multimodal researchers.

Requirements

Experience building model evaluation systems or experimental tools for reproducible comparisons across models or checkpoints.

Knowledge of language model evaluation methods, including log-likelihood and generative scoring, prompt sensitivity, few-shot variance, and formatting effects.

Strong statistical judgment, including uncertainty estimation, sample-size planning, and distinguishing statistical significance from practical improvements.

Experience identifying benchmark contamination, saturation, or gaps between benchmark scores and the intended capability.

Strong Python engineering skills, including building reliable evaluation pipelines that run efficiently across many checkpoints.

A record of developing useful benchmarks, evaluation methods, or tools that inform research and engineering decisions.

Bonus Qualifications

Experience designing benchmarks with human ratings or model-based graders.

Experience studying capability scaling or predicting larger-model performance from smaller experiments.

Experience with code execution sandboxes, math verifiers, or other programmatic graders.

Experience evaluating checkpoints during pretraining or mid-training.

Multimodal evaluation experience: audio, speech, vision, or interleaved inputs.

Experience evaluating model quality under constrained inference budgets.

Compensation

The US base salary range for this full-time position is between $180,000 - $450,000 annually.

The pay offered for this position may vary based on several individual factors, including job-related knowledge, skills, and experience. The total compensation package may also include additional components/benefits depending on the specific role. This information will be shared if an employment offer is extended.

Source: Hark careers (Greenhouse)

Similar roles