Member of Technical Staff, Multimodal Pretraining Data

Hark
San Jose, CA
On-site

Who this role is best for

Geared toward data engineers comfortable with multimodal dataset construction and model performance evaluation, with a focus on distributed systems and collaboration across hardware and AI teams in San Jose.

Best fit for

  • Candidates with a background in creating multimodal datasets and optimizing model training through data experimentation
    — “develop methods to select, filter, deduplicate, and prepare text, code, image, and video datasets at scale”
  • Individuals experienced in large-scale data curation and synthetic supervision methods for AI pretraining
    — “Build quality classifiers and sampling methods, and measure their impact through controlled training experiments”
  • Professionals capable of working with infrastructure engineers to scale data pipelines for training systems
    — “Work with infrastructure engineers to scale data processing and deliver datasets that training systems can load efficiently and reproducibly”

Things to consider

  • The role demands a high level of precision in data alignment and provenance tracking
    — “track source provenance and dataset versions”
  • Candidates must be prepared to handle complex data modalities including video and code
    — “develop text, code, image, and video datasets used to pretrain Hark's models”

How to stand out

  • Highlight experience in building interleaved datasets and preserving document structure
    — “Build paired and interleaved datasets, including image-text documents and video-text data. Preserve document structure”
  • Emphasize your ability to design and implement synthetic supervision techniques for multimodal data
    — “Develop synthetic captioning and other supervision methods for image and video data”
  • Showcase your track record in improving model performance through data optimization
    — “A record of improving training data or model performance through research, datasets, systems, open-source contributions, or production work”
Pace · Fast PacedCollaboration · HighAutonomy · MediumDecision Impact · Team

Derived from job-description analysis by Serendipath's career intelligence engine.

What success looks like

  • develop methods to select, filter, and prepare datasets
  • build paired and interleaved datasets
  • measure corpus quality and coverage
Typical background
experience in data sciencebackground in machine learning

Skills & requirements

Required

Multimodal Pretraining DataText, Code, Image, And Video DatasetsControlled Data ExperimentsPython EngineeringDistributed Processing Frameworks

Preferred

Image-text AlignmentVideo Frame SamplingSynthetic Captioning

Stack & domain

PythonGoReactTypeScriptAIMLProblem-solvingTeamworkCommunicationMachine LearningData Science

About the role

Original posting from Hark via Greenhouse

About Hark

Hark is an artificial intelligence company building advanced, personalized intelligence. One that is proactive, multimodal, and capable of interacting with the world through speech, text, vision, and persistent memory.

We're pairing that intelligence with next-generation hardware to create a universal interface between humans and machines. While today's AI largely operates through chat boxes and decade-old devices, Hark is focused on what comes next: agentic systems that interact naturally with people and the real world.

To get there, we're developing multimodal models and next-generation AI hardware together - designed from the ground up as a single, unified interface for a new era of intelligent systems.

About the Role 

The Omni team at Hark is building the next generation of AI experiences beyond text, enabling models to understand and generate content across multiple modalities, including text, audio, and beyond. Our goal is to create seamless, real-time multimodal intelligence that powers intuitive and immersive user experiences.

As part of the Omni team, you'll develop the text, code, image, and video datasets used to pretrain Hark's models, with a focus on language and visual understanding. You'll identify useful sources, build filtering and synthetic supervision methods, and design training mixtures. You'll measure the effect of data choices through controlled experiments and work with infrastructure engineers to scale successful methods.

Responsibilities

Develop methods to select, filter, deduplicate, and prepare text, code, image, and video datasets at scale.

Build paired and interleaved datasets, including image-text documents and video-text data. Preserve document structure and the alignment between visual content and text.

Build quality classifiers and sampling methods, and measure their impact through controlled training experiments.

Develop synthetic captioning and other supervision methods for image and video data, and assess their accuracy, coverage, and biases.

Design mixtures and curricula across sources, domains, and modalities. Evaluate how sampling, sequence construction, and media resolution affect model quality and training efficiency.

Measure corpus quality and coverage, check for benchmark contamination with evaluation researchers, and track source provenance and dataset versions.

Work with infrastructure engineers to scale data processing and deliver datasets that training systems can load efficiently and reproducibly.

Requirements

Experience developing training data for language or vision-language models, with depth in one or more areas such as text and code curation, image-text datasets, or video data.

Experience designing controlled data experiments, measuring model improvements, and assessing whether results transfer across training scales.

Strong Python engineering skills and familiarity with distributed processing frameworks such as Spark, Ray, Beam, or similar systems.

Understanding of data quality issues such as duplication, noisy supervision, source imbalance, and benchmark contamination.

A record of improving training data or model performance through research, datasets, systems, open-source contributions, or production work.

Bonus Qualifications

Experience with image-text alignment, interleaved document construction, video frame sampling, or temporal grounding.

Experience with synthetic captioning, model-based filtering, or embedding-based data selection.

Experience with data mixtures, curricula, or scaling laws for data.

Familiarity with tokenizer design, visual representations, or training language and vision-language models.

Experience with provenance, privacy filtering, licensing metadata, or multimodal benchmark decontamination.

Compensation

The US base salary range for this full-time position is between $180,000 - $450,000 annually.

The pay offered for this position may vary based on several individual factors, including job-related knowledge, skills, and experience. The total compensation package may also include additional components/benefits depending on the specific role. This information will be shared if an employment offer is extended.

Source: Hark careers (Greenhouse)

Similar roles

More roles at Hark