Senior Software Engineer

Lancedb
Americas +1 more
Hybrid

Who this role is best for

Aimed at experienced software engineers with a focus on distributed systems and open-source integration who thrive on independent problem-solving and collaboration.

Best fit for

  • Candidates with extensive experience in large-scale data systems and a passion for open-source contributions
    — “Contributor, committer, or PMC member in Apache or other large open-source projects”
  • Individuals with a strong background in distributed computing and performance optimization
    — “Designing and maintaining efficient distributed Lance dataset operations”
  • Professionals skilled in Rust and C++ with a track record in big data formats
    — “Experience with Rust (or willingness to learn it)”

Things to consider

  • This role demands a high level of technical independence and initiative in open-source projects
    — “Proven ability to move fast, work independently, and collaborate with a high-caliber team”
  • The position requires deep knowledge of open-source big data systems and their internals
    — “Deep understanding of internals of open-source Big Data or AI training systems”

How to stand out

  • Highlight contributions to distributed data systems and open-source integration projects
    — “Driving open-source community efforts to integrate the Lance format with Spark, Hive Metastore, Presto, Trino, Ray, and other data infrastructure systems”
  • Emphasize experience with high-performance computing and optimizing query execution
    — “Building efficient indices to enable predicate pushdown and accelerate queries in Spark, Ray, or Trino”
  • Showcase your ability to simplify complex distributed operations and improve usability
    — “Simplifying distributed operations, and usability and maintainability enhancements”
Pace · Fast PacedCollaboration · HighAutonomy · HighDecision Impact · Team

Derived from job-description analysis by Serendipath's career intelligence engine.

What success looks like

  • scale and performance improvements
  • integrations with data and AI ecosystem
  • improving internal data processing infrastructure
  • promoting the Lance format in open-source communities
Typical background
10+ years of experience building high-performance databasesdeep understanding of open-source Big Data or AI training systems

Skills & requirements

Required

High-performance ComputingBig Data SystemsDistributed SystemsOpen-source SystemsC++JavaScalaRustSparkRayTrinoHadoopFlinkIcebergDelta LakeClickhousePyTorchJAX

Preferred

Apache ArrowDatafusionParquetHive MetastorePrestoCommunity PresenceOpen-source Collaboration

Stack & domain

High-performance DatabasesBig Data SystemsLarge-scale Data ServicesOpen-source Big Data Or AI Training SystemsC++JavaScalaRustSparkRayTrinoPrestoPyTorchJAXAbility To Move FastWork IndependentlyCollaborate With A High-caliber TeamAIData InfrastructureDistributed Systems

About the role

Original posting from Lancedb via Ashby

About LanceDB

AI advances at the speed of its research, and research moves at the speed of its data. LanceDB is the AI-native Multimodal Lakehouse: one system where a researcher curates petabytes of video, audio, and every signal derived from them with a few lines of Python, and the next training run starts as fast as the next idea. Customers like Runway, Midjourney, and Netflix build the future of AI on LanceDB, from frontier and world models to robots and autonomous vehicles.

About the Role

We're looking for a Senior Software Engineer to help expand the reach of Lance and LanceDB within the broader data infrastructure ecosystem. You'll work at the intersection of high-performance computing, big data, and open-source systems. You will contribute scale and performance improvements, integrations with the wider data and AI ecosystem, simplifying distributed operations, and usability and maintainability enhancements.

What You'll Do

  • Designing and maintaining efficient distributed Lance dataset operations
  • Building efficient indices to enable predicate pushdown and accelerate queries in Spark, Ray, or Trino
  • Working on table formats, data encodings, and various aspects of the Lance format in Rust
  • Driving open-source community efforts to integrate the Lance format with Spark, Hive Metastore, Presto, Trino, Ray, and other data infrastructure systems
  • Operating and improving internal data processing infrastructure
  • Promoting the Lance format in open-source communities and at Big Data conferences

What We're Looking For

  • 10+ years of experience building high-performance databases, big data systems, or large-scale data services
  • Deep understanding of internals of open-source Big Data or AI training systems (e.g., Hadoop, Spark, Flink, Ray, Iceberg, Delta Lake, Hudi, ClickHouse, Trino, Presto, PyTorch, or JAX)
  • Strong experience with high-performance computing in C++, Java, and/or Scala
  • Experience with Rust (or willingness to learn it)
  • Proven ability to move fast, work independently, and collaborate with a high-caliber team

Nice to Have

  • Contributor, committer, or PMC member in Apache or other large open-source projects
  • Experience with Apache Arrow, DataFusion, Parquet, Iceberg, or Delta Lake
  • Track record of driving large features or integrations in distributed systems
  • Strong community presence and passion for open-source collaboration

Source: Lancedb careers (Ashby)

Similar roles

More roles at Lancedb