Senior AI Data Scientist

Charlestown, MA • Beth Israel Lahey Health • Full-time • Day

When you join the growing BILH team, you're not just taking a job, you’re making a difference in people’s lives.

The Senior AI Data Scientist role serves as a technical leader for complex machine learning and generative AI initiatives within the organization. This role is responsible for designing, fine-tuning, evaluating, and governing large and small language models (LLMs) and advanced machine learning models used in healthcare and operational settings. The Senior AI Data Scientist owns model quality, evaluation rigor, and responsible AI outcomes, while mentoring junior team members and guiding project-level technical direction. The position emphasizes model development, experimentation strategy, evaluation frameworks, and applied generative AI. The Senior AI Data Scientist partners closely with engineering, clinical, and data teams to ensure AI solutions are accurate, safe, explainable, and fit for production use.

Job Description:

Essential Duties & Responsibilities including but not limited to:

• Lead the design, development, and evaluation of advanced machine learning and generative AI models across healthcare use cases.

• Serve as a subject matter expert on large and small language models (LLMs), including architecture tradeoffs, inference behavior, strengths, and limitations.

• Design and oversee LLM fine-tuning strategies, including instruction tuning and parameter efficient fine-tuning approaches, ensuring alignment with domain needs and model governance standards.

• Define and execute model evaluation frameworks for LLMs, including quantitative metrics, qualitative review, bias analysis, hallucination assessment, and safety testing.

• Guide and review prompt design and optimization strategies, ensuring robustness, consistency, and interpretability of generative AI outputs.

• Lead and improve retrieval of augmented generation (RAG) solutions, including data selection, grounding quality, relevance evaluation, and retrieval performance analysis.

• Conduct advanced exploratory data analysis, feature engineering, and statistical modeling to support predictive and prescriptive analytics.

• Translate complex clinical or business problems into clear modeling strategies and evaluation plans.

• Mentor entry level and midlevel data scientists through model reviews, best practice guidance, and collaborative problem solving.

• Partner with data engineering, software engineering, and clinical teams to ensure AI models are deployable, monitored, and compliant with governance requirements.

• Define and promote Responsible AI best practices, including fairness, bias mitigation, transparency, documentation, and appropriate clinical use.

• Communicate model behavior, tradeoffs, and outcomes to technical leaders, clinicians, and executive stakeholders.

Minimum Qualifications:

Education:

Bachelor’s degree in Data Science, Computer Science, Artificial Intelligence, Statistics, Mathematics, or related quantitative discipline required.

Master’s degree preferred.

Licensure, Certification & Registration:

None required.

Experience:

5+ years of experience in data science, machine learning, or applied AI roles.

Demonstrated experience leading model development and evaluation efforts, including generative AI or NLP-based solutions.

Experience mentoring or guiding junior data scientists.

Skills, Knowledge & Abilities:

Deep understanding of machine learning and generative AI fundamentals.

Handson experience working with LLMs, including finetuning concepts and inference tradeoffs.

Strong expertise in model evaluation, including both traditional ML metrics and qualitative LLM assessment.

Advanced proficiency using Python for data analysis, modeling, and experimentation.

Strong analytical judgment and ability to assess model risks and limitations.

Excellent written and verbal communication skills, especially for explaining complex AI behavior.

Ability to operate independently on complex problems while providing technical leadership to others.

Preferred Qualifications & Skills:

Advanced experience with LLM evaluation frameworks, test harnesses, or human-in-the-loop review.

Experience designing or governing RAG architectures.

Familiarity with healthcare data, clinical terminology, or regulated data environments.

Knowledge of AI ethics, fairness, bias mitigation, and responsible AI governance.

Experience contributing to AI standards, documentation, or best practice frameworks.

Exposure to production monitoring or model lifecycle management in regulated environments.

Dept./Unit-Specific Skills:

Key Business Relationships: (Title and Purpose)

1 AI Data Scientists

Technical leadership, mentoring, and model review

2 Data Engineering

Data availability, quality, and pipeline alignment

3 Clinical Stakeholders

Validation of model relevance and safety

4 AI Governance / Compliance

Responsible AI alignment and oversight