Tech Lead Data Scientist, AI Evaluation – Monitoring

Job Description:

• Principal technical expert for how Geisinger evaluates, monitors, and optimizes AI systems
• Set technical direction for AI evaluation across a growing portfolio
• Provide technical leadership to a team of data analysts
• Partner with AI program teams to improve validation and monitoring methods
• Design validation studies and monitoring plans for AI systems
• Instrument production monitoring and define measurable metrics
• Mentor analysts on methodology and technical rigor

Requirements:

• 6+ years in data science, statistics, ML engineering, or applied quantitative research
• Strong foundation in experimental design and causal inference
• Hands-on experience designing and running model evaluation studies in real production settings
• Experience evaluating LLM or generative AI systems
• Proven ability to translate ambiguous failure modes into concrete evaluation designs
• Strong fluency in Python and SQL
• Experience with fairness and equity evaluation for ML systems
• MS or PhD in a quantitative field preferred

Benefits:

• Healthcare benefits from day one
• Vision insurance
• Dental insurance
• Domestic partner benefits
• Collaboration and cooperation atmosphere