What You Can Expect: Join a dynamic team that bridges the gap between AI research and production engineering. As a Machine Learning Engineer, you will play a crucial role in ensuring the scalability, architecture, and reliability of advanced AI systems. Collaborate with researchers to define metrics and policies while leading the development of safety infrastructure. Your contributions will shape AI model evaluations across various products, aiming to enhance both safety and reliability.
About the Team: In today's Generative AI landscape, trust is paramount. Our team is dedicated to creating AI models and products that are beneficial, safe, and truthful. We focus on developing robust systems designed to tackle risks such as inaccuracies, bias, harmful content, and security vulnerabilities at scale.
Responsibilities:
- Developing an Assessment Methodology and Strategy: Create a comprehensive approach for evaluating AI quality, safety, and alignment across diverse applications.
- Designing an Evaluation Framework: Strategize key performance metrics and safety standards for AI models.
- Identifying Systemic Risks: Proactively detect failure modes like hallucinations, bias, and vulnerabilities.
- Designing Risk Evaluation Protocols: Establish methods for measuring and addressing identified risks effectively.
- Providing Strategic Technical Guidance: Analyze options for developing or procuring evaluation tools to ensure optimal performance.
- Integrating Advanced Evaluation Techniques: Stay updated on cutting-edge methods for accurate assessments of AI models.
- Creating a Scalable Evaluation Infrastructure: Design a modular system for benchmarking and monitoring performance across product lines.
- Automating Quality Gates: Implement quality standards in the CI/CD pipeline for model deployments.
- Conducting A/B Testing: Build frameworks to validate the real-world impact of model adjustments and safety measures.
- Ensuring Engineering Excellence: Serve as the authority on AI evaluation and lead the team through complex system designs.
- Establishing Organizational Standards: Promote best practices for data labeling, model versioning, and more.
- Guiding Talent Development: Mentor team members to foster continuous learning and strong peer evaluations.
What We're Looking For:
- Strong academic and professional background in computer science or machine learning, with extensive experience in large-scale AI or distributed systems.
- Expert-level software development skills focused on architecture, maintainability, and reliability.
- Deep understanding of the LLM lifecycle and unique challenges in evaluating non-deterministic systems.
- Solid foundation in statistical analysis and experimental design (A/B testing).
- Extensive experience with PyTorch and the Hugging Face ecosystem.
- Proficiency in high-throughput inference engines (e.g., vLLM, SGLang, or TensorRT LLM).
- Expert-level Python skills; proficiency in languages such as Go, C++, or Java is a plus.
- Experience in CI/CD patterns and large-scale data processing pipelines.
- Knowledge of cloud-native infrastructure and distributed system monitoring.
Preferred Qualifications:
- Proven ability to define long-term technical roadmaps and mentor senior talent.
- Experience in model optimization and deployment of advanced evaluation methodologies.
- Deep expertise in AI alignment and reliability with a focus on benchmarking.
Salary Range: Minimum: $177,100.00 | Maximum: $387,500.00. Salary will reflect qualifications and experience.
Benefits: Enjoy a variety of perks aimed at supporting your physical, mental, emotional, and financial health while fostering work-life balance.
About Us: Zoomies strive to help people stay connected and be more productive together. Our collaborative environment emphasizes problem-solving and career growth opportunities.
Our Commitment: We value diversity and are committed to fair hiring practices, ensuring all candidates are evaluated on their skills and experiences. If you require accommodations during the hiring process, please inform us.