No applications. Teams find and message you directly.
Join Alpaca as a Senior Data Scientist focusing on AI Evaluation. Design quality metrics and evaluations for models, ensuring high standards in a fast-paced fintech environment. Competitive salary and benefits offered.
Alpaca is a US-headquartered, global leader in agent-first brokerage infrastructure for stocks, ETFs, options, crypto, fixed income, 24/5 trading, and more. Amongst our subsidiaries, Alpaca is a licensed financial services company, serving hundreds of financial institutions across 40 countries with our institutional-grade APIs. This includes broker-dealers, investment advisors, wealth managers, hedge funds, and crypto exchanges, totaling over 10 million brokerage accounts. Our global team is a diverse group of experienced engineers, traders, and brokerage professionals who are working to achieve our mission of opening financial services to everyone on the planet. We're deeply committed to open-source contributions and fostering a vibrant community, continuously enhancing our award-winning, developer-friendly API and the robust infrastructure behind it.
- Design AI evaluations: Define ground truth, metrics, and scoring methods for models and agents.
- Build repeatable eval loops: Track quality over time and catch regressions before release.
- Partner on infrastructure: Work with engineering and analytics engineering to operationalize eval harnesses.
- Drive iteration: Translate eval results into actionable recommendations for system improvements.
- Establish quality standards: Set evaluation guidelines, documentation, and review practices.
- Mentor and align: Foster evaluation best practices and build a culture of measurable AI quality across the team.
- Track record of quantitative measurement rigor (e.g., LLM/model evaluation, metric validation, or experimentation).
- Strong statistical and ML foundation—you treat evaluations as experiments (sample sizing, confidence intervals, significance, handling non-determinism) and validate automated graders against human ground truth.
- Proficiency in Python and SQL, with experience evaluating models in production environments.
- Strong judgment in defining quality metrics and ground truth for ambiguous outputs.
- Excellent communication and cross-functional collaboration skills to align technical teams and leadership.
- Strong problem-solving ability in fast-paced, greenfield environments.
- 6–10 years in quantitative data science or ML, with focused experience in measurement or evaluation. A quantitative degree is a plus; equivalent industry experience is equally welcome.
- Competitive Salary & Stock Options
- Health Benefits
- New Hire Home-Office Setup: One-time USD $500
- Monthly Stipend: USD $150 per month via a Brex Card
- Alpaca is proud to be an equal opportunity workplace dedicated to pursuing and hiring a diverse workforce.