Vals AI Raises $40M to Grade AI Models

The big picture: Vals AI, a San Francisco startup, has closed a $40 million Series A round at a $400 million valuation. The company, co-founded by Rayan Krishnan and Langston Nashold, aims to become the independent scorekeeper for AI models, addressing the breakdown of traditional benchmarking methods. Vals AI provides a robust system for evaluating frontier models on real-world tasks.

Why it matters:

• Broken Benchmarks: Traditional academic benchmarks are failing as frontier AI models increasingly ace public tests, which are often saturated, leaked into training data, or optimized against. This makes it difficult to accurately compare model performance.
• High Stakes: As AI models transition from answering questions to acting as unsupervised agents, the need for reliable, real-world performance evaluation becomes critical. Models that perform well on leaderboards may still fail in complex, multi-step deployment scenarios.
• Expertise-Driven: Founders Rayan Krishnan and Langston Nashold, both Stanford computer science alumni, bring prior experience in real-world measurement problems. Their team includes talent from Palantir, Microsoft, NVIDIA, Meta, and Hudson River Trading.

How it works:

• Hybrid Evaluation: Vals AI combines domain experts across fields like law, finance, healthcare, and coding with automated grading systems. This approach scores model output against professional standards, moving beyond simple textbook problem-solving.
• Private, Dynamic Test Sets: The company maintains private test sets, run in limited numbers to prevent contamination and gaming. Tests are retired and replaced (e.g., CorpFin for Excel-modeling) once they no longer effectively differentiate between strong and weak models, ensuring continuous relevance.
• Industry Adoption: Vals AI’s evaluations are cited in model cards by major developers including OpenAI, Anthropic, Google, Meta, and xAI. Enterprises utilize these scores to inform decisions on which AI models to deploy into production environments.

The catch: The AI evaluation space is becoming increasingly competitive, with other players like LMArena focusing on user-driven comparisons and Trismik applying psychometrics to the problem. Datacurve also addresses the need for private, high-quality datasets. Vals AI’s core challenge lies in continuously rebuilding and updating its test infrastructure at a pace that matches the rapid advancements and evolving capabilities of frontier AI models, ensuring its evaluations remain relevant and robust in a dynamic market.