Darius Emrani is the founder and chief executive of Scorecard, which helps companies evaluate and improve AI agents. After building evaluation systems for rockets and self-driving cars, he brought the discipline of safety-critical simulation to a central problem in applied AI: determining whether an agent will work reliably outside a benchmark.
Emrani worked on avionics at SpaceX, led simulation work at Uber’s Advanced Technologies Group, and spent four years in simulation product management at Waymo. There, he helped develop systems for testing autonomous vehicles against difficult scenarios before deployment. A subsequent healthcare application for transcribing patient notes sharpened his interest in the consequences of AI errors.
He founded Scorecard to apply those lessons to language-model applications and agents. His writing on simulation-based development describes testing as an iterative process: generate realistic scenarios, expose failures, improve the system, and repeat. Scorecard’s customers include Thomson Reuters; the company announced a $3.75 million seed round led by Kindred Ventures in September 2025.
What reliable evaluation requires
- Deployment-specific evaluation. An agent should be tested against representative production inputs and the errors that matter in its actual domain, whether legal work, healthcare, or finance. Generic leaderboard performance cannot substitute for evidence about a particular application.
- Cost-aware model comparisons. Benchmark results become misleading when one model receives more computation, more attempts, or a more expensive inference configuration than its competitors. Meaningful comparisons disclose comparable operating conditions and cost-performance tradeoffs.
- Trustworthy benchmark governance. Financial relationships, uneven access to evaluation material, undisclosed methods, and repeated optimization against familiar questions can undermine confidence in benchmark results even without proof of misconduct.
- Style-controlled evaluation. Preference rankings can reward verbosity, formatting, personality, or confidence over factual accuracy. Emrani argues for separating rhetorical appeal from correctness and task completion, particularly when leaderboard incentives encourage optimization for appearances.
- Continuous agent evaluation. Reliable deployment requires repeatable testing against application-specific quality, cost, and latency measures, followed by production monitoring and further iteration. His critique of distorted AI benchmarks frames evaluation as ongoing engineering infrastructure, not a one-time score.
