Emmanuel Turlay

Emmanuel Turlay is director of engineering at Weights & Biases by CoreWeave, where he develops infrastructure for observing, evaluating, and improving AI agents in production. Before entering engineering leadership, he researched particle physics at CERN, built infrastructure at Instacart and Cruise, and founded the machine-learning tooling company Sematic.

From experimental physics to machine-learning infrastructure

Turlay earned a physics doctorate from Paris-Sud University and worked on electron and photon reconstruction for CERN’s ATLAS detector. He joined Instacart around 2014, working on order and payment infrastructure, before moving to Cruise and the challenge of converting autonomous-vehicle sensor data into usable training datasets and production models.

At Cruise, he helped develop Terra, an Apache Beam-based data-processing platform. Terra standardized permissions, dependency packaging, dataset access, execution, and lineage, allowing engineers to build sophisticated pipelines without mastering their underlying distributed infrastructure.

In 2022, Turlay founded Sematic, which participated in Y Combinator and raised $3 million in seed financing. Its open-source pipeline framework moved machine-learning workflows from notebook experiments into production using ordinary Python, tracked artifacts, and cloud execution. His founding rationale emphasized continuous retraining and evaluation as real-world conditions change.

The company later developed Airtrain AI for curating datasets, evaluating and fine-tuning language models, and comparing proprietary systems with open-source alternatives. Airtrain is now inactive. Turlay subsequently joined Weights & Biases as a member of technical staff before moving into engineering leadership, with recent work centered on W&B Weave and agent observability.

• Application-specific model evaluation: Standard benchmarks and text-overlap metrics cannot establish whether a model reliably extracts medical symptoms, identifies recipe ingredients, or generates valid API payloads. Turlay evaluates models against representative application inputs and explicitly defined success criteria, as illustrated by his AI Engineer Summit work on practical model evaluation.

• LLM-as-a-judge with explicit criteria: A separate model can grade outputs against a task-specific rubric across an independent evaluation dataset, producing score distributions that expose meaningful differences between candidates. Evaluator selection creates its own engineering tradeoff: GPT-4 may deliver stronger judgments, while Flan-T5 can reduce cost and latency.

• Production traces as an evaluation flywheel: Agent failures become new evaluation cases, expanding a test harness built from representative examples, heuristics, and model-based judges. His framework for agent evaluation treats reliability as an iterative operational discipline.

• Agent-native observability: Useful debugging must preserve multi-turn sessions, tool calls, long-running context, and interactions among agents. Turlay distinguishes measurable optimization loops—propose a change, execute it, evaluate the result—from unsupported claims that agents automatically improve themselves, a distinction central to his work on observable agent swarms.