Kobie Crawford is a developer advocate at Snorkel AI working on agent evaluation, reinforcement learning, and the training data that makes autonomous systems dependable. His central argument is concrete: improving an agent’s behavior and testing the right capabilities can matter more than increasing model size.
Crawford previously worked in community and technical marketing at MosaicML, then joined Databricks through its acquisition of the company. As a Mosaic AI developer advocate, he worked on custom models, efficient training, enterprise data control, MLflow, and the Databricks Agent Framework. At Snorkel AI, his focus shifted toward measuring whether agents can reliably complete realistic work.
- Teach the missing behavior. In a Snorkel–UC Berkeley Agentica collaboration, Crawford described a four-billion-parameter Qwen3 model trained with GRPO and the rLLM framework to answer financial questions. While a 235-billion-parameter model queried nonexistent database tables and produced an unsupported answer, the smaller model learned disciplined tool use: discovering available tables, inspecting schemas, and correcting SQL errors. Performance on harder multi-table FinQA questions rose from 13.9% to 26.6%; notably, training on simpler single-table examples transferred to the harder tasks. The reported training run cost less than $500. His financial-analysis walkthrough makes the practical implication clear: targeted reinforcement learning can produce cheaper, more controllable enterprise systems.
- Make benchmark failures meaningful. Crawford evaluates task fidelity against four criteria: tasks must be achievable, nontrivial, logically correct, and supported by reliable environments. Broken dependencies, underspecified instructions, and mismatched tests create misleading failures. In a controlled comparison using the same model, compute budget, and task count, lower-quality tasks improved performance by approximately 1%, compared with approximately 6% for higher-quality tasks. He also advocates expert-written rubrics, human reviewers, and calibrated LLM judges to distinguish reasoning errors from infrastructure or tool-use failures.
- Evaluate software agents beyond one-shot success. Crawford introduced the Snorkel Agentic Coding benchmark and wrote about Terminal-Bench 2.0, emphasizing realistic, containerized engineering tasks. In his coverage of SlopCodeBench, led by Gabriel Orlanski, he highlighted another blind spot: agents can satisfy individual requests while accumulating technical debt and degrading software across successive revisions.
