Burak Cemil Özafşar is co-founder and chief technology officer of Mutagent, which builds tools for evaluating software agents, diagnosing production failures, and improving their performance. Before founding Mutagent, he was chief software architect at Beam AI; his background also includes study at the Technical University of Munich.
A production failure captures his practical approach. When an agent repeatedly mangled long UUIDs, Özafşar replaced the identifiers in its prompts with short labels, then mapped those labels back to their original values afterward. The fix addressed an architectural weakness instead of endlessly adjusting prompts. His account of the incident also highlights a constraint central to his thinking: diagnosing agent failures cannot cost more than running the agents themselves.
With fellow Mutagent co-founder Benedikt Sanftl, Özafşar develops systems linking development-time testing with production feedback. Their AI Engineer World’s Fair session described an offline loop for building, evaluating, and refining agents, paired with an online loop that converts real failures into new tests and targeted fixes.
- Eval-driven development: Özafşar treats evaluations as release gates for agent behavior. His evaluation methodology draws test criteria from domain expertise, user corrections, and production failures, allowing suites to improve as difficult cases emerge.
- Calibrated LLM-as-a-judge systems: Reliable scoring requires anchored rubrics, measurable judge variance, and validation against labeled holdout examples. Özafşar favors actionable binary checks where appropriate and deterministic code when ordinary programs can verify formats or schemas.
- Production-trace root cause analysis: Useful diagnosis considers an agent’s context, tool outputs, execution harness, and complete trajectory. Representative sampling controls cost, while explicit diagnostic assumptions help engineers distinguish supported findings from model speculation.
- Harness-independent agent specifications: Özafşar separates an agent’s responsibilities, tools, constraints, and success criteria from its implementation framework, making it easier to change execution environments without redefining the agent’s purpose.
- Human-supervised improvement loops: His approach to loop engineering connects automated execution, verification, diagnosis, and proposed repairs while preserving human judgment over consequential changes.
