Olmo Maldonado is a staff engineer at Braintrust who builds infrastructure for evaluating and improving AI products. Previously, he spent more than seven years at Zapier, developing its public API, distributed workflow systems, and AI-powered automation products, including AI Zap Builder and Zapier Copilot.
A former Google engineer with a master’s degree in electrical engineering from UCLA, Maldonado founded Code RGV and launched a coding school in Mission, Texas, in 2016. He created its full-stack curriculum and built GitHub Spy to help instructors distinguish meaningful student programming from superficial activity.
At Zapier, Maldonado helped turn natural-language instructions into working automations while measuring whether those workflows selected appropriate applications, triggers, actions, and configurations. His approach to evaluation-driven development incorporated synthetic datasets, continuous-integration checks, deterministic and model-based graders, and requirements defined jointly by engineers and product managers. As he explained in a conversation with Braintrust founder Ankur Goyal, Zapier expanded its coverage from seven manually executed tests to more than 800 evaluations.
Zapier Copilot added conversational interactions and tool calls, making tracing essential for identifying failures and latency inside multistep workflows. When switching to GPT-4o reduced evaluation scores, Maldonado’s team simplified prompts tuned for previous models and adjusted tool-calling settings. Performance recovered, and the copilot’s reported streaming response time fell from approximately 14 seconds to three.
After moving from Zapier to Braintrust, Maldonado contributed configurable OpenAI clients and Azure-related improvements to the open-source autoevals project.
- Evaluate what the product actually does. Useful tests verify workflow structure, application selection, field configuration, and regressions—not merely whether generated language sounds convincing.
- Retest every model transition. Changes in prompt sensitivity, tool selection, accuracy, cost, and latency can break existing behavior even when models appear interchangeable.
- Make interfaces explicit. Maldonado advocates clear software interfaces, including OpenAPI, Model Context Protocol, typed schemas, and documentation that help agents understand tools and intent.
- Keep engineering discipline around agents. He warns against indiscriminate AI-assisted coding, emphasizing production safety, bounded execution costs, manageable tasks, and preserving developers’ context.
