Alejandro Vidal is the founder and chief executive of Mindmakers, an applied-AI company, and an advocate for psychometric model evaluation. Drawing on a background in psychology and computer science, he applies techniques originally developed to measure human abilities to a stubborn problem in artificial intelligence: benchmark scores often obscure what models actually know, how reliably they perform, and whether the tests themselves are sound.
Mindmakers began with a personalized-education prototype tested in classrooms in 2022 and incorporated in 2023. As the company expanded into applied AI, Vidal also taught the subject, including at ESADE. A 2024 PyData Madrid workshop brought together several emerging priorities: building Python coding assistants, retrieval-augmented generation, tool use, and attacks against language-model applications.
That same year, his analysis of OpenAI’s o1 reasoning model examined the gap between visible reasoning summaries and hidden chains of thought, the security implications of unexpected model behavior, and the practical tradeoffs among capability, latency, and business value.
A more rigorous way to measure AI
At AI Engineer World’s Fair 2026, Vidal demonstrated how psychometrics can move model evaluation beyond counting correct answers:
- Item response theory: Estimate a model’s underlying ability while accounting for each question’s difficulty and discriminatory power. Claude and Gemini models with nearly identical raw scores separated considerably when the difficulty of their correct answers was incorporated.
- Benchmark calibration: Flag questions that stronger models unexpectedly miss, exposing ambiguous wording, mislabeled answers, or items that fail to measure the intended capability. One example confused the number of passengers with the combined number of passengers and crew.
- Efficient evaluation: Selecting the most informative questions preserved approximately 99 percent correlation with one original ranking using 97 items instead of 484. Vidal cautions that well-designed benchmarks such as GPQA may offer far less scope for reduction.
- Residual-based model fingerprinting: Analyze unexpected responses to investigate inconsistent inference, possible benchmark contamination, and similarities among related or distilled models. His proposed system combines shared anchor questions with organization-specific fingerprint sets; unusual patterns are investigative signals, not proof of misconduct.
Vidal also writes about cognitive debt in agentic software development: teams can produce more code while losing the understanding necessary to audit decisions and maintain architectural control. His security interests extend to physical attacks involving computer vision. Current research directions include multidimensional ability models, combining evidence across benchmarks, incorporating latency and token usage into evaluations, and connecting psychometrics with alignment and mechanistic interpretability.
