Hamel Husain is the founder of Parlance Labs, an independent machine-learning consultant, and a leading practitioner of domain-specific AI evaluation. He helps companies replace intuition about whether language-model applications work with disciplined analysis of actual failures, user needs, and expert judgment.
From enterprise analytics to practical AI
From 2011 to 2015, Husain was vice president of applied analytics at AlixPartners. He subsequently worked at DataRobot, Airbnb, and GitHub, progressing from automated machine learning and data science into developer-facing machine-learning infrastructure.
As a staff machine-learning engineer at GitHub, he led CodeSearchNet, an open-source initiative combining code datasets, baseline models, human relevance judgments, and a public challenge for evaluating semantic code search. His introduction to the project established a concern that still shapes his work: models improve faster when teams can measure outcomes against informed human expectations.
Husain also led fastpages, which made Jupyter notebooks easier to publish as blog posts, and later worked with fast.ai as an entrepreneur in residence. Through Parlance Labs, he now advises organizations building AI products and teaches practical AI evaluation with Shreya Shankar.
- Start with failures users actually encounter. Working with Rechat on its real-estate assistant, Lucy, Husain helped turn observed mistakes into inexpensive assertions and unit tests, run evaluations through existing continuous-integration workflows, and track results in Metabase. Synthetic prompts representing real-estate agents expanded test coverage. Lucy was Rechat’s product; Husain participated as an outside consultant. His account of the evaluation system emphasizes adapting tools to an established process.
- Make human review easy enough to happen. Husain advocates application-specific interfaces that place traces, relevant context, and labeling controls together so domain experts can inspect outputs without navigating disconnected dashboards. That review identifies concrete failure categories and creates useful data for subsequent model improvement.
- Calibrate automated judges against domain experts. Before trusting an LLM-generated assessment, he recommends collecting human-labeled examples, defining explicit criteria, and checking whether the automated evaluator agrees with knowledgeable reviewers. Straightforward assertions should come first; model-based judging becomes useful when product requirements cannot be captured by simple tests.
- Treat evaluation and data literacy as core engineering skills. Husain argues that teams should first hire people who can build applications, add data and platform capabilities as usage grows, and bring in specialized machine-learning expertise when meaningful production data exists. In his identifiable portion of a six-person AI engineering keynote, he challenges vague job descriptions that expect one engineer to master every stage simultaneously.
He also warns against organizational habits that exclude domain specialists: unnecessary technical jargon, inaccessible data, expensive infrastructure purchased before the problem is understood, and metrics disconnected from customer outcomes. His AI-strategy satire with Greg Ceccarelli frames these failures as management problems with technical consequences.
