Ben Perlmutter is an AI engineer, technical writer, and co-author specializing in production-ready AI assistants and the evaluation of models that interact with databases. His work spans MongoDB’s documentation chatbot, reusable conversational-AI infrastructure, and benchmarks measuring whether generated database queries actually work.
Earlier projects included full-stack JavaScript applications, interactive educational software, and documentation tools. At MongoDB, he helped develop the MongoDB Chatbot Framework, combining a TypeScript and Express server, React interface, Atlas Vector Search, data ingestion, conversation persistence, and streaming responses. The framework was subsequently deprecated, and its public repository was archived in 2025.
In 2024, Perlmutter joined lead instructor Apoorva Joshi for an AI Engineer World’s Fair workshop on building a research assistant capable of retrieving papers, answering questions, and retaining conversational context. He also co-authored Building AI Intensive Python Applications, covering language models, vector search, retrieval-augmented generation, and reliable application development.
- Organizational knowledge as AI infrastructure. Perlmutter argues that documentation, support materials, and code repositories can power retrieval systems, evaluations, synthetic training data, fine-tuning, and agents. Making those resources reusable also requires careful access controls and privacy safeguards.
- Natural-language-to-MongoDB evaluation. His database-query benchmark contains 766 test cases across eight sample databases and measures whether generated queries execute and return useful results. Annotated schemas and representative documents improve performance; agentic approaches can help but increase cost and latency.
- Task-specific search benchmarks. His Natural Language to Atlas Search Benchmark evaluates executable search queries, operator selection, and ranked-result quality. Its findings challenge universal prompt recipes: configurations that improve one model can degrade another, making application-specific evaluation essential.
