Rishabh Bhargava is director of engineering at Together AI, where he leads voice AI, and the co-founder and former chief executive of Refuel.ai, which Together acquired in 2025. He builds infrastructure for two difficult stages of deploying AI: preparing reliable data and delivering responsive, dependable spoken conversations.
Bhargava studied computer science and artificial intelligence at Stanford and Cambridge and worked on machine-learning and data infrastructure at Primer.ai and Datacoral before founding Refuel with Nihit Desai. Their company tackled the delays and expense of preparing training and evaluation datasets through LLM-powered data labeling. Its products included Autolabel, an open-source library for labeling, cleaning, and enriching text, and Refuel LLM-2, a model family specialized for data enrichment.
When Together acquired Refuel, Bhargava and Desai brought its technology into Together’s cloud and inference platform. Bhargava subsequently took responsibility for voice AI, applying his infrastructure background to the demanding tradeoffs of real-time conversation.
- Voice agents are systems, not standalone models. Streaming transcription, language-model reasoning, tool use, and speech synthesis must work together within a narrow latency budget. Misheard names or medication propagate through the pipeline; poor turn detection makes agents interrupt; oversized models slow responses even when they reason better.
- Co-located voice infrastructure reduces delays that model optimization alone cannot address. Bhargava’s work on real-time voice agents places speech recognition, inference, and synthesis near one another, targeting end-to-end responses below 500 milliseconds. He argues that shrinking network hops from approximately 75 milliseconds to five can substantially improve an already optimized pipeline.
- Streaming-native speech recognition improves on repeatedly chunking audio for batch-oriented transcription models. Short look-ahead windows and cached computations help deliver usable transcripts quickly; Bhargava has described approximately 100-millisecond transcription completion at the 90th percentile for certain Together-hosted models.
- Thinker-talker architecture separates conversational responsiveness from difficult decisions: a smaller model maintains the interaction while a stronger backend model handles complex reasoning and tools. Guardrails must operate before text becomes speech, because spoken mistakes cannot be withdrawn.
- Direct speech-to-speech models could preserve emotional nuance, hesitation, interruptions, and overlapping speech without converting everything into text. Bhargava considers instruction following, tool use, auditability, and conversation-level evaluation the major obstacles to their wider production deployment, as detailed in his AI Engineer Europe session.
