Neil Dwyer is a real-time media engineer at LiveKit and co-founder of Gabber, where he served as chief technology officer building infrastructure for expressive, affordable conversational AI. His specialty is making voice applications responsive enough for natural interaction and inexpensive enough for consumer products.
Dwyer started in live-streaming infrastructure at Bebo, developing a computer-vision pipeline that recognized moments in Fortnite gameplay and triggered on-screen effects. After working at Uber, he and his brother Jack founded a multiplayer-gaming startup focused on browser-based shared experiences and voice communication. Although the company closed, Dwyer’s work on a LiveKit software-development kit led him to join LiveKit, where he wrote the first code for LiveKit Agents, its framework for real-time conversational applications.
His expertise extended across the media stack: he authored practical guides to building spatial audio with WebRTC, React, and WebAudio and configuring LiveKit Cloud. In 2024, he and Jack founded Gabber to build AI personas incorporating voice, memory, tools, and eventually visual input. They concentrated on consumer applications including games, coaching, and interactive characters, where enterprise-oriented voice infrastructure could prove prohibitively expensive.
Engineering affordable conversational voice
- Eliminating head-of-line silence. While deploying the open-source Orpheus speech model, Dwyer found that generating silent introductory audio consumed valuable inference time even when the silence was removed before playback. Fine-tuning shortened that delay, freeing more of the conversational latency budget for language-model generation.
- LoRA-based voice adaptation. Gabber created expressive synthetic voices using lightweight fine-tuning, then combined multiple voice adapters with vLLM batching and dynamic quantization to support concurrent real-time synthesis on shared hardware.
- Consistent-hash load balancing. Dwyer routed voice sessions toward servers already holding the required adapter in memory, preserving continuity during streaming generation and limiting redistribution when infrastructure changed.
The resulting architecture combined LiveKit, WebRTC, Orpheus, SNAC audio tokens, and vLLM into a deployable consumer voice stack, documented through Gabber’s Python SDK and Dwyer’s AI Engineer World’s Fair presentation. He subsequently returned to LiveKit with the Gabber team, continuing to work on request routing and conversational latency.
