Spoken interaction
What a voice system must coordinate
Speech can make an interface usable when a keyboard or display is inconvenient. But spoken output arrives sequentially: a listener cannot scan several options at once or easily revisit an earlier detail. The design therefore needs manageable answers and inexpensive ways to repeat, correct, or switch channels. These concerns predate modern language models; early voice notetakers and telephone information services already had to address them.
A speech model is a learned system that recognizes, represents, or generates speech. Automatic speech recognition (ASR) converts speech into text; text to speech (TTS) generates spoken audio from text. A common conversational architecture places a large language model—a model that generates text responses—between them. This cascade exposes useful intermediate artifacts, but attaching those components does not settle when to answer or how to handle interruption.
| Responsibility | Artifact or decision |
|---|---|
| Capture | Microphone audio with a format and time basis |
| Recognition or audio understanding | Provisional words or audio-derived representations |
| Response production | An answer, clarification, or request to external software |
| Synthesis | Generated audio representing the response |
| Transport and playback | Audio received, queued, and eventually played |
| Interaction control | When to listen, speak, interrupt, or repair |
A conversational turn is a contribution to an exchange; turn-taking coordinates opportunities to make those contributions. Silence, overlap, and acknowledgment are part of that coordination, not merely gaps between API calls. A system needs understandable speech, appropriate timing, recoverability, and useful completion. Here, real-time responsiveness means serving the interaction while it unfolds, not promising a hard deadline for every operation. The broader choice between unified and modular processing is developed in Choose the boundary you can operate.
Foundations that still matter
Recognition and conversation developed around different problems. Recognition must interpret variable acoustic evidence. Conversation must coordinate contributions and resolve misunderstandings. Interactive systems bring these problems together, but improving one does not eliminate the others.
| Development | Contribution and continuing consequence |
|---|---|
| 1974 — Conversational turn allocation | Harvey Sacks, Emanuel Schegloff, and Gail Jefferson described possible completion points and rules for selecting the next speaker. A speaking opportunity is not simply an absence of sound. Original paper. |
| February 1989 — Statistical sequence recognition | Lawrence Rabiner's hidden Markov model tutorial synthesized earlier work on hidden evolving states that produce observed acoustic measurements. Scoring a recording and choosing its most likely state alignment are different computations. Tutorial. |
| May 1997 — Telephone dialogue in public use | MIT's JUPITER weather service combined recognition, language understanding, database queries, response generation, and synthesis. Its displayless interface had to narrow requests and keep spoken answers manageable; lack of barge-in also caused clipped input. System account. |
| 2009 — Explicit incremental revision | A general incremental-dialogue framework represented partial results as linked units that could be added, revoked, or committed. Downstream interpretations could begin early while retaining their dependence on revisable evidence. Original paper. |
The recurring problem is commitment under incomplete information. A recognizer can prepare a hypothesis before the sound ends; a dialogue system can prepare an answer before the turn ends. Once an answer becomes audible, however, internal revision alone cannot erase it. Later neural methods change how recognition and synthesis work, while these interaction obligations continue.
Representing speech
Samples, frames, and speech time
A waveform describes amplitude over time. Digitization measures it at discrete times and represents each amplitude with finite precision. Sample rate counts measurements per second per channel; channels contain separate signals. Pulse-code modulation (PCM) stores quantized amplitude values, but a PCM contract must also specify width and packing. For example, the documented Windows 16-bit stereo format interleaves signed little-endian left and right samples. The label PCM alone does not establish those details.
The word frame is overloaded. A sample frame contains simultaneous channel values; an analysis frame contains a short interval of samples. A codec frame and a transport packet are other groupings. None is intrinsically a word or a conversational turn. Opus, for example, supports several frame durations and can combine frames into longer packets. Longer packets reduce some overhead but add waiting and put more audio at risk when one packet is lost.
Resampling constructs values on a new time grid while preserving intended duration. Merely interpreting unchanged values at half their original rate doubles duration and halves playback frequencies. A one-cycle-per-second tone sampled at deliberately tiny rates makes the time-grid difference clear; this teaching example is not speech. Preserve stream identity, rate, and time origin through conversion, as explained in Keep the coordinates that make the claim true.
Resampling versus changing the rate label
A recognizer often needs changing frequency patterns rather than raw amplitudes. A spectrogram stacks frequency measurements from successive, usually overlapping windows. Log-Mel features pool frequency bands and apply logarithmic compression. These representations describe local acoustics; they do not directly identify words. Prosody—rhythm, stress, intonation, pauses, and related delivery—also carries information that plain transcription only partly retains. At capture time, inspect the track's actual settings: browser constraints request a configuration rather than proving which configuration was selected.
Recognizing and generating speech
From acoustic frames to words
Recognition aligns variable-duration acoustic sequences with fewer linguistic units, usually without frame-level training labels. An encoder maps audio features to numerical representations; see Different signals become different representations. Alignment determines which acoustic steps support each output symbol.
Connectionist Temporal Classification (Alex Graves and colleagues, 2006) learns this mapping by predicting labels plus blanks at each step. Training sums probabilities over all paths consistent with the target. To collapse a path, merge adjacent repeated labels before removing blanks. For example, a a ∅ a a → (merge adjacent repeats) a ∅ a → (remove blanks) aa, where ∅ is blank. A separating blank preserves two occurrences; it does not indicate conversational completion.
Recurrent neural network transduction, described by Graves in 2012, combines acoustic representations with previous output symbols. It also sums over valid alignments during training. At inference, a search procedure instead considers candidate output sequences. A bounded beam retains a limited set of candidates, exchanging computation and memory for an approximation to a broader search. The alignment objective and the decoding search are separate choices.
Neither alignment objective guarantees streaming. A causal encoder uses only audio already received; lookahead, also called right context, requires waiting for additional audio before making a prediction. Attaching a streaming decoder cannot remove an encoder's need for future input. NeMo's cache-aware streaming models train with restricted context and retain activations—the intermediate numerical results computed from earlier audio. Reusing those results avoids some repeated computation from processing overlapping windows. More lookahead supplies additional acoustic evidence at the cost of waiting. Removing lookahead eliminates that wait, not computation or transport delay.
Whisper, presented by Alec Radford and colleagues in 2022, pursued broad speech-recognition robustness through large-scale weak supervision. Its encoder-decoder maps log-Mel features to text and task tokens, supporting several speech tasks. Windowed long-form processing and incremental interactive recognition nevertheless impose different requirements. Wrapping a batch recognizer with chunking, repeated calls, and transcript stitching can expose partial text without making its internal computation streaming-native.
Partial words are revisable evidence
An incremental recognizer produces hypotheses before all relevant audio has arrived. Later sound can change earlier words, not merely append new ones. The incremental-dialogue example of four becoming forty captures the consequence: an interpretation that used the quantity four must be reconsidered. Give partial results stable identities and record support links to dependent interpretations and prepared work. Starting downstream work early is useful only when those links allow the application to invalidate and revise work based on a changed provisional interpretation.
Revised words invalidate dependent work
Support arrows identify dependencies. The separately labeled invalidation row shows revocation propagation; neither is audio transport or a time scale. Unit identities let the application invalidate dependent work.
Before revision
provisional wordsupports →Q1: quantity = 4supports →P1: preparation using 4
After later audio changes the word
P1 is internal preparation, not an executed external action.
replacement wordsupports →Q2: quantity = 40supports →P2: new preparation
Independent state has no W1/Q1 dependency and stays unchanged in both states.
| Signal | What it supports | What remains unresolved |
|---|---|---|
| Provisional hypothesis | The recognizer's current interpretation | Earlier words may change |
| Estimated stable prefix | A prediction that the prefix will persist | Persistence is not correctness or an irrevocability guarantee |
| Amazon Transcribe Stable=true | The item is fixed under that service's stabilization contract | The fixed word may still be wrong |
| Final recognition segment | The recognizer has completed that segment | The person may be pausing within a larger contribution |
Stability and confidence answer different questions. The 2012 word-stability study used evidence such as how long a prefix had survived and how much following audio was available. Waiting for more stable text adds lag. Whether the words are correct requires separate evidence; see Sources of predictive uncertainty.
Event identity matters as much as text content. OpenAI's documented realtime transcription events identify their input items, and completion events from different turns need not arrive in order. Associate each result with its item rather than appending by arrival time. In an audio-native system, a separately generated input transcript may also differ from the model's own interpretation of the audio.
Test recognition where the task is vulnerable: names, numbers, accents, language switching, and brief answers. Context can narrow plausible vocabulary, but it can also favor what the system expects over what was said. A clinical implementation describes using conversation and medication context, with another recognition pass for single-word replies. Separately, research on Whisper shows that non-speech audio can produce text. Fluent output is therefore not proof that corresponding speech was present.
From text to speakable phrases
Producing a response's meaning and producing its sound are different jobs. Text normalization chooses spoken forms for written numbers, dates, and abbreviations. Pronunciation determines how those words sound; grapheme-to-phoneme conversion maps written symbols to speech-sound symbols. A pronunciation lexicon supplies stored pronunciations, but context may still be needed: read sounds like “reed” in “I read every day” and “red” in “I read yesterday.” Phrasing, duration, and prosody determine how the resulting words unfold in speech.
A conventional neural synthesizer predicts an acoustic representation, such as a Mel spectrogram, and a vocoder converts that representation into waveform samples. This is a functional account, not a requirement that every modern model expose separate modules. Explicit pronunciation hints can improve control over difficult names, but supported controls and their interaction with automatic synthesis are implementation-dependent.
Waveform generation
WaveNet, introduced by Aaron van den Oord and colleagues at DeepMind in 2016, changed waveform generation. Earlier approaches included concatenating recorded units and predicting acoustic parameters for a vocoder. WaveNet instead predicted each waveform sample from preceding samples using dilated causal convolutions. Listening tests favored it over the selected English and Mandarin baselines, but sequential sample generation left a substantial serving problem.
Parallel WaveNet addressed that cost by training a parallel student generator against an autoregressive teacher. The November 2017 report described deployment in English and Japanese Google Assistant voices. This is an example of distillation: moving knowledge from a teacher into a student with different execution properties. Faster waveform production did not itself solve recognition or conversational timing.
Text lookahead
Incremental text is not automatically ready for incremental speech. A partial spelling, abbreviation, or unfinished phrase may lack the context needed for pronunciation and emphasis. Waiting for a coherent phrase gives the synthesizer more information; starting earlier reduces onset delay but can make adjacent chunks sound disconnected. A 2020 incremental-TTS study found that added lookahead brought its Tacotron 2 output closer to full-context synthesis, though listeners still distinguished limited-lookahead output. That model-specific result supports testing the tradeoff, not prescribing a universal number of words to buffer.
Audio-native models
Learned audio codes
Predicting every waveform sample creates a long sequence. A neural audio codec offers a more compact interface: an encoder transforms waveform intervals into numerical representations, a quantizer maps them to discrete codebook entries, and a decoder reconstructs sound. A codebook is a learned collection of vectors; its indices identify vectors, not written words. Reconstruction is lossy even when it sounds convincing.
SoundStream, described by Neil Zeghidour and colleagues in July 2021, used residual vector quantization: successive codebooks encode corrections to remaining representation error. Dropping quantization layers during training allowed one model to support different bitrates. EnCodec, first posted on October 24, 2022 by Alexandre Défossez and colleagues, built on this encoder–quantizer–decoder structure with additional training techniques. It distinguished a causal, streamable model from one using future context. In a simplified two-codebook case, the first selected vector q₁ approximates encoded representation z; the second quantizes the residual z−q₁ to select q₂. Their indices identify vectors, and q₁+q₂ approximates z for waveform decoding. Compression fidelity and conversational usefulness remained different evaluation targets.
Indices select vectors; vectors reconstruct the representation
Once audio has a discrete representation, a sequence model can predict audio codes rather than individual samples. VALL-E demonstrated this approach in its 2023 work: text-derived phonemes condition linguistic content, while a reference recording conditions acoustic characteristics. A codec decoder turns predicted codes into sound. Its evaluations separated intelligibility, speaker similarity, and human judgments; reported omitted or duplicated words show why a plausible voice is not enough.
Keep the role of each encoding explicit. A call codec packages sound for transmission; learned audio codes can serve as a model's prediction vocabulary. These roles can overlap, but an audio code is neither a transport packet nor a waveform sample. In the documented Moshi implementation, Mimi represents 24 kHz audio at 12.5 frames per second: one representation frame covers 80 ms containing many waveform samples.
Architecture and duplex behavior
A text handoff makes the words entering the response model inspectable, but it can discard delivery cues. Direct audio input retains more of those cues while changing which intermediate artifacts the application can inspect or control. These are information-boundary choices, not a guarantee that one architecture is faster or more capable for every task.
| Architecture | Information path | Operational consequence |
|---|---|---|
| Text-mediated cascade | Audio → transcript → response text → speech | Inspectable text boundaries and reusable text agents; recognition errors and lost delivery cues can propagate |
| Audio input, text output | Audio encoder and connector → language model → text | Avoids requiring a transcript-only input handoff, but still needs synthesis for spoken output |
| Speech-to-speech | Audio-derived input → generated audio, sometimes with associated text | Can retain acoustic information across response generation; alignment, controls, and diagnosis depend on the implementation |
Ultravox illustrates the intermediate architecture: its documented implementation accepts audio and text while emitting streaming text. GPT-4o's May 13, 2024 announcement marked an integration milestone, presenting one model trained across text, vision, and audio instead of the earlier voice cascade. The announcement did not mean immediate general availability of that voice experience: the initial rollout exposed text and image input with text output.
Simultaneous streams
Full duplex means simultaneous input and output; half duplex alternates them. A transport may support both directions while a model still alternates listening and speaking. Conversely, a model that accepts ongoing input still needs a media implementation that handles echo and lag. Moshi models incoming and outgoing audio streams with text associated with its own speech. Kyutai publicly unveiled the experimental prototype on July 3, 2024; that event is distinct from its later paper and downloadable-model availability.
Simultaneous streams do not settle role control or reliable task behavior. PersonaPlex, described by NVIDIA's Rajarshi Roy and colleagues on January 15, 2026, extended Moshi with text conditioning for role and background and audio conditioning for vocal characteristics. Its training mixed real conversations with synthesized role-directed dialogues. The work addresses control without abandoning concurrent speech; developer-reported evaluations do not establish reliable outcomes in every live service.
Retaining tone also does not prove that a model uses it appropriately. If training tasks can be solved from factual content alone, they may provide little incentive to respond to hesitation or delivery. Evaluate the required behavior directly rather than treating audio input as evidence of emotion understanding. Select the architecture around needed information, pronunciation control, diagnosis, and execution reliability—not a speech-to-speech label.
The live audio path
Buffering against playback deadlines
Audio production and audio consumption run on different schedules. Services may generate or deliver chunks in bursts; ordinary playback consumes one second of sound per second. Jitter is variation in arrival timing. A jitter buffer holds received audio before playout so temporary variation is less likely to exhaust available sound. An underflow occurs when playback needs audio that is not ready.
A larger startup buffer can absorb a longer arrival gap, but the listener waits longer before speech begins. Hold chunk arrivals fixed and vary only startup time to see the tradeoff. Its simplified player pauses on underflow; real media systems may instead conceal missing sound or adjust playback speed. WebRTC's NetEq adapts delay, handles late packets, and uses concealment. Concealment substitutes audio; it does not recover the missing original samples.
Startup waiting changes continuity
Only startup delay changes. Four 400 ms chunks arrive at 0, 0.3, 1, 1.2 seconds. Arrivals at an exact depletion boundary become available before playback resumes; no gap is counted there.
Onset 0s · Total gap 0.2s (0.8–1s) · Finish 1.8s
| Fixed comparison | Onset | Gap | Finish |
|---|---|---|---|
| Immediate | 0 s | 0.8–1.0 s | 1.8 s |
| Wait 0.4 s | 0.4 s | None | 2.0 s |
For this fixture, 200 ms is the smallest gap-free startup delay. It is not a production recommendation; later arrivals or an underspeed producer change the requirement.
Backpressure makes a slow consumer constrain upstream production. A bounded queue can block a producer when capacity is reached; cancelling, dropping, or rejecting audio is a separate policy. Measure capacity in the resource that matters. Five fixed-duration chunks bound pending duration, whereas five variable-duration chunks do not. Queue clearing on interruption is cleanup, not continuous backpressure: the described Pipecat FrameQueue implementation inherits an unbounded queue while providing interruption-reset behavior.
Transport changes who owns these responsibilities. WebRTC supplies specialized media handling; WebSockets offer a simpler ordered connection often useful for prototypes, control messages, and server-to-server communication. TCP's ordered delivery can make newer audio wait behind missing older data. Choosing WebSocket audio is therefore a decision to manage the required pacing, buffering, and adaptation—not an automatic error. One application can use both transports for different roles.
Packet order is not source time. The Real-time Transport Protocol (RTP) carries media with sequence numbers for packet ordering and timestamps for sampling instants. The payload format determines the timestamp clock rate. Neither field tells you when a device actually played the sound. Preserve the distinctions among generated, transmitted, received, decoded, scheduled, and played audio; Progress and completion metrics develops the corresponding serving principle. Finally, no finite startup buffer can sustain indefinitely a producer that generates less than one second of playable audio per second.
The room enters the microphone
A microphone receives the acoustic environment, not an isolated user utterance. Reverberation is sound persisting through reflections; distance changes the balance of direct and reflected energy. Background noise adds other signals. Channel bandwidth limits which frequencies survive. Clipping occurs when amplitude exceeds the representable range. Gain control changes level, while noise suppression tries to reduce unwanted sound; neither can restore information already destroyed by clipping.
Loudspeaker output creates another input: the system's own speech travels through the room and returns to the microphone. Acoustic echo cancellation (AEC) uses the playback signal as a reference and estimates how that signal was transformed on its acoustic path. Subtracting the estimated echo aims to preserve local speech. Generated text is not an adequate reference; the canceller needs the rendered signal and its timing relationship to capture.
Double-talk occurs when local speech and returning playback coexist. Local speech can disrupt an adaptive filter's learning, so echo-cancellation methods control adaptation during such interference. Freezing adaptation does not mean muting the microphone or stopping subtraction with the existing filter. Muting capture whenever the assistant speaks prevents ordinary spoken interruption. The engineering objective is to preserve that new contribution while reducing the returning assistant audio, not to remove everything during overlap.
More aggressive preprocessing is not always better. WebRTC's noise-suppression interface explicitly describes stronger suppression as trading less noise for more speech distortion. Test the combined capture path, particularly short acknowledgments and speech onsets, rather than assuming that cleaner-sounding output contains better recognition evidence.
Speaker diarization estimates who spoke when using consistent, usually anonymous labels. It is not echo removal or authentication. Joining diarization with recognition is not a trivial timestamp lookup: systems can disagree about boundaries, detect different speech, or find two speakers where recognition returned only one word. Preserve uncertainty in attribution. Similarly, missing packets are not observed silence; Absence and delay are distinct states explains why unavailable audio cannot establish that nobody spoke.
Conversational timing
Speech activity is not turn completion
Voice activity detection (VAD) estimates whether audio contains speech. Endpointing decides when an utterance or turn is complete enough to advance. These are different decisions: no speech now does not imply no more speech intended. A person can pause to think, retrieve a detail, or continue a difficult word. Recognition finalization also cannot settle this question because its segments need not coincide with conversational intentions.
A detector turns scores into events through stateful rules. Silero's VAD iterator uses one activation threshold and a lower release threshold, plus a minimum silence interval. This hysteresis reduces rapid switching near a boundary. Padding moves reported boundaries to include surrounding audio; the caller must retain the earlier samples. These rules stabilize speech segmentation, not semantic completion.
Semantic endpointing adds an estimate informed by meaning and sometimes acoustic delivery. An equal-length pause after “The destination is…” and after “The destination is Sydney” supplies the same silence duration but different completion evidence. Voice Activity Projection, introduced by Erik Ekstedt and Gabriel Skantze in 2022, predicts both speakers' future activity, providing a way to distinguish holding, shifting, and brief acknowledgments. Such predictions remain fallible and depend on conversational data.
Waiting has a real benefit and a real cost. Colin Lea and colleagues' CHI 2023 study of people who stutter evaluated 5,370 utterances from 41 participants with moderate or severe stuttering. A more patient threshold, tuned on separate data, reduced premature cutoffs while increasing the delay before the recognizer stopped listening. These are endpoint delays in the studied quiet-condition setup, not complete assistant-response times.
| Policy | Premature cutoff rate | Median endpoint delay |
|---|---|---|
| Baseline | 23.8% | 450 ms |
| Threshold tuned for mild stuttering | 4.9% | 1,670 ms |
Completion estimates need an application waiting policy. An unfinished sentence can remain unfinished because the microphone is muted or the person has stopped trying. Combine patience with a bounded, contextual check-in rather than an indefinite wait or a forced answer. The duration and wording should fit the task and user controls; a provider's turn event may reveal its decision without explaining the reasoning behind it.
Equal silence, different completion evidence
Illustrative policy fixture: compare both after 600 ms of silence; unresolved input reaches a check-in at 1600 ms. These are not recommended timeouts or calibrated predictions.
A check-in asks for the missing destination. Response preparation can begin earlier; permission to speak is a separate decision.
Acknowledging without taking the floor
The conversational floor is the opportunity to make the current contribution. A backchannel is a brief listener acknowledgment that need not claim it. Consequently, detecting overlap is not enough to choose a response. A short yeah can signal continued attention, agreement, or the beginning of a correction. Duration is useful evidence but not an intent label.
A transition-relevance place is a possible completion at which speaker change can occur. The earlier turn-allocation account explains why a person may continue when nobody else takes the floor. A voice policy therefore needs more than answer versus silence: it may keep listening, acknowledge without expanding, begin a response, or yield to a substantive interruption. In mixed-initiative dialogue, either participant can redirect the exchange rather than following only system-selected questions.
Human timing helps explain expectations, but it is not a universal machine deadline. A 2009 study across ten languages examined responses to questions ordinarily inviting yes or no. The most frequent response timing fell between zero and 200 ms, while averages differed across languages and response types. Those observations do not cover every task, speaking style, or access need.
Choose interruption handling separately from completion detection. LiveKit's documented policies distinguish activity-triggered interruption, adaptive handling intended to recognize backchannels, and false-interruption recovery. One recovery heuristic resumes an answer when detected speech is followed by no transcription. An empty transcript is not proof that nothing meaningful happened, so test this behavior on the intended speech population.
Acknowledgments should claim only what the system knows. A brief indication of listening is different from confirming a correctly interpreted identifier or announcing completed work. Continuous listening does not require constant speech; unnecessary acknowledgments can themselves seize attention or interrupt delivery.
Interruption and continuity
Stop obsolete speech
Barge-in lets user speech interrupt ongoing system output. The distinction between detecting sound and accepting an interruption is longstanding: the March 2004 VoiceXML 2.0 Recommendation distinguished stopping on detected speech from waiting for a complete match to an active grammar. Whichever policy accepts the interruption, stopping the output requires control over more than the language model.
The playback owner must suppress current sound and remove obsolete queued audio. Generation and synthesis should receive cancellation requests, but callbacks may still arrive. Give each response an identity and reject output whose identity is no longer admissible. At the same time, preserve new user input, including the audio preceding a delayed speech-start event. A short pre-roll buffer serves that last obligation.
Pipecat's interruption contract illustrates the coordination: an interruption bypasses ordinary queued processing, cancels current work, and discards interruptible frames. Some work can remain uninterruptible; function-call cancellation is configurable. Its TTS context APIs also check context identity when accepting later audio. These are concrete local controls, not proof of remote-provider cancellation or immediate silence at every device.
Illustrative pseudocode
Python-like pseudocodeThis guard prevents old callbacks from being accepted again; it does not clear audio already held by the device. Likewise, forwarding an interruption event is not an acknowledgment that all downstream buffers have cleared. Distinguish a cancellation request from quiescence, when the covered work has actually stopped, using Define interruption and cancellation honestly.
External effects have another lifecycle. Interrupting a spoken booking confirmation does not undo a booking; a timed-out request may already have committed. Resolve the operation using its authoritative state and supported retry identity rather than replaying it because the audio stopped. Recover when the effect is unknown teaches that boundary.
Retain only supported playback history
Generation commonly gets ahead of playback. If the user interrupts halfway through a response, retaining the complete generated message as delivered history can make the next answer assume the person received an explanation they never heard. The relevant boundary is output progress, not generation completion. Even completed playback does not establish attention or comprehension.
| Contract | Available boundary | Limit |
|---|---|---|
| Realtime WebSocket truncation | Client-accounted played duration identifies an audio endpoint | The guide does not provide precise transcript-to-audio alignment or a precise shortened transcript |
| Pipecat with supported TTS word timing | Output-synchronized text frames can update assistant context incrementally | Precision depends on provider timing and aggregator placement; it does not prove hearing |
| Twilio playback marks | A normally completed mark identifies preceding playout completion | Clear also returns outstanding marks, so a returned mark alone does not prove playback |
For Twilio Media Streams, maintain the ordered media and mark records together with the clear lifecycle. Treat marks outstanding during clearing conservatively. A partly played chunk may have no exact word boundary available. Do not manufacture one by taking a percentage of the generated text.
Stop response A without inventing delivered history
A conceptual chunk-accounting example across application responsibilities, not a combined provider protocol. External action state is outside this output-control boundary.
| Lane | Before interruption | Interruption accepted | Later observations |
|---|---|---|---|
| Incoming capture | Capture stays open; retain pre-roll. | Preserve the new user contribution and its onset audio. | Input survives response A invalidation. |
| Response production | Response A generation runs ahead of playback. | Invalidate A, then request generation/TTS cancellation. | Late A4 → identity guard → discard. A4 cannot enter playback. |
| Playback owner | A1 complete; A2 playing; A3 queued. | Clear queued A3 and separately request stopping current device output. | Observe device cessation separately; a cancel request is not proven silence. |
| History assembly | Generated text may exceed played content. | Retain ordered chunk/mark records and the clear lifecycle. | Use the media-status ledger; do not claim the whole generated answer was delivered. |
| Response A media | Status after interruption | Supported next-history claim |
|---|---|---|
| A1 | Normally completed | Supported playout boundary; not attention or comprehension. |
| A2 | Partly played | Keep uncertainty about exact word coverage without alignment. |
| A3 | Pending → cleared | Exclude from delivered history; a mark returned during clear is not normal completion. |
| A4 | Late → discarded | Failed the invalidated-response guard; never re-entered playback. |
Serialize invalidation with callback acceptance. The guard protects future acceptance; it cannot remove samples already held by a device. Never convert played-audio percentage into a text prefix.
Keep generated content available for diagnosis if permitted, but distinguish it from content supported as delivered in the next model input. Where delivery is uncertain, a targeted repeat or confirmation is more honest than assuming the whole suffix arrived. Conversation history is a representation assembled for use, not complete conversation truth; State beyond the prompt develops that distinction.
Audible performance
Measure onset, continuity, and stopping
Latency needs a start event, an end event, and an observation location. First generated audio is useful for diagnosing synthesis, but the listener may still wait for transport, buffering, and device output. A system can begin quickly and then stall, or finish generating while a long queue continues playing. Measure those outcomes separately.
| Measurement | Boundary to record |
|---|---|
| Audible response onset | End of the relevant user contribution → first audible response |
| Synthesis startup | Accepted synthesis input → first generated audio chunk |
| Interruption stopping | Onset of the accepted interruption → cessation of obsolete playback |
| Playback continuity | Unintended gaps or concealed intervals during the response |
| Useful answer completion | User contribution end → delivery of the required answer content |
Some tools report RTFX, audio duration divided by computation time. It is the reciprocal only when workload and accounting interval match. Concurrent aggregate throughput must not be mistaken for one stream's sustainable cadence. Riva's performance documentation accordingly separates first-chunk latency, later chunk intervals, and throughput.
Streaming stages overlap. Recognition can run during speech, and synthesis can begin before all response text is complete. The critical path is the dependency chain determining the event of interest; summing every stage duration overcounts concurrent work. In a hypothetical schedule, speech ends at 600 ms, a usable response prefix arrives at 1,100 ms, the first generated audio at 1,200 ms, and playback starts at 1,300 ms. The 700 ms interval to audible onset includes the dependencies and delivery waiting. Latency has boundaries and Attribute elapsed time provide the general measurement framework.
Audible onset is not summed model time
Record distributions under the intended load, devices, networks, and conversation lengths. A 95th percentile describes a boundary below which 95% of the measured values fall; it exposes slow turns hidden by a median. Device latency remains relevant after generation: AudioContext.outputLatency is an estimate, not an acoustic measurement at the ear. Deployment placement also matters—several distant inference round trips can cost more than one long media path with nearby processing stages.
Prepare early without claiming completion
Response preparation can start before speaking is appropriate. LiveKit's preemptive generation can prepare language-model output before turn completion; synthesis waits by default unless separately enabled. If the completion hook changes context or tools, the prepared response is discarded. The benefit depends on its assumptions surviving. Long dictation and revisable input can turn preparation into wasted work.
A tool call is a structured request to external software, not evidence that the requested effect occurred; see A call is an envelope, not an effect. For a slow lookup, the agent can briefly announce what it is doing while the request runs. Longer operations can return a task identifier and expose a separate status operation. The announcement reduces unexplained silence, not backend execution time.
| Known state | Appropriate feedback |
|---|---|
| Possible answer prepared | No claim that the user finished or that the answer is valid |
| Lookup started | A brief description of the lookup |
| Result received | An answer supported by that result |
| External action confirmed | A completion statement matching the confirmation |
An early sound is not the same as an early completed answer. In a 2010 Swedish bargaining-system study, ten participants used incremental and nonincremental versions in counterbalanced order, with a human supplying transcriptions. Incremental generation reduced mean response-onset delay from 2.84 to 0.58 seconds, but completion delay changed from 5.66 to 5.02 seconds. Some ratings improved; overall preference did not differ significantly. This isolates a useful distinction without establishing current end-to-end agent performance.
Output checks also compete for the response budget. A policy violation detected after speech has played cannot retract that speech. Place required checks before the relevant audible commitment, and measure the added waiting. Streaming is a choice about commitment granularity, not permission to postpone every validation until the end.
Reconstruct the audible interaction
A trace correlates operations; an event records an occurrence. Their general structure is covered in Traces, spans, events, and links. For voice, correlate them with media and response identities. A transcript plus successful server calls cannot explain whether the user was cut off, an obsolete chunk played, or a delegated model received incomplete context.
| Field group | Purpose |
|---|---|
| Session, stream, turn, response, chunk IDs | Connect audio, hypotheses, generated output, and cancellation |
| Event type and revision | Distinguish proposed text, final text, policy decisions, and later corrections |
| Observation location and clock | Identify capture, server, transport, or playback timing |
| Media position and format | Map an event to the corresponding source interval |
| Queue or playback status | Separate generated, pending, cleared, and normally completed output |
| Protected content reference and availability | Permit authorized inspection without treating omitted data as empty data |
Do not compare unrelated clock values as though they share an origin. RTP sampling timestamps, packet arrival times, server clocks, and device presentation times describe different coordinates. The accompanying RTP Control Protocol (RTCP) supplies sender reports that pair media timestamps with a shared reference clock. Those pairs help align streams, but do not measure actual device output. Record the mapping and its uncertainty. Where only an output-latency estimate exists, label the resulting playback time as estimated.
Investigate the first consequential divergence. Compare captured audio with recognition, then recognition with the turn decision, the decision with generated output, and output with playback. Include tool arguments, results, and delegated context when they matter. Permissioned audio replay can reveal timing that text hides, but recordings, transcripts, and derived speaker representations need separate purposes, access rules, and retention. Govern telemetry as sensitive data covers those responsibilities.
Understandable and controllable exchanges
Repair the misunderstanding
Conversational repair resolves a breakdown in mutual understanding. It starts by identifying what is uncertain: the requested operation, a particular detail, or whether the answer was received. Establishing enough mutual understanding to proceed is called grounding. To preserve progress during repair, distinguish the transcript from dialogue state, the interpreted constraints accumulated across exchanges. That state includes the intent, or requested operation, and its slots, arguments such as destination and date. Workflow state separately records what was executed.
| Breakdown | Repair |
|---|---|
| Missing intent or argument | Ask for the specific missing information |
| Possibly misrecognized detail | Confirm the candidate interpretation |
| Content may not have played | Repeat or summarize the relevant portion |
| The response was wrong | Correct it explicitly and update dependent state |
Explicit confirmation asks the person to verify an interpretation before proceeding. Implicit confirmation incorporates it into the next contribution, leaving room for correction. Use stronger confirmation when a wrong detail would have greater consequences. For example, if only a place name is uncertain, ask “Did you say Newark?” rather than requesting the entire journey again. Checking that interpretation does not itself authorize an external action.
Spoken answers should expose a useful next step before demanding extensive listening. Group choices, pause at meaningful boundaries, and leave room for correction. If clarification repeatedly fails, offer text, another input method, or human help. After connection loss, recover from supported dialogue and workflow state: uncertainty about what played should remain separate from uncertainty about what an external service did.
Make listening and stopping controllable
People need to know whether the system is receiving sound, preparing a reply, or still playing one. Apply Separate receipt, progress, and outcome to those actual states. Mute affects input; stop affects output; cancelling work and deleting a recording are different operations. One indicator cannot truthfully stand for all of them.
| State | Truthful cue | Relevant control |
|---|---|---|
| Microphone input enabled | Microphone enabled | Mute or an explicit input gate |
| Response preparation running | Preparing a response | Cancel or redirect |
| Audio queued or playing | Speaking or stopping | Stop playback and clear pending speech |
| Playback cessation observed | Playback stopped | Repeat or continue |
| Recording retained | Recording and retention status | Applicable retention and deletion controls |
Push-to-talk supplies an explicit input boundary: press to begin and release to finish. It can reduce ambiguity when a button is practical, but it neither reverses external work nor removes the need to stop existing playback. The 1993 VoiceNotes prototype combined speech and buttons for different circumstances. Its formative testing also exposed ambiguous acknowledgments: selecting a category and asking one's location initially sounded the same. Distinct feedback made the state change clearer.
Access needs affect timing and recovery. The W3C's 2026 draft guidance for voice interfaces recommends adjustable timing and speech rate, manageable choices, and alternatives when speech fails. Some people need longer to respond; silence must not automatically mean rejection. Provide repeat, captions or a conversation record where appropriate, text input, and human alternatives. Keep audible cues brief enough not to compete with speech. These are design considerations, not a conformance claim; Accessibility is interaction behavior develops the broader obligation.
Live microphone processing does not establish that audio is stored, and a microphone indicator does not establish permission for every later use. Explain collection and retention separately, following Establish permitted uses.
Evaluation and design choices
Measure speech properties separately
Component evaluation should identify what changed without claiming more than it measured. Recognition fidelity, speaker attribution, pronunciation, naturalness, and timing need different references. Follow Choose checks that match the requirement, then connect those checks to the voice-specific failure.
Specify normalization and tokenization before scoring. Removing punctuation, standardizing written numbers, or changing word segmentation changes the comparison. WER also counts word disagreements rather than consequences. In the invented reference “Please send fifteen boxes,” replacing Please with Kindly and replacing fifteen with fifty each creates one substitution out of four words—25% WER—but only the latter changes quantity. Track critical details separately.
Mean opinion score (MOS) averages human ratings under a stated protocol; it is not a universal objective unit. Speech material, listeners, equipment, noise, instructions, and rating scale affect the result. ITU-T P.800 separates listening quality from effort and loudness preference. For synthesis, test intelligibility and intended words alongside naturalness: pleasant audio can still pronounce the wrong name or omit content.
Disaggregate recognition by relevant speakers and conditions. Allison Koenecke and colleagues' 2020 study evaluated five commercial services on interview speech from 73 Black and 42 white speakers. Average WER across those services was 35% and 19%, respectively. These historical measurements are not current rankings or an inherent difference in recognizability; corpus and geographic differences constrain interpretation. They demonstrate why an aggregate alone is insufficient.
| Target | Required observation | What it can miss |
|---|---|---|
| Premature endpointing | Whether listening stopped before the reference contribution ended | Words misrecognized after complete capture |
| Interruption handling | Accepted interruption, obsolete playback cessation, and subsequent response | An answer-latency score alone misses stopping delay |
| Backchannel handling | Whether acknowledgment preserves the ongoing contribution | Short duration alone does not establish intent |
| Diarization | Speaker timelines, including overlap | Correct words assigned to the wrong participant |
| Response latency | Named onset boundary plus nonresponse count | A conditional percentile can hide turns with no answer |
Full-Duplex-Bench separates pauses, backchannels, transitions, and interruptions using timed input and output. Its response latency is conditional on taking a turn, so nonresponses require a separate rate. Post-interruption response latency is also not playback-stop latency. Preserve these distinctions when adapting a benchmark to a product.
Test complete conversations
Component gains matter when they improve the intended exchange. The PARADISE framework, introduced in 1997, separated task success, dialogue costs, and user satisfaction. A corrected misunderstanding adds repair cost; an unresolved one can prevent success. Its illustrative framework does not prescribe universal metric weights or imply that fewer turns are always better.
| Method | What it establishes | What needs another test |
|---|---|---|
| Recorded-audio replay | Behavior on the same words, pauses, and acoustic conditions | How a person changes behavior in response to the system |
| Timed media fault tests | Responses to scheduled network changes, echo, and playback disturbances | Correct interpretation and useful dialogue |
| Dialogue simulation | Coverage of specified dialogue patterns under the simulator | Unmodeled acoustic failures and human adaptation |
| Representative participant studies | Task success, effort, recovery, and perceived control during interaction | Broader populations and conditions not included |
Timed integration tests should observe output, not merely successful calls. WebRTC's media testing framework can schedule network changes and export captured or rendered audio with concealment and buffer statistics. Add application assertions: an invalidated response must not re-enter playback, new user audio must survive interruption, and a clear event must not become a false delivery claim. These assertions test the coordination between otherwise functioning components.
Validate simulations against the behavior they are meant to represent. Hua Ai and Fuliang Weng's 2008 restaurant-dialogue study related dialogue properties to satisfaction from 20 participants, then used that relationship with simulated dialogues. Predicted and observed satisfaction did not differ significantly, but the simulator used the human corpus and bypassed recognition and synthesis. The result was not independent end-to-end validation of spoken interaction.
Compare policies on matched tasks with the intended participants and devices. Include completion, critical-detail correctness, repair success, repeated effort, unwanted overlap, abandonment, and control. A controlled human-conversation study found greater sensitivity to delay in more interactive tasks, reinforcing that acceptable timing depends on the activity. Evaluate use, reliance, and recovery develops the complete-workflow perspective.
Finish with a decision record, not a single quality score: which information requires an audio-preserving path; which words need confirmation; what evidence ends a turn; how much audio may queue; who owns stopping; how uncertain delivery is repaired; and which participant and fault tests support those choices. More patient endpointing may be worth extra waiting. A smaller buffer may be worth greater underrun risk. The justified choice is the one whose combined behavior serves the intended conversation.
Open questions
Adaptive turn policies must become more patient without creating persistent unexplained silence. The difficulty is distinguishing hesitation, atypical speech, acknowledgment, and abandoned input across people and tasks. Progress would mean fewer cutoffs and successful bounded recovery, measured together with added waiting in representative conversations.
Full-duplex models must combine fluid overlap with dependable role and task behavior. Timing competence and instruction-following are different targets, and convincing demonstrations do not establish their joint reliability. Progress would require matched interactive evaluations that retain both conversational and task outcomes.
Audio-preserving models need evidence that they use vocal cues appropriately rather than merely receiving them. Delivery depends on context, and factual-answer training can ignore it. Useful progress would demonstrate task-relevant sensitivity to controlled changes in delivery without claiming certainty about a speaker's internal state.
Playback history remains limited by transport and alignment granularity. Systems can know that some audio completed or was cleared without knowing the exact interrupted words. Better contracts would expose compatible output identities and timing uncertainty while preserving the distinction between playout and comprehension.
Replay and simulation need stronger links to human adaptation. A fixed speaker recording cannot slow down, repeat, abandon, or learn a system's timing habits. Progress would establish which replay improvements predict reduced effort and better completion in independent live interactions.
