Voice agents, part 3 · 5 min read
What happens when you interrupt a voice agent
Part 2 ended with a sentence no detector could save: "table for four… actually, make it five." The fix is letting the caller barge in. Here is what that costs, with real audio.
The answer: barge-in
Barge-in is the agent's ability to stop talking when the caller starts. It reuses the voice activity detector from part 2, now pointed at the caller while the agent's own audio is playing. Three things have to happen in order:
- Decide the caller is really talking, not just making listening noises.
- Cut the audio, fast enough that the caller does not hear themselves being talked over.
- Tell the language model what was actually said, because the agent's sentence was never finished.
Step 1 is the hard one, and it is the same trade-off as the silence timeout: react fast and you stop for every "mm-hmm", react slow and you talk over the correction.
Two cut-ins, three policies
The agent reply is synthetic. The caller is me, recorded twice: once cutting in with a correction, once just making listening noises. Pick a policy and press play. Both tracks play together, exactly as the caller would hear it.
Instant stop reacts in about 30 ms and fails the second clip. It cuts the agent off mid-sentence, at "Saturday at eight", for an "mm-hmm". Waiting half a second passes both. The correction stops the agent about half a second after "Sorry", which sounds like a person noticing, and the listening noises never reach half a second, so the agent keeps going. No barge-in fails the first. "Make it seven thirty" arrives on top of "shall I add the high chair now?", and the model gets both at once.
The three policies
| Policy | How it works | Good | Bad | Verdict |
|---|---|---|---|---|
| No barge-in | Microphone is ignored while the agent speaks. The caller waits. | Simple. Nothing to get wrong. | Corrections land late or get lost. Feels like an IVR menu. | Only for announcements |
| Stop on any sound | The moment the VAD hears the caller, cut the audio. | Fastest possible reaction, about 100 ms. | "Mm-hmm", a cough, a door: all of them stop the agent mid-word. | Too twitchy |
| Stop after sustained speech | Cut only when the caller has spoken for roughly half a second without stopping. | Ignores listening noises and coughs. Still reacts inside a second. | Adds the wait to every real interruption. Loud "no!" still needs 500 ms. | The usual default |
The part the demo skips: echo
On a real call the microphone also picks up the agent's own voice from the speaker. Without acoustic echo cancellation the VAD hears the agent and stops it, over and over. Phones and browsers do this cancellation for you; a raw telephony stream often does not, and it is the first thing to check when barge-in "fires on its own". This demo mixes the two tracks digitally, so there is no echo to cancel.
What to tell the language model
When the agent is cut off, the transcript it sends to the model must be truncated to the words actually spoken, not the full planned reply. Otherwise the model believes it asked about the high chair and the conversation drifts. Most frameworks track which text-to-speech chunks have finished playing and trim the assistant turn to that point; the demo's "what the language model sees" box shows the trimmed version, with the half-spoken word marked.
Better than a fixed half second
Production stacks combine the duration rule with words. If the streaming transcript of the interruption starts with "yeah", "okay", "mm-hmm" or "right", treat it as a backchannel and keep talking. If it starts with "no", "wait", "sorry" or "actually", stop immediately. That gets the correction case down to around 300 ms without becoming twitchy. LiveKit, Pipecat and Vapi all expose some version of this, usually as a minimum-words or minimum-duration setting plus an ignore-list.
This series
- Anatomy and metrics. Where the second of latency and the ten cents a minute go.
- End-of-turn detection. Three real detectors on one clip: which ones talk over you.
- Interruptions. This piece.
- Why it sounds robotic. Streaming, chunking, and prosody.
- When it hears wrong. Transcription errors and how they cascade.
- Speech-to-speech models. Does the pipeline go away?