Voice agents, part 2 · 5 min read
How a voice agent knows you stop talking
The answer is a voice activity detector. Here is what it hears, three ways to build one, and a recording where you can see which ones would have talked over me.
The answer: a voice activity detector
A voice activity detector, VAD for short, answers one question about thirty times a second: is someone speaking right now? Yes or no. That is all it does, and everything about turn-taking is built on it.
From that yes/no stream the agent gets:
- When speech starts and stops, to within about 30 milliseconds.
- How long each pause is, which is what the silence timeout counts.
- How much of the call is actually speech, so it can skip sending silence to the transcriber.
What it does not get is words. A VAD cannot tell "I'd like a table for… four" from "I'd like a table for four." Both are sound, then silence.
Three ways to build one
| Approach | How it works | Good | Bad | Verdict |
|---|---|---|---|---|
| Energy threshold | Speech if the last 30 ms is louder than the room by some margin. | Free, zero latency, runs anywhere. | Breaths, taps and keyboards count as speech. Quiet endings count as silence. | Prototype only |
| WebRTC VAD | Google's classic detector from the WebRTC stack: a small statistical model over six frequency bands. | Tiny, fast, in every browser and most telephony stacks. | Tuned for phone calls. On a laptop mic with room noise it hears speech almost everywhere. | Fallback |
| Silero VAD | A neural net of about 2 MB that outputs a speech probability every 32 ms. | Ignores breaths and taps, catches quiet words, runs in under 1 ms per chunk on a CPU. | Still only hears sound. It cannot tell a thinking pause from a finished sentence. | Default in 2026 |
The layer above: semantic turn detectors
A VAD hears sound. A turn detector reads the transcript so far and predicts whether the sentence is finished, so "…and" or "around…" can hold the floor through a long pause. LiveKit's turn detector and Pipecat's Smart Turn are open examples. It sits on top of a VAD rather than replacing it, and it needs a streaming transcript first, so it adds a hop. It is not run on the clip below.
Which one talks over you
Same clip, all three detectors. The agent speaks as soon as a detector has reported silence for longer than the timeout. Move the slider and watch the markers.
- Red marker: the detector went quiet long enough while I was only pausing. The agent would have interrupted me there.
- Green marker: it fired after my last word. That gap is the reply delay.
- Shorter timeout: more red. Longer timeout: green moves right, the agent feels slow.
The timeout is the real dial. At 300 ms even Silero interrupts six times. At 1,200 ms it never interrupts and replies 1.2 seconds late. The detector decides how honest the silence is. WebRTC hears speech in the room noise, so it only fires once the recording ends. The energy gate fires on a breath before the first word. Silero's silences are real, which is exactly why it can interrupt a thinking pause.
Why WebRTC VAD lit up the whole clip
WebRTC VAD was built for narrowband telephone audio and decides on six frequency bands with a Gaussian mixture model. Even at its strictest setting it treats the broadband hiss of a laptop microphone as possible speech, so on this recording it reports speech for almost every frame. On a real phone line with echo cancellation and a proper noise gate it behaves much better, which is why telephony stacks still ship it as a cheap first pass.
What Silero got right and where it still fails
Silero cleanly separated every word group, dropped the table tap to a single 30 ms blip, and kept the "umm" as speech, which is correct: a filler means the caller is holding the floor. Its failure is structural, not a bug. After "book a table for four" the caller says nothing but "umm" for 1.5 seconds, and no amount of listening to sound can know that "actually, make it five" is coming. The words do not help either: the sentence was complete. That case is solved by letting the caller barge in and revise, which is part 3.
What production stacks do
The common 2026 recipe is Silero for sound, a short timeout of roughly 200 to 300 milliseconds, and a semantic turn model on top that extends the wait when the transcript looks unfinished. Deepgram and AssemblyAI ship similar logic inside their streaming endpoints. The semantic model would have held the floor after "and" and "around" in this clip. It would not have saved "four".
This series
- Anatomy and metrics. Where the second of latency and the ten cents a minute go.
- End-of-turn detection. This piece.
- Interruptions. Barge-in with real audio: two cut-ins, three stop policies.
- Why it sounds robotic. Streaming, chunking, and prosody.
- When it hears wrong. Transcription errors and how they cascade.
- Speech-to-speech models. Does the pipeline go away?