Give Whisper three seconds of an air conditioner and it will not return an empty string. It will return "Thank you." Or "Thanks for watching!" Or the same four words eleven times. The model was trained to produce text from audio; hand it audio with no text in it and it produces text anyway, confidently, with no signal that anything went wrong.
In a meeting recorder that failure is not cosmetic. Hallucinated lines get speaker labels, land in the transcript, and get fed to a summarizer that dutifully treats them as things somebody said. One phantom “Thanks for watching” in a client call is embarrassing. A hundred of them across a two-hour recording is a product nobody trusts.
This is part two of a series on the local speech stack. It is about the layer everybody skips: deciding what the model should never be asked about in the first place.
Energy detection is not voice activity detection
The hero diagram draws the distinction the whole design rests on. There are two gates, and they answer different questions at wildly different costs.
Stage one is an energy segmenter, running on every 64 ms block, and its question is has this clip ended? It is pure arithmetic — root-mean-square of 1024 samples — which is what lets it run on the real-time audio thread where allocating memory is already too expensive. Four numbers define it: a silence floor at RMS 0.008, a 700 ms silence hang before flushing, a 400 ms minimum clip length, and a 14-second forced flush during continuous speech.
Stage two is a neural gate — Silero VAD, running once per finished clip in its own worker — and its question is was that speech at all? Energy cannot answer this. A slammed door has enormous energy. A keyboard has rhythmic energy. A dog, a fan ramping up, a chair scraping: all loud, none of them speech. RMS says “loud enough, transcribe it” and the model obliges with fiction.
The gate is deliberately permissive — minSpeechFrames: 2, positive threshold 0.5, negative 0.35, redemption 8 frames. Its job is to drop clips that are clearly not speech, not to trim quiet speech. And when it errors, it fails open: the clip is kept and Whisper decides. That direction is chosen on purpose. A broken speech gate that silently swallows real audio produces a transcript with holes nobody can see, which is a far worse failure than a stray line of noise text that a human can delete.
This is stage one doing its job across one real conversational turn. Blue blocks cleared the threshold; grey ones didn’t. Three details in that picture took the longest to get right:
The hang has to be generous. Set it to 300 ms and every thinking pause cuts a sentence in half, which wrecks both the transcription (Whisper uses context) and the speaker embedding (short clips are noisy). 700 ms is long enough to survive “so the thing is… pause …we’re behind.”
The trailing silence goes in the clip. Once speech has started, silent blocks keep accumulating until the flush fires — they are not trimmed. Cutting at the last loud block clips word endings, and Whisper’s guesses at a truncated final word are exactly the kind of plausible wrongness that’s hard to spot on review.
Blips under 400 ms are never sent. A cough, a mouse click, one syllable of crosstalk. They cost a model call and return garbage, so they don’t get one.
And the 14-second cap exists for one reason: someone who monologues for four minutes without a 700 ms pause would otherwise hold the entire queue hostage behind one enormous clip.
Hallucination is a gap at three layers
Even with a good gate, some noise gets through, and some genuinely quiet speech decodes badly. So the defense is layered.
Before decode — don’t send it. Everything above. The cheapest way to not get a hallucination is to not make the call.
During decode — don’t let it loop. temperature: 0, no_repeat_ngram_size: 3, repetition_penalty: 1.2. Greedy decoding removes the sampling randomness that starts a loop; the n-gram block is what actually breaks the “thank you thank you thank you” cascade once it starts. Also chunk_length_s: 30 and timestamps off, because timestamps on noise are a second thing to be wrong about.
After decode — don’t save it. A text filter drops known non-speech annotations: musical-note-wrapped strings, [BLANK_AUDIO]-style all-caps markers, and bracketed sound tags from a fixed list — laughter, cough, applause, crosstalk, keyboard, and friends. Then a word histogram: if the output has at least eight words and any single word is more than 40% of them, the whole thing is discarded as junk.
The hard part of that last layer is narrowness. Real short speech looks a lot like a sound tag. "(yeah)" is a person agreeing. "[pause] right" is a person thinking. A filter written with a generous regex eats both. So the tag list is explicit and the all-caps rule only fires on a whole-string match — which is the difference between a filter and a shredder.
What this buys you downstream
A transcript that only contains speech is worth more than a transcript with better word error rate, because everything downstream inherits its assumptions.
The speaker clusterer assigns an identity to every clip that reaches it. Feed it door slams and it will happily discover Speaker 4, Speaker 5 and Speaker 6 — three people who do not exist, now attached to phantom lines in the meeting summary. The durable audio queue pays disk and model time for every clip; gating cheaply up front is a straight throughput win. And background language-model analysis is required to quote the transcript verbatim, which means noise lines are not just noise — they become citable evidence for a finding that is entirely imaginary.
That is the next piece: once you have decided a clip deserves a model, where does it wait? The obvious answer is an array. The obvious answer is wrong in an interesting way.
The tuning advice, condensed
If you are building this and want somewhere to start:
- Tune the hang before the threshold. Most “the transcription is bad” complaints are actually clips being cut mid-sentence. The threshold mostly controls how much silence you pay a model to look at.
- Never let the gate fail closed. Log the error, keep the audio, move on.
- Constrain decoding even when the audio is clean. Repetition penalties cost nothing and cover the case where your gate is wrong.
- Test with the worst five seconds you own — a laptop fan, a sneeze, a door, an empty room with the mic gain high. This is a rate-limiting and back-pressure problem in disguise: every piece of junk you admit costs the same as real work, all the way down the pipeline.
Series: the whole stack · this post · the durable queue · speaker diarization · local LLM scheduling