Skip to main content
Blog

Who Said That? Online Speaker Diarization With No Enrollment

Labelling speakers in a live meeting without voiceprints, registration, or knowing how many people are in the room — 256-dimensional embeddings, one cosine comparison, and a threshold that is a trade rather than a truth.

· Dev3lop Team

Speaker diarization flow: a finished clip embedded by a wespeaker resnet34 model into a 256-dimensional L2-normalized vector, compared by cosine similarity to each live centroid, with the best match at 0.71 clearing a 0.50 threshold and adapting its centroid

A transcript without speaker labels is a wall. A transcript with speaker labels is a meeting. That difference is worth more than a couple of points of word error rate, and it is the single feature people notice first.

The textbook way to get it is enrollment: everyone records a sample, you store a voiceprint, you match against the register. That approach is a non-starter for the two situations that matter most. Nobody enrolls before a client call. And storing biometric voiceprints for a room full of people is exactly the kind of data you do not want to be holding when somebody asks what you keep.

So: no enrollment, no stored voiceprints, no idea how many people are in the room. Label them anyway, live, while the transcription queue is already running. This is part four of a series on the local speech stack.

One vector per turn

The hero diagram is the loop, and it runs once per finished clip.

The same audio that goes to Whisper also goes to a speaker-embedding model — wespeaker-voxceleb-resnet34-LM, as ONNX, quantized to q8, running on the WASM backend. That last choice is deliberate: WebGPU is left free for Whisper, because the transcription model is the one whose latency the user can feel.

The model returns a 256-dimensional vector in last_hidden_state, L2-normalized in the worker before it goes anywhere. Normalizing up front is not tidiness — it means cosine similarity is literally a dot product, so comparing a turn against every known speaker is one tight loop over a handful of short arrays. There is no vector index here and no vector database. At meeting scale you are comparing against three to eight centroids; anything fancier would be slower.

Then the decision: find the highest cosine similarity to an existing speaker centroid, and if it clears the threshold, that’s who spoke. Fold the new embedding into that centroid with an exponential moving average — centroid = L2(0.8 × centroid + 0.2 × embedding) — and re-normalize. Otherwise, spawn a new speaker.

That EMA is doing more work than it looks like. Matching and adapting are the same step, and they have to be, because a voice at minute ninety of a meeting is not the voice from minute one. People get tired, lean back from the microphone, put on a headset, get louder when they disagree. A fixed centroid captured at first utterance drifts out of range by the second hour; one that follows at α = 0.2 keeps up without being yanked around by a single odd clip.

The clusterer is five branches

The speaker assignment decision ladder: clips under 700 milliseconds attach without spawning or moving a centroid, an empty clusterer seeds Speaker 1, a cosine at or above 0.50 assigns and adapts, a fixed speaker count force-assigns to the nearest, and otherwise a new speaker is spawned

Every incoming turn walks this ladder and the first branch that matches wins. Two of the five rules exist entirely because of failures we watched happen.

The 700 ms reliability floor. Short clips produce noisy embeddings — there simply isn’t enough voiced audio to characterize a voice. Without a floor, every "mm-hm", "right" and half-laugh gets an unstable vector, misses every centroid, and spawns a brand-new speaker. You end up with a four-person meeting containing fourteen speakers, most of whom said one syllable. So clips under 700 ms attach to the nearest existing speaker and are forbidden from doing anything else: they never spawn, and they never move a centroid. The number is not arbitrary — it’s the same 700 ms as the segmenter’s silence hang, which means a normal conversational turn always clears it.

The bootstrap exception. If there are no speakers yet, even a short clip seeds Speaker 1. Otherwise a meeting that opens with someone saying “hi” loses its first cluster and the labels start one turn late.

The remaining branches are the ordinary path (assign and adapt), the user override (if somebody has said “there are three people in this room”, force-assign to the nearest rather than inventing a fourth), and the default: spawn Speaker N+1. Speaker count is an output of the meeting, not an input.

Labels are Speaker 1, Speaker 2, in order of first appearance, and they are renameable and mergeable afterwards. A merge is count-weighted: fold the source centroid into the target proportionally to how many turns each has seen, then re-normalize. Which means correcting the machine also teaches it, for the rest of that meeting.

The threshold is a trade, not a truth

Schematic of two overlapping similarity distributions — same speaker and different speakers — with a 0.50 threshold line showing that lowering it merges two people into one label while raising it splits one person into several

This is the part worth internalizing if you build anything with a similarity threshold in it. Speaker embedding distributions overlap. There is no value that separates “same person” from “different people” cleanly, because two colleagues with similar voices on the same microphone genuinely produce similar vectors, and one person across a two-hour meeting genuinely produces varied ones.

So 0.50 is not a discovered truth. It is a choice about which mistake to make:

  • Lower it and you merge. Two quiet colleagues become one person, and the meeting summary attributes a decision to the wrong human.
  • Raise it and you split. One person becomes Speaker 3, Speaker 4 and Speaker 7 within the same meeting, and the transcript reads like a crowd scene.

Which error is worse depends entirely on the meeting. Two people, a quiet room, clean headsets: a high threshold is safe and splits are rare. Eight people around a conference speakerphone: merging is nearly guaranteed at any threshold, and the useful posture is a lower one plus easy manual merge, because a human correcting four labels is fine and a human untangling fourteen is not.

Which is why the threshold is a live slider rather than a constant, and why the figure above is labelled a schematic. It’s the shape of the trade — anyone claiming a universal best value hasn’t run their system in a room with a speakerphone in it.

What honest labels mean downstream

Speaker identity is an inference, and everything built on top has to treat it that way. Three rules follow, and they are less about audio than about not lying to the user:

  1. A label is a cluster, not a person. Speaker 2 means “these turns sound alike.” It does not mean “Priya.” Linking a label to a real contact is an explicit human action, never a model guess.
  2. Centroids aren’t persisted. A recovered session may rediscover its labels rather than reuse them. That’s a real limitation, and the right response is to say so rather than let stale identities silently reattach to the wrong voice.
  3. Nothing downstream may assert identity it didn’t earn. The background analysis pipeline can quote what was said and cite the row it came from; it cannot conclude who committed to what from a speaker label alone.

That’s the same discipline the rest of this stack runs on, and it is the difference between analytics people trust and analytics people quietly stop opening.

Local LLM scheduling lanes: interactive, normal and background priorities sharing one Ollama runtime, with same-model serialization, a concurrency ceiling, and early rejection when three ordinary jobs are already waiting

One post left. The transcript exists, the speakers are labelled, and now a 7B model wants to read it — on the same laptop that is still holding the microphone open. Scheduling that is where the whole stack either stays usable or falls over.

Series: the whole stack · the speech gate · the durable queue · this post · local LLM scheduling

← All posts