Get Started

Speaker Recognition & Verification

ASR asks "what did they say?" Speaker recognition asks "who said it?" The math looks the same, embeddings plus cosine, but every production decision hinges on a single EER number.

45 minutesBuildPythonFree lesson

The Problem

A user says a passphrase. You want to know: is this the person they claim to be (verification1:1), or is it the first person in your enrollment bank (identification1:N)? Or neither, is this an unknown speaker (open-set)?

Pre-2018: GMM-UBM + i-vectors. Reasonable EER but fragile to channel shift (phone vs laptop) and emotion. 2018 to 2022: x-vectors (TDNN backbone trained with angular margin). 2022+: ECAPA-TDNN and WavLM-large embeddings. By 2026 the field is dominated by three models and one metric.

The metric is EEREqual Error Rate. Set your decision threshold so False Accept Rate = False Reject Rate. The crossover is EER. Used in every paper, every leaderboard, every procurement call.

The Concept

Enrollment + verification pipeline with embedding + cosine + EER

The pipeline. Enrollment: record 5 to 30 seconds of the target speaker; compute a fixed-dimension embedding (192-d for ECAPA-TDNN, 256-d for WavLM-large). Verification: get the test utterance embedding; compute cosine similarity; compare to a threshold.

ECAPA-TDNN (2020, still dominant 2026). Emphasized Channel Attention, Propagation and Aggregation - Time-Delay Neural Network. 1D conv blocks with squeeze-excitation, multi-head attention pooling, followed by a linear layer to 192-d. Trained on VoxCeleb 1+2 (2,700 speakers, 1.1M utterances) with Additive Angular Margin loss (AAM-softmax).

WavLM-SV (2022+). Fine-tune a pretrained WavLM-large SSL backbone with AAM loss. Higher quality but slower, 300+ MB vs 15 MB.

x-vector (baseline). TDNN + statistics pooling. Classic; still useful on CPU / edge.

AAM-softmax. Standard softmax with added margin m in the angular space: cos(θ + m) for the correct class. Forces inter-class angular separation. Typical m=0.2scale s=30.

Scoring

  • Cosine between enrollment and test embeddings. Threshold-based decision.
  • PLDA (Probabilistic LDA). Project embeddings into a latent space where same-speaker vs different-speaker has a closed-form likelihood ratio. Added on top of cosine for +10 to 20% EER reduction. Standard pre-2020; now used only in closed-set setups.
  • Score normalization. S-norm or AS-norm: normalize each score against a cohort of imposter means and stds. Essential for cross-domain eval.

Numbers you should know (2026)

ModelVoxCeleb1-O EERParamsThroughput (A100)
x-vector (classic)3.10%5 M400× RT
ECAPA-TDNN0.87%15 M200× RT
WavLM-SV large0.42%316 M20× RT
Pyannote 3.1 segmentation + embedding0.65%6 M100× RT
ReDimNet (2024)0.39%24 M100× RT

Diarization

"Who spoke when" in a multi-speaker clip. Pipeline: VAD → segment → embed each segment → cluster (agglomerative or spectral) → smooth boundaries. Modern stack: pyannote.audio 3.1, which bundles speaker segmentation + embedding + clustering behind one call. 2026 SOTA DER on AMI is ~15% (down from 23% in 2022).

Build It

Step 1: toy embedding from MFCC statistics

Python
def embed_mfcc_stats(signal, sr):
    frames = featurize_mfcc(signal, sr, n_mfcc=13)
    mean = [sum(f[i] for f in frames) / len(frames) for i in range(13)]
    std = [
        math.sqrt(sum((f[i] - mean[i]) ** 2 for f in frames) / len(frames))
        for i in range(13)
    ]
    return mean + std  # 26-d

Not SOTA by a mile, for teaching only. code/main.py uses this as a proof-of-concept on synthetic speaker data.

Step 2: cosine similarity + threshold

Python
def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    na = math.sqrt(sum(x * x for x in a))
    nb = math.sqrt(sum(x * x for x in b))
    return dot / (na * nb) if na and nb else 0.0

def verify(enroll, test, threshold=0.75):
    return cosine(enroll, test) >= threshold

Step 3: EER from similarity pairs

Python
def eer(same_scores, diff_scores):
    thresholds = sorted(set(same_scores + diff_scores))
    best = (1.0, 1.0, 0.0)  # (fa, fr, threshold)
    for t in thresholds:
        fr = sum(1 for s in same_scores if s < t) / len(same_scores)
        fa = sum(1 for s in diff_scores if s >= t) / len(diff_scores)
        if abs(fa - fr) < abs(best[0] - best[1]):
            best = (fa, fr, t)
    return (best[0] + best[1]) / 2, best[2]

Returns (eer, threshold_at_eer). Report both.

Step 4: production with SpeechBrain

Python
from speechbrain.pretrained import EncoderClassifier

clf = EncoderClassifier.from_hparams(source="speechbrain/spkrec-ecapa-voxceleb")

# enroll: average the embeddings of 3-5 clean samples
enroll = torch.stack([clf.encode_batch(load(x)) for x in enrollment_clips]).mean(0)
# verify
score = clf.similarity(enroll, clf.encode_batch(load("test.wav"))).item()
verdict = score > 0.25   # ECAPA typical threshold; tune on your data

Step 5: diarize with pyannote

Python
from pyannote.audio import Pipeline

pipe = Pipeline.from_pretrained("pyannote/speaker-diarization-3.1")
diarization = pipe("meeting.wav", num_speakers=None)
for turn, _, speaker in diarization.itertracks(yield_label=True):
    print(f"{turn.start:.1f}–{turn.end:.1f}  {speaker}")

Use It

The 2026 stack:

SituationPick
Closed-set 1:1 verification, edgeECAPA-TDNN + cosine threshold
Open-set verification, cloudWavLM-SV + AS-norm
Diarization (meetings, podcasts)pyannote/speaker-diarization-3.1
Anti-spoofing (replay / deepfake detection)AASIST or RawNet2
Tiny embedded (KWS + enrollment)Titanet-Small (NeMo)

Pitfalls

  • Channel mismatch. Model trained on VoxCeleb (web video) ≠ phone-call audio. Always evaluate on target channel.
  • Short utterances. EER degrades sharply below 3 seconds of test audio.
  • Enrollment with noise. One noisy enrollment poisons the anchor. Use ≥3 clean samples and average.
  • Fixed threshold across conditions. Always tune the threshold on a held-out dev set from the target domain.
  • Cosine on non-normalized embeddings. L2-normalize first; otherwise the magnitude dominates.

Ship It

Save as outputs/skill-speaker-verifier.md. Pick model, enrollment protocol, threshold-tuning plan, and fraud safeguards.

Exercises

  1. Easy. Run code/main.py. Builds synthetic "speakers" (different tone profiles), enrolls, computes EER on a 100-pair trial list.
  2. Medium. Use SpeechBrain ECAPA on 30 VoxCeleb1 utterances (5 speakers × 6 each). Compute EER with cosine vs PLDA.
  3. Hard. Build the full enroll → diarize → verify pipeline with pyannote.audio. Evaluate DER on AMI dev set.

Key Terms

TermWhat people sayWhat it actually means
EERThe headline metricThreshold where False Accept = False Reject.
Verification1:1"Is this Alice?"
Identification1:N"Who is speaking?"
Open-setUnknown possibleTest set can contain unenrolled speakers.
EnrollmentRegisteringComputing a speaker's reference embedding.
AAM-softmaxThe lossSoftmax with additive angular margin; forces cluster separation.
PLDAClassic scoringProbabilistic LDA; likelihood-ratio scoring on top of embeddings.
DERDiarization metricDiarization Error Rate, miss + false alarm + confusion.

Further Reading

The full code

Every file for this lesson. Copy it, run it, then change it.

main.py
"""Speaker verification: toy MFCC-stat embeddings, cosine scoring, EER.

Synthetic "speakers" are sinusoid mixtures with different harmonic profiles.
We enroll each speaker, build same/diff trial pairs, compute the EER.

Run: python3 code/main.py
"""

import math
import random
from collections import defaultdict


def tone_mix(freqs, sr, seconds, amp=0.4, noise=0.02):
    n = int(sr * seconds)
    out = []
    for i in range(n):
        val = 0.0
        t = i / sr
        for f in freqs:
            val += math.sin(2.0 * math.pi * f * t)
        val = amp * val / max(1, len(freqs))
        val += random.gauss(0, noise)
        out.append(val)
    return out


def hann(N):
    return [0.5 * (1.0 - math.cos(2.0 * math.pi * n / (N - 1))) for n in range(N)]


def dft_mag(x):
    n = len(x)
    half = n // 2 + 1
    out = []
    for k in range(half):
        re = 0.0
        im = 0.0
        for j in range(n):
            angle = -2.0 * math.pi * k * j / n
            re += x[j] * math.cos(angle)
            im += x[j] * math.sin(angle)
        out.append(math.sqrt(re * re + im * im))
    return out


def frame_signal(sig, frame_len, hop):
    n = 1 + max(0, (len(sig) - frame_len) // hop)
    return [sig[i * hop : i * hop + frame_len] for i in range(n)]


def stft_mag(sig, frame_len, hop):
    w = hann(frame_len)
    frames = frame_signal(sig, frame_len, hop)
    return [dft_mag([w[j] * f[j] for j in range(frame_len)]) for f in frames]


def hz_to_mel(f):
    return 2595.0 * math.log10(1.0 + f / 700.0)


def mel_to_hz(m):
    return 700.0 * (10 ** (m / 2595.0) - 1.0)


def mel_filterbank(n_mels, n_fft, sr):
    fmin, fmax = 0.0, sr / 2
    mels = [hz_to_mel(fmin) + (hz_to_mel(fmax) - hz_to_mel(fmin)) * i / (n_mels + 1) for i in range(n_mels + 2)]
    hzs = [mel_to_hz(m) for m in mels]
    half = n_fft // 2 + 1
    bins = [min(half - 1, int(round(h * n_fft / sr))) for h in hzs]
    fb = [[0.0] * half for _ in range(n_mels)]
    for m in range(n_mels):
        left, center, right = bins[m], bins[m + 1], bins[m + 2]
        for k in range(left, center):
            fb[m][k] = (k - left) / max(1, center - left)
        for k in range(center, right):
            fb[m][k] = (right - k) / max(1, right - center)
    return fb


def apply_filterbank(spec, fb):
    return [[sum(w * frame[k] for k, w in enumerate(f) if w) for f in fb] for frame in spec]


def log_transform(x, eps=1e-10):
    return [[math.log(max(v, eps)) for v in row] for row in x]


def dct_ii(x, n_coeffs):
    N = len(x)
    return [sum(x[n] * math.cos(math.pi * k * (2 * n + 1) / (2 * N)) for n in range(N)) for k in range(n_coeffs)]


def featurize(signal, sr, n_mfcc=13, n_mels=40, frame_len=256, hop=128):
    mag = stft_mag(signal, frame_len, hop)
    fb = mel_filterbank(n_mels, frame_len, sr)
    mels = apply_filterbank(mag, fb)
    lm = log_transform(mels)
    return [dct_ii(f, n_mfcc) for f in lm]


def l2_normalize(v):
    norm = math.sqrt(sum(x * x for x in v)) or 1e-12
    return [x / norm for x in v]


def embed_mfcc_stats(signal, sr):
    frames = featurize(signal, sr)
    n = len(frames[0])
    mean = [sum(f[i] for f in frames) / len(frames) for i in range(n)]
    var = [sum((f[i] - mean[i]) ** 2 for f in frames) / len(frames) for i in range(n)]
    std = [math.sqrt(v) for v in var]
    return l2_normalize(mean + std)


def cosine(a, b):
    return sum(x * y for x, y in zip(a, b))


def eer(same_scores, diff_scores):
    thresholds = sorted(set(same_scores + diff_scores))
    best_gap = float("inf")
    best_fa, best_fr, best_t = 1.0, 0.0, thresholds[0] if thresholds else 0.0
    for t in thresholds:
        fr = sum(1 for s in same_scores if s < t) / len(same_scores)
        fa = sum(1 for s in diff_scores if s >= t) / len(diff_scores)
        gap = abs(fa - fr)
        if gap < best_gap:
            best_gap = gap
            best_fa, best_fr, best_t = fa, fr, t
    return (best_fa + best_fr) / 2, best_t


def main():
    random.seed(123)
    sr = 8000
    duration = 0.4

    speakers = {
        "alice": [200, 400, 600],
        "bob":   [220, 330, 880],
        "carol": [300, 600, 1200],
        "dave":  [180, 540, 1080],
        "eve":   [260, 520, 780],
    }

    n_per = 5
    print("=== Enroll 5 synthetic speakers, 5 utterances each ===")
    enroll = defaultdict(list)
    for spk, freqs in speakers.items():
        for _ in range(n_per):
            sig = tone_mix(freqs, sr, duration, noise=0.04)
            enroll[spk].append(embed_mfcc_stats(sig, sr))
        print(f"  {spk}: {len(enroll[spk])} embeddings, dim={len(enroll[spk][0])}")

    print()
    print("=== Build trial pairs (same vs different speaker) ===")
    same_scores = []
    diff_scores = []
    spk_list = list(speakers.keys())
    for spk in spk_list:
        embs = enroll[spk]
        for i in range(len(embs)):
            for j in range(i + 1, len(embs)):
                same_scores.append(cosine(embs[i], embs[j]))
    for i, s1 in enumerate(spk_list):
        for s2 in spk_list[i + 1:]:
            for e1 in enroll[s1]:
                for e2 in enroll[s2]:
                    diff_scores.append(cosine(e1, e2))
    print(f"  same-speaker pairs: {len(same_scores)}  mean cosine: {sum(same_scores)/len(same_scores):.3f}")
    print(f"  diff-speaker pairs: {len(diff_scores)}  mean cosine: {sum(diff_scores)/len(diff_scores):.3f}")

    print()
    print("=== Equal Error Rate ===")
    e, t = eer(same_scores, diff_scores)
    print(f"  EER: {e * 100:.2f}%   at threshold: {t:.3f}")
    print(f"  synthetic speakers are near-orthogonal, so this toy hits 0% EER.")
    print(f"  real ECAPA-TDNN on VoxCeleb1-O lands at 0.87% after training on 2700 speakers.")

    print()
    print("=== 2026 speaker-verification leaderboard ===")
    table = [
        ("ReDimNet (2024)",       0.39, "24M"),
        ("WavLM-SV large",        0.42, "316M"),
        ("Pyannote 3.1",          0.65, "6M"),
        ("ECAPA-TDNN",            0.87, "15M"),
        ("x-vector (classic)",    3.10, "5M"),
    ]
    print("  | Model              | EER  | Params |")
    for name, e, p in table:
        print(f"  | {name:<18} | {e:.2f} | {p:<6} |")


if __name__ == "__main__":
    main()