Get Started

Text-to-Speech (TTS), From Tacotron to F5 and Kokoro

ASR inverts speech to text; TTS inverts text to speech. The 2026 stack is three parts: text → tokens, tokens → mel, mel → waveform. Each part has a default model that fits in a laptop.

75 minutesBuildPythonFree lesson

The Problem

You have a string: "Please remind me to water the plants at 6 pm." You need a 3-second audio clip that sounds natural, has correct prosody (pauses, stress), pronounces "plants" with the right vowel, and runs in under 300 ms on a CPU for a live voice assistant. You also need to swap voices, handle code-switched input ("remind me at 6 pm, daijoubu?"), and not embarrass yourself on names.

Modern TTS pipelines look like this:

  1. Text frontend. Normalize text (dates, numbers, emails), convert to phonemes or subword tokens, predict prosody features.
  2. Acoustic model. Text → mel spectrogram. Tacotron 2 (2017), FastSpeech 2 (2020), VITS (2021), F5-TTS (2024), Kokoro (2024).
  3. Vocoder. Mel → waveform. WaveNet (2016), WaveRNN, HiFi-GAN (2020), BigVGAN (2022), neural codec vocoders in 2024+.

In 2026 the acoustic + vocoder split blurs with end-to-end diffusion and flow-matching models. But the mental model of three parts still holds for debugging.

The Concept

Tacotron, FastSpeech, VITS, F5/Kokoro side-by-side

Tacotron 2 (2017). Seq2seq: char-embedding → BiLSTM encoder → location-sensitive attention → autoregressive LSTM decoder emits mel frames. Slow (AR), wobbly on long text. Still cited as a baseline.

FastSpeech 2 (2020). Non-autoregressive. Duration predictor outputs how many mel frames each phoneme gets. 1-pass, 10× faster than Tacotron. Loses some naturalness (monotonic alignment) but ships everywhere.

VITS (2021). Jointly trains encoder + flow-based duration + HiFi-GAN vocoder end-to-end with variational inference. High quality, single model. Dominant open-source TTS 2022 to 2024. Variants: YourTTS (multi-speaker zero-shot), XTTS v2 (2024, Coqui).

F5-TTS (2024). Diffusion transformer over flow matching. Natural prosody, zero-shot voice cloning with 5 seconds of reference audio. Top of the 2026 open-source TTS leaderboards. 335M params.

Kokoro (2024). Small (82M), CPU-runnable, best-in-class English TTS for real-time use. Closed-vocabulary English-only, apache-2.0.

OpenAI TTS-1-HD, ElevenLabs v2.5, Google Chirp-3. Commercial state of the art. ElevenLabs v2.5 emotion tags ("[whispered]", "[laughing]") and character voices dominate audiobook production in 2026.

Vocoder evolution

EraVocoderLatencyQuality
2016WaveNetoffline onlySOTA at release
2018WaveRNN~realtimegood
2020HiFi-GAN100× realtimenear-human
2022BigVGAN50× realtimegeneralizes across speakers/langs
2024SNAC, DAC (neural codecs)integrated with AR modelsdiscrete tokens, bit-efficient

By 2026 most "TTS" models are end-to-end from text to waveform; the mel spectrogram is an internal representation.

Evaluation

  • MOS (Mean Opinion Score). 1 to 5 scale, crowd-sourced. Still the gold standard; painfully slow.
  • CMOS (Comparative MOS). A-vs-B preference. Tighter confidence intervals per annotation.
  • UTMOS, DNSMOS. Reference-free neural MOS predictors. Used for leaderboards.
  • CER (Character Error Rate) via ASR. Run TTS output through Whisper, compute CER against the input text. Proxy for intelligibility.
  • SECS (Speaker Embedding Cosine Similarity). Voice-cloning quality.

2026 numbers on LibriTTS test-clean:

ModelUTMOSCER (via Whisper)Size
Ground truth4.081.2%
F5-TTS3.952.1%335M
XTTS v23.813.5%470M
VITS3.623.1%25M
Kokoro v0.193.871.8%82M
Parler-TTS Large3.762.8%2.3B

Build It

Step 1: phonemize input

Python
from phonemizer import phonemize
ph = phonemize("Hello world", language="en-us", backend="espeak")
# 'həloʊ wɜːld'

Phonemes are the universal bridge. Avoid feeding raw text to anything below VITS-level quality.

Step 2: run Kokoro (2026 CPU default)

Python
from kokoro import KPipeline
tts = KPipeline(lang_code="a")  # "a" = American English
audio, sr = tts("Please remind me to water the plants at 6 pm.", voice="af_bella")
# audio: float32 tensor, sr=24000

Runs offline, single file, 82M params.

Step 3: run F5-TTS with voice cloning

Python
from f5_tts.api import F5TTS
tts = F5TTS()
wav = tts.infer(
    ref_file="my_voice_5s.wav",
    ref_text="The quick brown fox jumps over the lazy dog.",
    gen_text="Please remind me to water the plants.",
)

Pass a 5-second reference clip + its transcript; F5 clones prosody and timbre.

Step 4: HiFi-GAN vocoder from scratch

Too big to fit in a tutorial script, but the shape is:

Python
class HiFiGAN(nn.Module):
    def __init__(self, mel_channels=80, upsample_rates=[8, 8, 2, 2]):
        super().__init__()
        # 4 upsample blocks, total 256x to go from mel-rate to audio-rate
        ...
    def forward(self, mel):
        return self.blocks(mel)  # -> waveform

Training: adversarial (discriminator on short windows) + mel-spectrogram reconstruction loss + feature-matching loss. Commoditized, use pretrained checkpoints from hifi-gan repo or nvidia-NeMo.

Step 5: the full pipeline (pseudocode)

Python
text = "Please remind me at 6 pm."
phones = phonemize(text)
mel = acoustic_model(phones, speaker=alice)      # [T, 80]
wav = vocoder(mel)                                # [T * 256]
soundfile.write("out.wav", wav, 24000)

Use It

The 2026 stack:

SituationPick
Real-time English voice assistantKokoro (CPU) or XTTS v2 (GPU)
Voice cloning from 5 s referenceF5-TTS
Commercial character voicesElevenLabs v2.5
Audiobook narrationElevenLabs v2.5 or XTTS v2 + fine-tune
Low-resource languageTrain VITS on 5 to 20 h target-lang data
Expressive / emotion tagsElevenLabs v2.5 or StyleTTS 2 fine-tune

Open-source leader as of 2026: F5-TTS for quality, Kokoro for efficiency. Don't reach for Tacotron unless you are a historian.

Pitfalls

  • No text normalizer. "Dr. Smith" reads as "Doctor" or "Drive"? "2026" as "twenty twenty six" or "two zero two six"? Normalize BEFORE phonemizer.
  • OOV proper nouns. "Ghumare" → "ghyu-mair"? Ship a fallback grapheme-to-phoneme model for unknown tokens.
  • Clipping. Vocoder output rarely clips, but mel scaling mismatch at inference can overshoot ±1.0. Always np.clip(wav, -1, 1).
  • Sample-rate mismatch. Kokoro outputs 24 kHz; your downstream pipeline expects 16 kHz → resample or get aliasing.

Ship It

Save as outputs/skill-tts-designer.md. Design a TTS pipeline for a given voice, latency, and language target.

Exercises

  1. Easy. Run code/main.py. Builds a phoneme dictionary from a toy vocab, estimates duration per phoneme, and prints a fake "mel" schedule.
  2. Medium. Install Kokoro, synthesize the same sentence at voice af_bella and am_adam. Compare audio durations and subjective quality.
  3. Hard. Record a 5-second reference clip of yourself. Use F5-TTS to clone it. Report SECS between reference and cloned output.

Key Terms

TermWhat people sayWhat it actually means
PhonemeSound unitAbstract sound class; 39 in English (ARPABet).
Duration predictorHow long each phoneme lastsNon-AR model output; integer frames per phoneme.
VocoderMel → waveformNeural net mapping mel-spec to raw samples.
HiFi-GANStandard vocoderGAN-based; dominant 2020 to 2024.
MOSSubjective quality1 to 5 mean opinion score from human raters.
SECSVoice-clone metricCosine similarity between target and output speaker embedding.
F5-TTS2024 open-source SOTAFlow-matching diffusion; zero-shot cloning.
KokoroCPU English leader82M-param model, Apache 2.0.

Further Reading

The full code

Every file for this lesson. Copy it, run it, then change it.

main.py
"""TTS internals demo: phoneme lookup + duration estimation + mel frame schedule.

Stdlib only. Builds a toy English grapheme-to-phoneme table, estimates
durations, and prints the frame schedule a FastSpeech-style model would
use. For real synthesis, install kokoro or f5-tts (see docs).

Run: python3 code/main.py
"""

import math
import random


# Minimal grapheme-to-phoneme table: one mapping per common grapheme cluster.
# Real systems use espeak-ng or g2p-en (CMU dictionary) — this is toy scope.
G2P = {
    " ":    ["_"],
    "a":    ["AH"],  "b":    ["B"],  "c":    ["K"],  "d":    ["D"],  "e":    ["EH"],
    "f":    ["F"],   "g":    ["G"],  "h":    ["HH"], "i":    ["IH"], "j":    ["JH"],
    "k":    ["K"],   "l":    ["L"],  "m":    ["M"],  "n":    ["N"],  "o":    ["AO"],
    "p":    ["P"],   "q":    ["K"],  "r":    ["R"],  "s":    ["S"],  "t":    ["T"],
    "u":    ["UH"],  "v":    ["V"],  "w":    ["W"],  "x":    ["K", "S"],
    "y":    ["Y"],   "z":    ["Z"],
    "the":  ["DH", "AH"],
    "ing":  ["IH", "NG"],
    "er":   ["ER"],
    "sh":   ["SH"],
    "ch":   ["CH"],
    "th":   ["TH"],
    "ee":   ["IY"],
    "oo":   ["UW"],
    "ow":   ["AW"],
    "ay":   ["EY"],
    ".":    ["_PAUSE_"],
    ",":    ["_SHORT_"],
    "?":    ["_PAUSE_"],
    "!":    ["_PAUSE_"],
}

# Typical durations (frames @ 12.5 ms hop); roughly matches FastSpeech stats
DURATION_FRAMES = {
    "AA": 9, "AE": 7, "AH": 6, "AO": 8, "AW": 9, "AY": 8, "B": 4, "CH": 6,
    "D": 4, "DH": 5, "EH": 6, "ER": 7, "EY": 8, "F": 6, "G": 5, "HH": 4,
    "IH": 5, "IY": 7, "JH": 6, "K": 5, "L": 5, "M": 5, "N": 5, "NG": 6,
    "OW": 8, "OY": 9, "P": 5, "R": 5, "S": 6, "SH": 7, "T": 4, "TH": 5,
    "UH": 6, "UW": 8, "V": 5, "W": 5, "Y": 5, "Z": 6, "ZH": 7,
    "_": 3,           # word boundary
    "_SHORT_": 6,     # comma pause
    "_PAUSE_": 12,    # sentence pause
}


def phonemize(text):
    text = text.lower()
    phones = []
    i = 0
    while i < len(text):
        matched = False
        for length in (3, 2, 1):
            if i + length <= len(text):
                chunk = text[i : i + length]
                if chunk in G2P:
                    phones.extend(G2P[chunk])
                    i += length
                    matched = True
                    break
        if not matched:
            i += 1
    return phones


def duration(phones, jitter=0.1, seed=0):
    random.seed(seed)
    out = []
    for p in phones:
        base = DURATION_FRAMES.get(p, 5)
        noise = int(round(base * random.uniform(-jitter, jitter)))
        out.append(max(1, base + noise))
    return out


def mel_schedule(phones, durs, hop_ms=12.5):
    schedule = []
    t = 0.0
    for p, d in zip(phones, durs):
        schedule.append((p, t, t + d * hop_ms))
        t += d * hop_ms
    return schedule, t


def main():
    text = "Please remind me to water the plants at 6 pm."
    print("=== Step 1: grapheme to phoneme ===")
    print(f"  text: {text!r}")
    phones = phonemize(text)
    print(f"  phones ({len(phones)}): {' '.join(phones[:20])}{'...' if len(phones) > 20 else ''}")

    print()
    print("=== Step 2: estimate per-phoneme duration ===")
    durs = duration(phones, jitter=0.1, seed=42)
    print(f"  durations (frames): {durs[:20]}{'...' if len(durs) > 20 else ''}")

    print()
    print("=== Step 3: mel frame schedule (12.5 ms hop) ===")
    sched, total_ms = mel_schedule(phones, durs)
    print(f"  total duration: {total_ms:.1f} ms  ({total_ms / 1000:.2f} s)")
    print(f"  first 10 frames:")
    for p, s, e in sched[:10]:
        print(f"    {p:<10} {s:6.1f} – {e:6.1f} ms")

    print()
    print("=== Step 4: total mel frames sent to vocoder ===")
    total_frames = sum(durs)
    audio_samples = total_frames * 300  # 12.5 ms @ 24 kHz = 300 samples
    print(f"  mel frames: {total_frames}  audio samples @ 24 kHz: {audio_samples}")
    print(f"  pipeline memory budget: {total_frames * 80 * 4 / 1024:.1f} KB (mel, float32)")

    print()
    print("=== Step 5: 2026 TTS quality board (UTMOS / CER / size) ===")
    table = [
        ("ground truth",   4.08, 1.2,   "—"),
        ("F5-TTS",         3.95, 2.1,   "335M"),
        ("Kokoro v0.19",   3.87, 1.8,   "82M"),
        ("XTTS v2",        3.81, 3.5,   "470M"),
        ("Parler-TTS L",   3.76, 2.8,   "2.3B"),
        ("VITS",           3.62, 3.1,   "25M"),
    ]
    print("  | Model             | UTMOS | CER%  | Size |")
    for name, u, c, s in table:
        print(f"  | {name:<17} | {u:.2f}  | {c:.1f}   | {s:<4} |")


if __name__ == "__main__":
    main()