Text-to-Speech (TTS), From Tacotron to F5 and Kokoro
ASR inverts speech to text; TTS inverts text to speech. The 2026 stack is three parts: text → tokens, tokens → mel, mel → waveform. Each part has a default model that fits in a laptop.
The Problem
You have a string: "Please remind me to water the plants at 6 pm." You need a 3-second audio clip that sounds natural, has correct prosody (pauses, stress), pronounces "plants" with the right vowel, and runs in under 300 ms on a CPU for a live voice assistant. You also need to swap voices, handle code-switched input ("remind me at 6 pm, daijoubu?"), and not embarrass yourself on names.
Modern TTS pipelines look like this:
- Text frontend. Normalize text (dates, numbers, emails), convert to phonemes or subword tokens, predict prosody features.
- Acoustic model. Text → mel spectrogram. Tacotron 2 (2017), FastSpeech 2 (2020), VITS (2021), F5-TTS (2024), Kokoro (2024).
- Vocoder. Mel → waveform. WaveNet (2016), WaveRNN, HiFi-GAN (2020), BigVGAN (2022), neural codec vocoders in 2024+.
In 2026 the acoustic + vocoder split blurs with end-to-end diffusion and flow-matching models. But the mental model of three parts still holds for debugging.
The Concept
Tacotron 2 (2017). Seq2seq: char-embedding → BiLSTM encoder → location-sensitive attention → autoregressive LSTM decoder emits mel frames. Slow (AR), wobbly on long text. Still cited as a baseline.
FastSpeech 2 (2020). Non-autoregressive. Duration predictor outputs how many mel frames each phoneme gets. 1-pass, 10× faster than Tacotron. Loses some naturalness (monotonic alignment) but ships everywhere.
VITS (2021). Jointly trains encoder + flow-based duration + HiFi-GAN vocoder end-to-end with variational inference. High quality, single model. Dominant open-source TTS 2022 to 2024. Variants: YourTTS (multi-speaker zero-shot), XTTS v2 (2024, Coqui).
F5-TTS (2024). Diffusion transformer over flow matching. Natural prosody, zero-shot voice cloning with 5 seconds of reference audio. Top of the 2026 open-source TTS leaderboards. 335M params.
Kokoro (2024). Small (82M), CPU-runnable, best-in-class English TTS for real-time use. Closed-vocabulary English-only, apache-2.0.
OpenAI TTS-1-HD, ElevenLabs v2.5, Google Chirp-3. Commercial state of the art. ElevenLabs v2.5 emotion tags ("[whispered]", "[laughing]") and character voices dominate audiobook production in 2026.
Vocoder evolution
| Era | Vocoder | Latency | Quality |
|---|---|---|---|
| 2016 | WaveNet | offline only | SOTA at release |
| 2018 | WaveRNN | ~realtime | good |
| 2020 | HiFi-GAN | 100× realtime | near-human |
| 2022 | BigVGAN | 50× realtime | generalizes across speakers/langs |
| 2024 | SNAC, DAC (neural codecs) | integrated with AR models | discrete tokens, bit-efficient |
By 2026 most "TTS" models are end-to-end from text to waveform; the mel spectrogram is an internal representation.
Evaluation
- MOS (Mean Opinion Score). 1 to 5 scale, crowd-sourced. Still the gold standard; painfully slow.
- CMOS (Comparative MOS). A-vs-B preference. Tighter confidence intervals per annotation.
- UTMOS, DNSMOS. Reference-free neural MOS predictors. Used for leaderboards.
- CER (Character Error Rate) via ASR. Run TTS output through Whisper, compute CER against the input text. Proxy for intelligibility.
- SECS (Speaker Embedding Cosine Similarity). Voice-cloning quality.
2026 numbers on LibriTTS test-clean:
| Model | UTMOS | CER (via Whisper) | Size |
|---|---|---|---|
| Ground truth | 4.08 | 1.2% | |
| F5-TTS | 3.95 | 2.1% | 335M |
| XTTS v2 | 3.81 | 3.5% | 470M |
| VITS | 3.62 | 3.1% | 25M |
| Kokoro v0.19 | 3.87 | 1.8% | 82M |
| Parler-TTS Large | 3.76 | 2.8% | 2.3B |
Build It
Step 1: phonemize input
from phonemizer import phonemize
ph = phonemize("Hello world", language="en-us", backend="espeak")
# 'həloʊ wɜːld'Phonemes are the universal bridge. Avoid feeding raw text to anything below VITS-level quality.
Step 2: run Kokoro (2026 CPU default)
from kokoro import KPipeline
tts = KPipeline(lang_code="a") # "a" = American English
audio, sr = tts("Please remind me to water the plants at 6 pm.", voice="af_bella")
# audio: float32 tensor, sr=24000Runs offline, single file, 82M params.
Step 3: run F5-TTS with voice cloning
from f5_tts.api import F5TTS
tts = F5TTS()
wav = tts.infer(
ref_file="my_voice_5s.wav",
ref_text="The quick brown fox jumps over the lazy dog.",
gen_text="Please remind me to water the plants.",
)Pass a 5-second reference clip + its transcript; F5 clones prosody and timbre.
Step 4: HiFi-GAN vocoder from scratch
Too big to fit in a tutorial script, but the shape is:
class HiFiGAN(nn.Module):
def __init__(self, mel_channels=80, upsample_rates=[8, 8, 2, 2]):
super().__init__()
# 4 upsample blocks, total 256x to go from mel-rate to audio-rate
...
def forward(self, mel):
return self.blocks(mel) # -> waveformTraining: adversarial (discriminator on short windows) + mel-spectrogram reconstruction loss + feature-matching loss. Commoditized, use pretrained checkpoints from hifi-gan repo or nvidia-NeMo.
Step 5: the full pipeline (pseudocode)
text = "Please remind me at 6 pm."
phones = phonemize(text)
mel = acoustic_model(phones, speaker=alice) # [T, 80]
wav = vocoder(mel) # [T * 256]
soundfile.write("out.wav", wav, 24000)Use It
The 2026 stack:
| Situation | Pick |
|---|---|
| Real-time English voice assistant | Kokoro (CPU) or XTTS v2 (GPU) |
| Voice cloning from 5 s reference | F5-TTS |
| Commercial character voices | ElevenLabs v2.5 |
| Audiobook narration | ElevenLabs v2.5 or XTTS v2 + fine-tune |
| Low-resource language | Train VITS on 5 to 20 h target-lang data |
| Expressive / emotion tags | ElevenLabs v2.5 or StyleTTS 2 fine-tune |
Open-source leader as of 2026: F5-TTS for quality, Kokoro for efficiency. Don't reach for Tacotron unless you are a historian.
Pitfalls
- No text normalizer. "Dr. Smith" reads as "Doctor" or "Drive"? "2026" as "twenty twenty six" or "two zero two six"? Normalize BEFORE phonemizer.
- OOV proper nouns. "Ghumare" → "ghyu-mair"? Ship a fallback grapheme-to-phoneme model for unknown tokens.
- Clipping. Vocoder output rarely clips, but mel scaling mismatch at inference can overshoot ±1.0. Always
np.clip(wav, -1, 1). - Sample-rate mismatch. Kokoro outputs 24 kHz; your downstream pipeline expects 16 kHz → resample or get aliasing.
Ship It
Save as outputs/skill-tts-designer.md. Design a TTS pipeline for a given voice, latency, and language target.
Exercises
- Easy. Run
code/main.py. Builds a phoneme dictionary from a toy vocab, estimates duration per phoneme, and prints a fake "mel" schedule. - Medium. Install Kokoro, synthesize the same sentence at voice
af_bellaandam_adam. Compare audio durations and subjective quality. - Hard. Record a 5-second reference clip of yourself. Use F5-TTS to clone it. Report SECS between reference and cloned output.
Key Terms
| Term | What people say | What it actually means |
|---|---|---|
| Phoneme | Sound unit | Abstract sound class; 39 in English (ARPABet). |
| Duration predictor | How long each phoneme lasts | Non-AR model output; integer frames per phoneme. |
| Vocoder | Mel → waveform | Neural net mapping mel-spec to raw samples. |
| HiFi-GAN | Standard vocoder | GAN-based; dominant 2020 to 2024. |
| MOS | Subjective quality | 1 to 5 mean opinion score from human raters. |
| SECS | Voice-clone metric | Cosine similarity between target and output speaker embedding. |
| F5-TTS | 2024 open-source SOTA | Flow-matching diffusion; zero-shot cloning. |
| Kokoro | CPU English leader | 82M-param model, Apache 2.0. |
Further Reading
- Shen et al. (2017). Tacotron 2the seq2seq baseline.
- Kim, Kong, Son (2021). VITSend-to-end flow-based.
- Chen et al. (2024). F5-TTScurrent open-source SOTA.
- Kong, Kim, Bae (2020). HiFi-GANthe vocoder that still ships in 2026.
- Kokoro-82M on HuggingFace2024 CPU-friendly English TTS.
The full code
Every file for this lesson. Copy it, run it, then change it.
"""TTS internals demo: phoneme lookup + duration estimation + mel frame schedule.
Stdlib only. Builds a toy English grapheme-to-phoneme table, estimates
durations, and prints the frame schedule a FastSpeech-style model would
use. For real synthesis, install kokoro or f5-tts (see docs).
Run: python3 code/main.py
"""
import math
import random
# Minimal grapheme-to-phoneme table: one mapping per common grapheme cluster.
# Real systems use espeak-ng or g2p-en (CMU dictionary) — this is toy scope.
G2P = {
" ": ["_"],
"a": ["AH"], "b": ["B"], "c": ["K"], "d": ["D"], "e": ["EH"],
"f": ["F"], "g": ["G"], "h": ["HH"], "i": ["IH"], "j": ["JH"],
"k": ["K"], "l": ["L"], "m": ["M"], "n": ["N"], "o": ["AO"],
"p": ["P"], "q": ["K"], "r": ["R"], "s": ["S"], "t": ["T"],
"u": ["UH"], "v": ["V"], "w": ["W"], "x": ["K", "S"],
"y": ["Y"], "z": ["Z"],
"the": ["DH", "AH"],
"ing": ["IH", "NG"],
"er": ["ER"],
"sh": ["SH"],
"ch": ["CH"],
"th": ["TH"],
"ee": ["IY"],
"oo": ["UW"],
"ow": ["AW"],
"ay": ["EY"],
".": ["_PAUSE_"],
",": ["_SHORT_"],
"?": ["_PAUSE_"],
"!": ["_PAUSE_"],
}
# Typical durations (frames @ 12.5 ms hop); roughly matches FastSpeech stats
DURATION_FRAMES = {
"AA": 9, "AE": 7, "AH": 6, "AO": 8, "AW": 9, "AY": 8, "B": 4, "CH": 6,
"D": 4, "DH": 5, "EH": 6, "ER": 7, "EY": 8, "F": 6, "G": 5, "HH": 4,
"IH": 5, "IY": 7, "JH": 6, "K": 5, "L": 5, "M": 5, "N": 5, "NG": 6,
"OW": 8, "OY": 9, "P": 5, "R": 5, "S": 6, "SH": 7, "T": 4, "TH": 5,
"UH": 6, "UW": 8, "V": 5, "W": 5, "Y": 5, "Z": 6, "ZH": 7,
"_": 3, # word boundary
"_SHORT_": 6, # comma pause
"_PAUSE_": 12, # sentence pause
}
def phonemize(text):
text = text.lower()
phones = []
i = 0
while i < len(text):
matched = False
for length in (3, 2, 1):
if i + length <= len(text):
chunk = text[i : i + length]
if chunk in G2P:
phones.extend(G2P[chunk])
i += length
matched = True
break
if not matched:
i += 1
return phones
def duration(phones, jitter=0.1, seed=0):
random.seed(seed)
out = []
for p in phones:
base = DURATION_FRAMES.get(p, 5)
noise = int(round(base * random.uniform(-jitter, jitter)))
out.append(max(1, base + noise))
return out
def mel_schedule(phones, durs, hop_ms=12.5):
schedule = []
t = 0.0
for p, d in zip(phones, durs):
schedule.append((p, t, t + d * hop_ms))
t += d * hop_ms
return schedule, t
def main():
text = "Please remind me to water the plants at 6 pm."
print("=== Step 1: grapheme to phoneme ===")
print(f" text: {text!r}")
phones = phonemize(text)
print(f" phones ({len(phones)}): {' '.join(phones[:20])}{'...' if len(phones) > 20 else ''}")
print()
print("=== Step 2: estimate per-phoneme duration ===")
durs = duration(phones, jitter=0.1, seed=42)
print(f" durations (frames): {durs[:20]}{'...' if len(durs) > 20 else ''}")
print()
print("=== Step 3: mel frame schedule (12.5 ms hop) ===")
sched, total_ms = mel_schedule(phones, durs)
print(f" total duration: {total_ms:.1f} ms ({total_ms / 1000:.2f} s)")
print(f" first 10 frames:")
for p, s, e in sched[:10]:
print(f" {p:<10} {s:6.1f} – {e:6.1f} ms")
print()
print("=== Step 4: total mel frames sent to vocoder ===")
total_frames = sum(durs)
audio_samples = total_frames * 300 # 12.5 ms @ 24 kHz = 300 samples
print(f" mel frames: {total_frames} audio samples @ 24 kHz: {audio_samples}")
print(f" pipeline memory budget: {total_frames * 80 * 4 / 1024:.1f} KB (mel, float32)")
print()
print("=== Step 5: 2026 TTS quality board (UTMOS / CER / size) ===")
table = [
("ground truth", 4.08, 1.2, "—"),
("F5-TTS", 3.95, 2.1, "335M"),
("Kokoro v0.19", 3.87, 1.8, "82M"),
("XTTS v2", 3.81, 3.5, "470M"),
("Parler-TTS L", 3.76, 2.8, "2.3B"),
("VITS", 3.62, 3.1, "25M"),
]
print(" | Model | UTMOS | CER% | Size |")
for name, u, c, s in table:
print(f" | {name:<17} | {u:.2f} | {c:.1f} | {s:<4} |")
if __name__ == "__main__":
main()