Get Started

T5, BART, Encoder-Decoder Models

Encoders understand. Decoders generate. Put them back together and you get a model built for input → output tasks: translate, summarize, rewrite, transcribe.

45 minutesLearnPythonFree lesson

The Problem

Decoder-only GPT and encoder-only BERT each strip down the 2017 architecture for a different goal. But many tasks are naturally input-output:

  • Translation: English → French.
  • Summarization: 5,000-token article → 200-token summary.
  • Speech recognition: audio tokens → text tokens.
  • Structured extraction: prose → JSON.

For these, encoder-decoder makes the cleanest fit. The encoder produces a dense representation of the source. The decoder generates the output, cross-attending to that representation at every step. Training is shift-by-one on the output side. Same loss as GPT, just conditioned on the encoder output.

Two papers defined the modern playbook:

  1. T5 (Raffel et al. 2019). "Text-to-Text Transfer Transformer." Every NLP task reframed as text-in, text-out. Single architecture, single vocabulary, single loss. Pretrained on masked span prediction (corrupt spans in the input, decode them in the output).
  2. BART (Lewis et al. 2019). "Bidirectional and Auto-Regressive Transformer." Denoising autoencoder: corrupt input in multiple ways (shuffle, mask, delete, rotate), ask the decoder to reconstruct the original.

In 2026 the encoder-decoder format lives on where input structure matters:

  • Whisper (speech → text).
  • Google's translation stack.
  • Some code-completion / repair models that have distinct context-and-edit structures.
  • Flan-T5 and variants for structured reasoning tasks.

Decoder-only won the spotlight, but encoder-decoder never went away.

The Concept

Encoder-decoder with cross-attention

The forward loop

Code
source tokens ─▶ encoder ─▶ (N_src, d_model)  ──┐
                                                 │
target tokens ─▶ decoder block                   │
                 ├─▶ masked self-attention       │
                 ├─▶ cross-attention ◀───────────┘
                 └─▶ FFN
                ↓
              next-token logits

Crucially, the encoder runs once per input. The decoder runs autoregressively but cross-attends to the same encoder output at every step. Caching the encoder output is a free speedup for long inputs.

T5 pretraining, span corruption

Pick random spans of the input (average length 3 tokens, 15% total). Replace each span with a unique sentinel: <extra_id_0><extra_id_1>etc. The decoder outputs only the corrupted spans with their sentinel prefix:

Code
source: The quick <extra_id_0> fox jumps <extra_id_1> dog
target: <extra_id_0> brown <extra_id_1> over the lazy

Cheaper signal than predicting the whole sequence. Competitive with MLM (BERT) and prefix-LM (UniLM) in the T5 paper's ablation.

BART pretraining, multi-noise denoising

BART tries five noising functions:

  1. Token masking.
  2. Token deletion.
  3. Text infilling (mask a span, decoder inserts the right length).
  4. Sentence permutation.
  5. Document rotation.

Combining text infilling + sentence permutation produced the best downstream numbers. The decoder always reconstructs the original. BART's output is the full sequence, not just the corrupted spans, so pretraining compute is higher than T5.

Inference

Same autoregressive generation as GPT. Greedy / beam / top-p sampling apply. Beam search (width 4 to 5) is standard for translation and summarization because the output distribution is narrower than chat.

When to pick each variant in 2026

TaskEncoder-decoder?Why
TranslationYes, usuallyClear source sequence; fixed output distribution; beam search works
Speech-to-textYes (Whisper)Input modality differs from output; encoder shapes audio features
Chat / reasoningNo, decoder-onlyNo persistent "input", the conversation is the sequence
Code completionUsually noDecoder-only with long context wins; code models like Qwen 2.5 Coder are decoder-only
SummarizationEither worksBART, PEGASUS beat earlier decoder-only baselines; modern decoder-only LLMs match them
Structured extractionEitherT5 is clean because "text → text" absorbs any output format

The trend since ~2022: decoder-only takes over tasks that encoder-decoder used to own because (a) instruction-tuned decoder-only LLMs generalize to anything via prompting, (b) one architecture scales easier than two, (c) RLHF assumes a decoder. Encoder-decoder holds on where input modality differs (speech, images) or where beam search quality matters.

Build It

See code/main.py. We implement T5-style span corruption for a toy corpus, the most useful single piece of this lesson because it shows up in every encoder-decoder pretraining recipe since.

Step 1: span corruption

Python
def corrupt_spans(tokens, mask_rate=0.15, mean_span=3.0, rng=None):
    """Pick spans summing to ~mask_rate of tokens. Return (corrupted_input, target)."""
    n = len(tokens)
    n_mask = max(1, int(n * mask_rate))
    n_spans = max(1, int(round(n_mask / mean_span)))
    ...

The target format is the T5 convention: <sent0> span0 <sent1> span1 .... The corrupted input interleaves unchanged tokens with the sentinel tokens at span locations.

Step 2: verify round-trip

Given the corrupted input and target, reconstruct the original sentence. If your corruption is reversible, the forward pass is well-defined. This is a sanity check, real training never does this, but the test is cheap and catches off-by-one bugs in your span bookkeeping.

Step 3: BART noising

Five functions: token_masktoken_deletetext_infillsentence_permutedocument_rotate. Compose two of them and show the result.

Use It

HuggingFace reference:

Python
from transformers import T5ForConditionalGeneration, T5Tokenizer
tok = T5Tokenizer.from_pretrained("google/flan-t5-base")
model = T5ForConditionalGeneration.from_pretrained("google/flan-t5-base")

inputs = tok("translate English to French: Attention is all you need.", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=32)
print(tok.decode(out[0], skip_special_tokens=True))

The T5 trick: the task name goes into the input text. Same model handles dozens of tasks because each task is text-in, text-out. In 2026 this pattern has been generalized by instruction-tuned decoder-only models, but T5 codified it first.

Ship It

See outputs/skill-seq2seq-picker.md. The skill picks between encoder-decoder and decoder-only for a new task given input-output structure, latency, and quality targets.

Exercises

  1. Easy. Run code/main.pyapply span corruption to a 30-token sentence, verify that concatenating the non-sentinel source tokens with the decoded target spans reproduces the original.
  2. Medium. Implement BART's text_infill noise: replace random spans with a single <mask> token, and the decoder must infer the correct span length plus contents. Show one example.
  3. Hard. Fine-tune flan-t5-small on a tiny English → pig-Latin corpus (200 pairs). Measure BLEU on a held-out 50-pair set. Compare against fine-tuning Llama-3.2-1B on the same data with the same compute.

Key Terms

TermWhat people sayWhat it actually means
Encoder-decoder"Seq2seq transformer"Two stacks: bidirectional encoder for input, causal decoder with cross-attention for output.
Cross-attention"Where source talks to target"Decoder's Q × encoder's K/V. The only place encoder information enters the decoder.
Span corruption"T5's pretraining trick"Replace random spans with sentinel tokens; decoder outputs the spans.
Denoising objective"BART's game"Apply a noise function to the input, train the decoder to reconstruct the clean sequence.
Sentinel token"The <extra_id_N> placeholder"Special tokens that tag corrupted spans in the source and re-tag them in the target.
Flan"Instruction-tuned T5"T5 fine-tuned on >1,800 tasks; made encoder-decoder competitive at instruction-following.
Beam search"Decoding strategy"Keep top-k partial sequences at each step; standard for translation/summarization.
Teacher forcing"Training-time input"During training, feed the true previous output token to the decoder, not the sampled one.

Further Reading

The full code

Every file for this lesson. Copy it, run it, then change it.

main.py
"""T5 span corruption + BART denoising noise functions.

Pure stdlib. Shows how encoder-decoder models turn any input into
a supervised (corrupted_input -> clean_spans) training pair.
"""

import random


def sentinel(i):
    return f"<extra_id_{i}>"


def corrupt_spans(tokens, mask_rate=0.15, mean_span=3.0, rng=None):
    """T5-style span corruption.

    Returns (corrupted_source, decoder_target) as lists of tokens (strings).
    """
    if rng is None:
        rng = random.Random()
    n = len(tokens)
    n_mask = max(1, int(round(n * mask_rate)))
    n_spans = max(1, int(round(n_mask / mean_span)))
    # Pick span start positions with no overlap.
    positions = list(range(n))
    rng.shuffle(positions)
    starts = []
    used = [False] * n
    span_lengths = []
    remaining = n_mask
    for _ in range(n_spans):
        if remaining <= 0:
            break
        # pick a random starting point not yet used and with room
        random_order = list(range(n))
        rng.shuffle(random_order)
        chosen_start = None
        for start in random_order:
            if used[start]:
                continue
            # span length
            length = max(1, int(rng.gauss(mean_span, 1.0)))
            length = min(length, remaining, n - start)
            if length < 1:
                continue
            if any(used[i] for i in range(start, start + length)):
                continue
            chosen_start = start
            for i in range(start, start + length):
                used[i] = True
            starts.append(start)
            span_lengths.append(length)
            remaining -= length
            break
        if chosen_start is None:
            break

    ordered = sorted(zip(starts, span_lengths), key=lambda x: x[0])

    source = []
    target = []
    prev_end = 0
    for idx, (start, length) in enumerate(ordered):
        source.extend(tokens[prev_end:start])
        source.append(sentinel(idx))
        target.append(sentinel(idx))
        target.extend(tokens[start:start + length])
        prev_end = start + length
    source.extend(tokens[prev_end:])
    target.append(sentinel(len(ordered)))  # closing sentinel
    return source, target


def round_trip(source, target):
    """Reconstruct original by replacing sentinels in source with corresponding target spans."""
    # Parse target into sentinel->span map
    spans = {}
    current_key = None
    current_span = []
    for tok in target:
        if tok.startswith("<extra_id_"):
            if current_key is not None:
                spans[current_key] = current_span
            current_key = tok
            current_span = []
        else:
            current_span.append(tok)
    # Last sentinel in target has no following span (closing marker).
    out = []
    for tok in source:
        if tok.startswith("<extra_id_"):
            out.extend(spans.get(tok, []))
        else:
            out.append(tok)
    return out


def token_mask(tokens, rate=0.15, rng=None, mask_token="<mask>"):
    if rng is None:
        rng = random.Random()
    return [mask_token if rng.random() < rate else t for t in tokens]


def token_delete(tokens, rate=0.15, rng=None):
    if rng is None:
        rng = random.Random()
    return [t for t in tokens if rng.random() >= rate]


def text_infill(tokens, rate=0.15, mean_span=3.0, rng=None, mask_token="<mask>"):
    """BART text infill: mask spans with a SINGLE mask; decoder infers length."""
    if rng is None:
        rng = random.Random()
    out = []
    i = 0
    n = len(tokens)
    budget = int(n * rate)
    while i < n:
        if budget > 0 and rng.random() < 0.3:
            span_len = max(1, min(int(rng.gauss(mean_span, 1.0)), budget, n - i))
            out.append(mask_token)
            budget -= span_len
            i += span_len
        else:
            out.append(tokens[i])
            i += 1
    return out


def sentence_permute(sentences, rng=None):
    if rng is None:
        rng = random.Random()
    sents = list(sentences)
    rng.shuffle(sents)
    return sents


def document_rotate(tokens, rng=None):
    if rng is None:
        rng = random.Random()
    if len(tokens) <= 1:
        return tokens
    pivot = rng.randrange(1, len(tokens))
    return tokens[pivot:] + tokens[:pivot]


def main():
    rng = random.Random(42)

    sentence = (
        "the quick brown fox jumps over the lazy dog a stitch in time saves nine "
        "language models learn statistical patterns subword tokenization helps rare words"
    ).split()

    print("=== T5 span corruption ===")
    source, target = corrupt_spans(sentence, mask_rate=0.20, mean_span=3.0, rng=rng)
    print("corrupted source:")
    print("  " + " ".join(source))
    print()
    print("decoder target:")
    print("  " + " ".join(target))
    print()
    reconstructed = round_trip(source, target)
    print("reconstruction matches original:",
          "YES" if reconstructed == sentence else "NO")
    if reconstructed != sentence:
        print("  reconstructed: " + " ".join(reconstructed))

    print()
    print("=== BART noise functions ===")
    print("original: " + " ".join(sentence[:14]))
    print()
    print("token mask:     " + " ".join(token_mask(sentence[:14], rate=0.2, rng=random.Random(1))))
    print("token delete:   " + " ".join(token_delete(sentence[:14], rate=0.2, rng=random.Random(2))))
    print("text infill:    " + " ".join(text_infill(sentence[:14], rate=0.3, rng=random.Random(3))))

    sentences = [
        ["the", "quick", "brown", "fox"],
        ["a", "stitch", "in", "time"],
        ["language", "models", "learn", "patterns"],
    ]
    perm = sentence_permute(sentences, rng=random.Random(4))
    print("sentence permute:")
    for s in perm:
        print("  " + " ".join(s))

    print()
    print("document rotate: " + " ".join(document_rotate(sentence[:14], rng=random.Random(5))))


if __name__ == "__main__":
    main()
You've finished the free half of Transformers Deep Dive

Most people stop here. Don't be most people.

The next 8 lessons are where Phase 07 turns from understanding into ability: the harder builds, the production patterns and the quizzes that prove you've actually got it. Unlock the full AI Expert Masterclass: all 542 expert lessons, every line of code, every phase.

  1. 09Vision Transformers (ViT)
  2. 10Audio Transformers, Whisper Architecture
  3. 11Mixture of Experts (MoE)
  4. 12KV Cache, Flash Attention & Inference Optimization
  5. + 4 more in this phase, and 542 across the whole masterclass