Get Started

Multi-Head Self-Attention

One linear projection, three views, H parallel heads, one mask. The attention block as the model actually uses it.

90 minutesBuildPythonFree lesson

Learning Objectives

  • Implement a batched Query/Key/Value projection as a single linear layer split into H heads.
  • Compute scaled dot-product attention with the correct normalization and dtype handling.
  • Apply a causal mask that prevents a position from attending to future positions.
  • Inspect per-head attention weights for a fixed input and reason about what each head looks at.
  • Train a small attention block on a toy task and watch the loss fall as the heads specialize.

The frame

Attention is the function that lets a token's representation pull information from other tokens in the same sequence. Self-attention means queries, keys, and values are all derived from the same input. Multi-head means the projection is split into H parallel attention problems whose outputs are concatenated and projected back.

The efficient implementation pattern is one linear layer that projects from D to 3 * D and gets sliced into three views, then reshaped into H heads of size D // H each. The matmul, softmax, and weighted sum happen as batched tensor operations so the heads run in parallel on the accelerator.

This lesson builds that block. It also adds the causal mask so the same code works as the attention layer in a decoder-only language model. The next lesson stacks the block into a full transformer and the lesson after trains it.

The shape contract

The input is (B, T, D). The output is (B, T, D). The mask is (T, T) or broadcastable to it. Inside the block the intermediate tensors have shape (B, H, T, d_head) where d_head = D // H. The constraint is D % H == 0.

The two linear layers (the QKV projection and the output projection) are the only parameters in the block. The mask, the softmax, the matmuls, and the reshapes are all parameter-free.

The QKV split

The naive implementation has three separate linear layers, one each for Q, K, and V. The efficient one has a single layer that outputs 3 * D features and splits the result. The two are mathematically equivalent because three separate matrix multiplications by (D, D) weights are exactly one matrix multiplication by a (3D, D) weight stacked from them.

The efficient version is faster because the accelerator launches one matmul instead of three. It is also easier to initialize because the three sub-matrices live in the same parameter tensor and can be initialized together.

The head reshape

After the split, each of Q, K, V is (B, T, D). To turn that into H parallel attention problems, we reshape to (B, T, H, d_head) and transpose to (B, H, T, d_head). The head dimension now sits next to the batch dimension so PyTorch treats the per-head attention as a batched operation across B * H independent instances.

The d_head dimension stays last so the score matmul Q @ K.transpose(-2, -1) contracts it. The result is (B, H, T, T) per-head attention scores.

Scaling

The scores get divided by sqrt(d_head) before softmax. Without that scaling, dot products grow as d_head grows and push the softmax into a regime where one entry has almost all the mass and the others are vanishingly small. The gradients in that regime are tiny and learning stalls. Dividing by sqrt(d_head) keeps the variance of the scores roughly constant across head sizes.

The causal mask

A decoder-only language model can only condition on the past when predicting the next token. The mask enforces that. Concretely, before the softmax, every entry above the diagonal of the (T, T) score matrix gets replaced by negative infinity. After softmax those positions get weight zero.

We register the mask as a buffer at construction so it lives on the same device as the model and is not part of the gradient graph. The mask covers the maximum context length the block will ever see. At forward time we slice the top-left (T, T) corner.

The output projection

After per-head context vectors (B, H, T, d_head)we transpose back to (B, T, H, d_head)reshape to (B, T, D)and apply a final (D, D) linear projection. The output projection lets the model mix the heads. Without it, the H heads would only ever recombine through later layers and the block would be artificially constrained.

Attention weight inspection

The lesson exposes a return_weights=True flag on the forward pass. When set, the block returns the per-head attention weights of shape (B, H, T, T) alongside the output. The demo prints a heatmap of one head's weights on a short input so you can see the causal-triangle structure and the per-position focus.

In a trained model, different heads learn different patterns. Some heads attend to the immediately previous token. Some heads attend to the start of the sequence. Some heads spread attention almost uniformly. The inspection hook is the entry point for that interpretability work.

The training demo

The demo at the bottom of main.py wires the attention block to a tiny LM head and trains the whole thing on a repeat task. Each row of the input is a single random id replicated across the context. The target is the input shifted by one, so the model must learn that the next token is the same as the previous token. The loss is cross-entropy. With H=4, D=32, T=12, and a vocabulary of 64, the loss falls from random (around log(64) ~ 4.16) down to well under 1.0 over three epochs on CPU.

The point of the demo is not to train a useful model. The point is to confirm the gradients flow through every piece of the block and the heads learn something on a problem where the answer is obvious.

What this lesson does not do

It does not add a feed-forward block. The transformer layer in a real model is attention followed by a two-layer MLP with a residual connection and layer norm around each. The next lesson adds those.

It does not implement rotary or AliBi positional encoding. Both apply at the QKV projection step in the same block, but they are a separate teaching unit. The block as built here is compatible with either by transforming Q and K before the matmul.

It does not implement KV cache for inference. Caching keys and values across forward passes is the optimization that makes autoregressive decoding fast. It changes the shape contract on the K and V tensors but not on Q. It belongs in the inference lesson.

How to read the code

main.py defines MultiHeadSelfAttention. The class holds two linear layers and a registered mask buffer. The forward pass projects, reshapes, scores, masks, softmaxes, weights, reshapes, and projects again. The demo at the bottom builds a small model that wraps the attention with token and positional embeddings and an LM head, trains it on a copy task for three epochs, and prints the loss curve and a per-head attention heatmap. The tests in code/tests/test_attention.py pin the shape contract, the causality property, the softmax property, the head-split property, and the gradient flow.

Run the demo. Then increase n_heads from 4 to 8 (keeping d_model=32so d_head=4) and watch the heatmap change.

Check what you learned

6 questions. Answer each one to see why.

Question 1 of 60 correct

Why does an efficient self-attention block use a single linear layer that outputs 3*D features?

The full code

Every file for this lesson. Copy it, run it, then change it.

main.py
"""Multi-head self-attention with causal mask, single QKV projection, and
weight inspection.

The demo trains a tiny model (attention + token/positional embeddings + LM head)
on a copy task and prints the loss curve plus a per-head attention heatmap.

Run: python3 code/main.py
"""

from __future__ import annotations

import math
from dataclasses import dataclass

import torch
import torch.nn as nn
import torch.nn.functional as F


class MultiHeadSelfAttention(nn.Module):
    """Multi-head self-attention with a single QKV linear and a causal mask."""

    def __init__(
        self,
        d_model: int,
        n_heads: int,
        max_context_length: int,
        attn_dropout: float = 0.0,
        out_dropout: float = 0.0,
    ) -> None:
        super().__init__()
        if d_model < 1:
            raise ValueError(f"d_model must be >= 1, got {d_model}")
        if n_heads < 1:
            raise ValueError(f"n_heads must be >= 1, got {n_heads}")
        if d_model % n_heads != 0:
            raise ValueError(
                f"d_model ({d_model}) must be divisible by n_heads ({n_heads})"
            )
        if max_context_length < 1:
            raise ValueError(f"max_context_length must be >= 1, got {max_context_length}")
        self.d_model = d_model
        self.n_heads = n_heads
        self.d_head = d_model // n_heads
        self.max_context_length = max_context_length

        self.qkv_proj = nn.Linear(d_model, 3 * d_model, bias=True)
        self.out_proj = nn.Linear(d_model, d_model, bias=True)
        self.attn_dropout = nn.Dropout(attn_dropout)
        self.out_dropout = nn.Dropout(out_dropout)

        causal_mask = torch.tril(torch.ones(max_context_length, max_context_length))
        self.register_buffer("causal_mask", causal_mask, persistent=False)

    def _split_heads(self, x: torch.Tensor) -> torch.Tensor:
        b, t, _ = x.shape
        return x.view(b, t, self.n_heads, self.d_head).transpose(1, 2)

    def _merge_heads(self, x: torch.Tensor) -> torch.Tensor:
        b, h, t, dh = x.shape
        return x.transpose(1, 2).contiguous().view(b, t, h * dh)

    def forward(
        self,
        x: torch.Tensor,
        return_weights: bool = False,
    ) -> torch.Tensor | tuple[torch.Tensor, torch.Tensor]:
        if x.dim() != 3:
            raise ValueError(f"input must be (B, T, D), got shape {tuple(x.shape)}")
        b, t, d = x.shape
        if d != self.d_model:
            raise ValueError(f"feature dim {d} != d_model {self.d_model}")
        if t > self.max_context_length:
            raise ValueError(f"seq_len {t} exceeds max_context_length {self.max_context_length}")

        qkv = self.qkv_proj(x)
        q, k, v = qkv.chunk(3, dim=-1)

        q = self._split_heads(q)
        k = self._split_heads(k)
        v = self._split_heads(v)

        scores = torch.matmul(q, k.transpose(-2, -1)) / math.sqrt(self.d_head)
        mask_slice = self.causal_mask[:t, :t]
        scores = scores.masked_fill(mask_slice == 0, float("-inf"))

        weights = F.softmax(scores, dim=-1)
        weights = self.attn_dropout(weights)

        context = torch.matmul(weights, v)
        context = self._merge_heads(context)

        out = self.out_proj(context)
        out = self.out_dropout(out)

        if return_weights:
            return out, weights
        return out


class TokenEmbedding(nn.Module):
    """Vocab id to vector lookup (compact copy of lesson 32)."""

    def __init__(self, vocab_size: int, d_model: int) -> None:
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, d_model)
        with torch.no_grad():
            self.embedding.weight.normal_(0.0, 0.02)

    def forward(self, ids: torch.Tensor) -> torch.Tensor:
        return self.embedding(ids)


class SinusoidalPositionalEmbedding(nn.Module):
    """Parameter-free sin/cos positional table (compact copy of lesson 32)."""

    def __init__(self, max_context_length: int, d_model: int, base: float = 10000.0) -> None:
        super().__init__()
        if max_context_length < 1:
            raise ValueError(f"max_context_length must be >= 1, got {max_context_length}")
        if d_model < 1:
            raise ValueError(f"d_model must be >= 1, got {d_model}")
        if d_model % 2 != 0:
            raise ValueError(f"d_model must be even, got {d_model}")
        self.max_context_length = max_context_length
        pos = torch.arange(max_context_length, dtype=torch.float32).unsqueeze(1)
        i = torch.arange(d_model // 2, dtype=torch.float32)
        denom = base ** (2 * i / d_model)
        angle = pos / denom
        pe = torch.zeros(max_context_length, d_model, dtype=torch.float32)
        pe[:, 0::2] = torch.sin(angle)
        pe[:, 1::2] = torch.cos(angle)
        self.register_buffer("pe", pe, persistent=False)

    def forward(self, seq_len: int) -> torch.Tensor:
        if seq_len < 1:
            raise ValueError(f"seq_len must be >= 1, got {seq_len}")
        if seq_len > self.max_context_length:
            raise ValueError(
                f"seq_len {seq_len} exceeds max_context_length {self.max_context_length}"
            )
        return self.pe[:seq_len]


class TinyAttentionLM(nn.Module):
    """Embedding + attention + LM head. Just enough to train a copy task."""

    def __init__(
        self,
        vocab_size: int,
        d_model: int,
        n_heads: int,
        max_context_length: int,
    ) -> None:
        super().__init__()
        self.token_emb = TokenEmbedding(vocab_size, d_model)
        self.pos_emb = SinusoidalPositionalEmbedding(max_context_length, d_model)
        self.attn = MultiHeadSelfAttention(
            d_model=d_model,
            n_heads=n_heads,
            max_context_length=max_context_length,
        )
        self.lm_head = nn.Linear(d_model, vocab_size)

    def forward(
        self,
        ids: torch.Tensor,
        return_weights: bool = False,
    ) -> torch.Tensor | tuple[torch.Tensor, torch.Tensor]:
        b, t = ids.shape
        tok = self.token_emb(ids)
        pos = self.pos_emb(t)
        x = tok + pos.unsqueeze(0)
        if return_weights:
            attn_out, weights = self.attn(x, return_weights=True)
            logits = self.lm_head(attn_out)
            return logits, weights
        attn_out = self.attn(x)
        logits = self.lm_head(attn_out)
        return logits


@dataclass
class DemoConfig:
    vocab_size: int = 64
    d_model: int = 32
    n_heads: int = 4
    seq_len: int = 12
    batch_size: int = 16
    n_epochs: int = 3
    steps_per_epoch: int = 120
    learning_rate: float = 5e-3
    seed: int = 42


def _make_repeat_batch(cfg: DemoConfig, generator: torch.Generator) -> tuple[torch.Tensor, torch.Tensor]:
    """Repeat task. Pick a random id per row, repeat it across the row.

    The model must learn that the next token is the same as the previous one.
    A single attention head looking one token back is enough to solve it.
    """
    base = torch.randint(
        0, cfg.vocab_size, (cfg.batch_size, 1), generator=generator, dtype=torch.long
    )
    ids = base.expand(cfg.batch_size, cfg.seq_len + 1).contiguous()
    return ids[:, :-1], ids[:, 1:]


def _train(model: TinyAttentionLM, cfg: DemoConfig) -> list[float]:
    optimizer = torch.optim.Adam(model.parameters(), lr=cfg.learning_rate)
    generator = torch.Generator()
    generator.manual_seed(cfg.seed)
    loss_curve: list[float] = []
    for epoch in range(cfg.n_epochs):
        total = 0.0
        for _ in range(cfg.steps_per_epoch):
            inputs, targets = _make_repeat_batch(cfg, generator)
            logits = model(inputs)
            loss = F.cross_entropy(logits.reshape(-1, cfg.vocab_size), targets.reshape(-1))
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
            total += loss.item()
        avg = total / cfg.steps_per_epoch
        loss_curve.append(avg)
        print(f"epoch {epoch + 1}/{cfg.n_epochs}  avg loss: {avg:.4f}")
    return loss_curve


def _print_section(title: str) -> None:
    bar = "-" * len(title)
    print(f"\n{title}\n{bar}")


def _heatmap_row(row: torch.Tensor, width: int = 28) -> str:
    glyphs = " .:-=+*#%@"
    cells: list[str] = []
    for v in row.tolist():
        idx = int(min(len(glyphs) - 1, max(0, v * len(glyphs))))
        cells.append(glyphs[idx])
    return "".join(cells[:width])


def main() -> int:
    cfg = DemoConfig()
    torch.manual_seed(cfg.seed)

    _print_section("Shape contract")
    attn = MultiHeadSelfAttention(
        d_model=cfg.d_model,
        n_heads=cfg.n_heads,
        max_context_length=cfg.seq_len,
    )
    x = torch.randn(cfg.batch_size, cfg.seq_len, cfg.d_model)
    out = attn(x)
    print(f"input  : {tuple(x.shape)}")
    print(f"output : {tuple(out.shape)}")
    assert out.shape == x.shape

    _print_section("Causal mask check")
    out_with_weights, weights = attn(x, return_weights=True)
    upper = torch.triu(torch.ones(cfg.seq_len, cfg.seq_len), diagonal=1).bool()
    upper_mass = weights[0, 0][upper].abs().sum().item()
    print(f"weights shape          : {tuple(weights.shape)}")
    print(f"sum over future cells  : {upper_mass:.6f}")
    assert upper_mass < 1e-5, "future positions must have zero weight"
    rows = weights[0, 0].sum(dim=-1)
    print(f"row sums (head 0, batch 0): min={rows.min().item():.4f}, max={rows.max().item():.4f}")

    _print_section("Train tiny model on repeat task")
    model = TinyAttentionLM(
        vocab_size=cfg.vocab_size,
        d_model=cfg.d_model,
        n_heads=cfg.n_heads,
        max_context_length=cfg.seq_len,
    )
    initial_loss = math.log(cfg.vocab_size)
    print(f"random-init expected loss ~ log(V) = {initial_loss:.4f}")
    curve = _train(model, cfg)
    assert curve[-1] < curve[0], "loss must fall over training"

    _print_section("Per-head attention heatmap")
    model.eval()
    with torch.no_grad():
        sample_gen = torch.Generator()
        sample_gen.manual_seed(cfg.seed + 1)
        base = torch.randint(0, cfg.vocab_size, (1, 1), generator=sample_gen, dtype=torch.long)
        sample_ids = base.expand(1, cfg.seq_len).contiguous()
        _, sample_weights = model(sample_ids, return_weights=True)
    head_id = 0
    print(f"head {head_id}, query rows top-down, key cols left-to-right")
    for t in range(cfg.seq_len):
        row = sample_weights[0, head_id, t]
        print(f"  q={t:>2}: |{_heatmap_row(row, width=cfg.seq_len)}|")

    print("\nDemo OK.")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())