Get Started

GPT Model Assembly

Twelve blocks stacked, a token embedding, a learned position embedding, a final LayerNorm, and a tied language model head. That is the entire 124 million parameter GPT model. This lesson assembles those pieces into a working class, counts the parameters to confirm the model matches the reference 124M shape, and generates text with multinomial sampling, temperature, and top-k.

90 minutesBuildPythonFree lesson

Learning Objectives

  • Assemble the transformer block from lesson 34 into a full GPT model: token embedding, position embedding, N blocks, final LayerNorm, language model head.
  • Reproduce the 124 million parameter configuration: vocab 50257, context 1024, embedding 768, twelve heads, twelve layers.
  • Tie the language model head weights to the token embedding and explain why that saves ~38 million parameters at this scale.
  • Generate text from a prompt with multinomial sampling, temperature scaling, and top-k truncation, holding context length with a sliding window.
  • Measure parameter count and forward pass cost against the 124M target.

The Problem

A transformer block does nothing on its own. You need to turn token ids into vectors, mix in positional information, run them through the stack, and project back to vocabulary logits. Forget any one of those four steps and the model either fails to forward, drifts in position information, or cannot speak.

The shape of the model also matters. The reference GPT-2 small is 124 million parameters at exactly the configuration above. The numbers are not magic. Vocab 50257 times embedding 768 is the token table. Position 1024 times 768 is the position table. Twelve blocks at roughly 7 million parameters each is 84 million. The final head reuses the token table by weight tying. Sum the pieces and you land on 124 million. Building a model whose parameter count does not match the reference is a sign you wired something wrong.

The Concept

Token ids become token vectors. Position ids become position vectors. The two are added and sent through the stack. The final LayerNorm is the one piece outside the blocks that survives every modern variant. The LM head reuses the token embedding matrix, which is what weight tying means.

Weight tying

The token embedding has shape (vocab, d_model). The language model head needs to project from d_model back to vocab. Those are transposes of each other. Tying the two means literally the same parameter tensor, used twice. At vocab 50257 and d_model 768, the matrix is 38 million parameters. Untied, you pay for it twice. Tied, you pay for it once and you also get a slightly cleaner gradient signal because the embedding and head update together.

Position embedding is learned, not sinusoidal

GPT-2 ships a learned position embedding. The position table is one parameter tensor of shape (1024, 768). The model looks up position 0 through T-1 at every forward and adds the lookup to the token embedding. This is the simplest of the position schemes (RoPE, ALiBi, T5 relative bias are the alternatives) and it is what the 124M reference uses.

Generation: temperature, top-k, multinomial

Generation is autoregressive. At every step, the model returns logits over the full vocabulary at every position. You take the last position only, divide by temperature, optionally mask all but the top k logits to negative infinity, softmax to get probabilities, and sample one token from the resulting distribution.

Three knobs, three different behaviors. Temperature near zero collapses to greedy. Temperature one matches the model's natural distribution. Top-k one is greedy. Top-k forty filters the long tail. The combinations matter; the next lesson on training uses generation as a qualitative eval signal.

Build It

code/main.py implements:

  • class GPTConfig dataclass with the 124M defaults: vocab_size=50257context_length=1024d_model=768num_heads=12num_layers=12mlp_expansion=4dropout=0.1use_bias=Trueweight_tying=True.
  • class GPTModel with token embedding, position embedding, embedding dropout, twelve TransformerBlocks, final LayerNorm, and an lm_head that ties to the token embedding when the flag is set.
  • A count_parameters helper that returns the unique parameter count (so weight tying is honored in the count).
  • A generate function that does temperature, top-k, multinomial, and sliding window context.
  • A demo that builds the model, prints the parameter count next to the reference 124M, and generates a short sequence from a fixed prompt to show the pipeline ends to end.

Run it:

Terminal
python3 code/main.py

Output: parameter count alongside the 124M reference, generated token ids from a random prompt, and a confirmation that the LM head and token embedding share storage when tying is on.

To keep the demo fast, the script also runs a tiny config (d_model=64num_layers=2) end to end and prints the generated token sequence inline. The 124M config is built but only its parameter count and one forward pass are exercised.

Stack

  • torch for the tensor math, autograd, and module plumbing.
  • code/main.py reimplements the same block pattern from lesson 34 locally.

Production patterns in the wild

Three patterns make the difference between a model that runs and a model that ships.

Initialize the residual projections small. The output projection of attention and the second linear of the MLP both feed directly into a residual add. Initializing those with the same standard deviation as every other linear gives a residual stream that grows with depth and pushes the final LayerNorm into a hot regime. Scale the std by 1 / sqrt(2 * num_layers) for those two projections; the residual stream stays in a sane range through twelve layers.

Cache the position id tensor, do not recompute. torch.arange(T) allocates fresh memory at every forward. Allocate once in __init__ for the maximum context, slice the first T entries per call, and skip the allocator round trip.

Tie weights at parameter level, not just by copying. Setting lm_head.weight = token_embedding.weight shares the tensor; copying does not. The optimizer needs to update one parameter and the autograd graph needs one accumulation. If you copy, the head drifts away from the embedding and weight tying buys you nothing.

Use It

  • The model class in this lesson is the same shape as the one the next lesson trains.
  • Replacing the learned position embedding with RoPE gets you the LLaMA family without touching the block or the head.
  • Replacing the GELU with SiLU and the LayerNorm with RMSNorm gets you the rest of the LLaMA family changes.
  • The generation function works with any logits source, not only this model. You can pull logits from a pretrained GPT-2 file in lesson 37 and reuse the same generation loop.

Exercises

  1. Untie the LM head from the token embedding and recount parameters. Verify the delta is 50257 times 768 = 38 million.
  2. Replace the learned position embedding with a sinusoidal table computed at construction time. Confirm the model still forwards and the parameter count drops by 786,432.
  3. Add a greedy=True flag to generation that skips sampling and picks argmax. Confirm the sequence is deterministic across runs.
  4. Add a repetition_penalty knob that divides the logit of any token in the prompt or generated history by a constant before softmax. Show on a fixed prompt that values above one reduce repeat counts in the output.
  5. Add top_p (nucleus) sampling next to top_k. Two-line check that the sum of probabilities of the kept tokens exceeds top_p.

Key Terms

TermWhat people sayWhat it actually means
Weight tying"Tied embeddings"The LM head and the token embedding share the same parameter tensor; saves vocab times d_model parameters and matches the GPT-2 reference
Position embedding"Learned positions"A separate table of shape (context length, d_model) added to token vectors; learned end to end
Sliding window context"Context cap"When the prompt plus generated tokens exceed the context length, drop the oldest tokens so the active window fits
Top-k sampling"K truncation"Keep the K logits with the highest values, mask the rest to negative infinity, softmax over the remainder
Temperature"Sampling temperature"Divide logits by T before softmax; T less than 1 sharpens, T equal to 1 keeps the natural distribution, T greater than 1 flattens

Further Reading

  • Phase 19 lesson 34 for the block this model stacks.
  • Phase 19 lesson 36 for the training loop that drives this model with cross entropy loss.
  • Phase 19 lesson 37 for loading pretrained GPT-2 weights into this exact architecture.
  • Phase 7 lesson 07 (GPT causal language modeling) for the math of next token prediction.
  • Phase 10 lesson 04 (pre training mini GPT) for the original training procedure on the same architecture.

Check what you learned

6 questions. Answer each one to see why.

Question 1 of 60 correct

What is the reference GPT-2 small configuration this lesson reproduces?

The full code

Every file for this lesson. Copy it, run it, then change it.

main.py
"""Assemble the lesson 34 transformer block into a 124M parameter GPT model.

Twelve blocks, a token embedding, a learned position embedding, a final LayerNorm,
and a language model head that ties to the token embedding. Parameter count
lands on ~124M at the reference configuration. The demo also runs a tiny
configuration end to end and exercises generation with temperature, top-k, and
multinomial sampling under a sliding window context.

Run: python3 code/main.py
"""

from __future__ import annotations

import math
from dataclasses import dataclass

import torch
import torch.nn as nn
import torch.nn.functional as F


@dataclass
class GPTConfig:
    """Reference 124M configuration matches the GPT-2 small architecture."""

    vocab_size: int = 50257
    context_length: int = 1024
    d_model: int = 768
    num_heads: int = 12
    num_layers: int = 12
    mlp_expansion: int = 4
    dropout: float = 0.1
    use_bias: bool = True
    weight_tying: bool = True


class LayerNorm(nn.Module):
    def __init__(self, d_model: int, eps: float = 1e-5) -> None:
        super().__init__()
        self.eps = eps
        self.scale = nn.Parameter(torch.ones(d_model))
        self.shift = nn.Parameter(torch.zeros(d_model))

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        mean = x.mean(dim=-1, keepdim=True)
        var = x.var(dim=-1, keepdim=True, unbiased=False)
        return self.scale * (x - mean) / torch.sqrt(var + self.eps) + self.shift


class MultiHeadAttention(nn.Module):
    def __init__(self, cfg: GPTConfig) -> None:
        super().__init__()
        if cfg.d_model % cfg.num_heads != 0:
            raise ValueError("d_model must be divisible by num_heads")
        self.d_model = cfg.d_model
        self.num_heads = cfg.num_heads
        self.head_dim = cfg.d_model // cfg.num_heads
        self.context_length = cfg.context_length

        self.qkv = nn.Linear(cfg.d_model, 3 * cfg.d_model, bias=cfg.use_bias)
        self.out_proj = nn.Linear(cfg.d_model, cfg.d_model, bias=cfg.use_bias)
        self.attn_dropout = nn.Dropout(cfg.dropout)
        self.resid_dropout = nn.Dropout(cfg.dropout)

        mask = torch.triu(
            torch.ones(cfg.context_length, cfg.context_length, dtype=torch.bool),
            diagonal=1,
        )
        self.register_buffer("causal_mask", mask, persistent=False)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        batch, seq, dim = x.shape
        if seq > self.context_length:
            raise ValueError(
                f"sequence length {seq} exceeds context length {self.context_length}"
            )
        qkv = self.qkv(x)
        q, k, v = qkv.split(self.d_model, dim=-1)
        q = q.view(batch, seq, self.num_heads, self.head_dim).transpose(1, 2)
        k = k.view(batch, seq, self.num_heads, self.head_dim).transpose(1, 2)
        v = v.view(batch, seq, self.num_heads, self.head_dim).transpose(1, 2)
        scores = q @ k.transpose(-2, -1) / math.sqrt(self.head_dim)
        mask = self.causal_mask[:seq, :seq]
        scores = scores.masked_fill(mask, float("-inf"))
        attn = F.softmax(scores, dim=-1)
        attn = self.attn_dropout(attn)
        out = attn @ v
        out = out.transpose(1, 2).contiguous().view(batch, seq, dim)
        out = self.out_proj(out)
        out = self.resid_dropout(out)
        return out


class FeedForward(nn.Module):
    def __init__(self, cfg: GPTConfig) -> None:
        super().__init__()
        hidden = cfg.mlp_expansion * cfg.d_model
        self.fc1 = nn.Linear(cfg.d_model, hidden, bias=cfg.use_bias)
        self.act = nn.GELU(approximate="tanh")
        self.fc2 = nn.Linear(hidden, cfg.d_model, bias=cfg.use_bias)
        self.dropout = nn.Dropout(cfg.dropout)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        return self.dropout(self.fc2(self.act(self.fc1(x))))


class TransformerBlock(nn.Module):
    """Pre-LN block. Lesson 34 explains both configurations; the GPT-2 reference is pre-LN."""

    def __init__(self, cfg: GPTConfig) -> None:
        super().__init__()
        self.ln1 = LayerNorm(cfg.d_model)
        self.attn = MultiHeadAttention(cfg)
        self.ln2 = LayerNorm(cfg.d_model)
        self.mlp = FeedForward(cfg)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        x = x + self.attn(self.ln1(x))
        x = x + self.mlp(self.ln2(x))
        return x


class GPTModel(nn.Module):
    """A decoder only transformer language model with weight tied LM head."""

    def __init__(self, cfg: GPTConfig) -> None:
        super().__init__()
        self.cfg = cfg
        self.tok_embed = nn.Embedding(cfg.vocab_size, cfg.d_model)
        self.pos_embed = nn.Embedding(cfg.context_length, cfg.d_model)
        self.embed_dropout = nn.Dropout(cfg.dropout)
        self.blocks = nn.ModuleList([TransformerBlock(cfg) for _ in range(cfg.num_layers)])
        self.final_ln = LayerNorm(cfg.d_model)
        self.lm_head = nn.Linear(cfg.d_model, cfg.vocab_size, bias=False)

        if cfg.weight_tying:
            self.lm_head.weight = self.tok_embed.weight

        position_ids = torch.arange(cfg.context_length, dtype=torch.long)
        self.register_buffer("position_ids", position_ids, persistent=False)

        self.apply(self._init_weights)
        self._scale_residual_projections()

    def _init_weights(self, module: nn.Module) -> None:
        if isinstance(module, nn.Linear):
            nn.init.normal_(module.weight, mean=0.0, std=0.02)
            if module.bias is not None:
                nn.init.zeros_(module.bias)
        elif isinstance(module, nn.Embedding):
            nn.init.normal_(module.weight, mean=0.0, std=0.02)

    def _scale_residual_projections(self) -> None:
        scale = 1.0 / math.sqrt(2 * self.cfg.num_layers)
        for block in self.blocks:
            block.attn.out_proj.weight.data.mul_(scale)
            block.mlp.fc2.weight.data.mul_(scale)

    def forward(self, tokens: torch.Tensor) -> torch.Tensor:
        batch, seq = tokens.shape
        if seq > self.cfg.context_length:
            raise ValueError(
                f"sequence length {seq} exceeds context length {self.cfg.context_length}"
            )
        tok = self.tok_embed(tokens)
        pos = self.pos_embed(self.position_ids[:seq])
        x = self.embed_dropout(tok + pos)
        for block in self.blocks:
            x = block(x)
        x = self.final_ln(x)
        logits = self.lm_head(x)
        return logits


def count_parameters(model: nn.Module) -> int:
    """Count unique parameters. Weight tied tensors are counted once."""
    seen: dict[int, int] = {}
    for param in model.parameters():
        seen[id(param)] = param.numel()
    return sum(seen.values())


def top_k_filter(logits: torch.Tensor, top_k: int) -> torch.Tensor:
    if top_k is None or top_k <= 0:
        return logits
    top_k = min(top_k, logits.size(-1))
    values, _ = torch.topk(logits, top_k, dim=-1)
    threshold = values[..., -1:]
    return torch.where(logits < threshold, torch.full_like(logits, float("-inf")), logits)


def generate(
    model: GPTModel,
    prompt: torch.Tensor,
    max_new_tokens: int,
    temperature: float = 1.0,
    top_k: int | None = None,
    seed: int | None = None,
) -> torch.Tensor:
    """Autoregressive generation with multinomial sampling, temperature, top-k.

    Holds the active window to model.cfg.context_length by sliding the oldest
    tokens out when the running sequence overflows.
    """
    if temperature <= 0:
        raise ValueError("temperature must be positive")
    if seed is not None:
        torch.manual_seed(seed)

    was_training = model.training
    model.eval()
    tokens = prompt.clone()
    try:
        with torch.no_grad():
            for _ in range(max_new_tokens):
                window = tokens[:, -model.cfg.context_length:]
                logits = model(window)
                next_logits = logits[:, -1, :] / temperature
                next_logits = top_k_filter(next_logits, top_k)
                probs = F.softmax(next_logits, dim=-1)
                next_token = torch.multinomial(probs, num_samples=1)
                tokens = torch.cat([tokens, next_token], dim=1)
        return tokens
    finally:
        model.train(was_training)


def demo() -> None:
    torch.manual_seed(0)

    print("Building 124M reference GPT...")
    ref_cfg = GPTConfig()
    ref_model = GPTModel(ref_cfg)
    ref_params = count_parameters(ref_model)
    print(f"  reference params         : {ref_params:,}")
    print(f"  expected near 124M       : within 5% target {abs(ref_params - 124_000_000) / 124_000_000:.2%}")

    head_tied = ref_model.lm_head.weight.data_ptr() == ref_model.tok_embed.weight.data_ptr()
    print(f"  weight tying enforced    : {head_tied}")
    assert head_tied, "weight tying should share storage"

    print("\nUntying and re-counting to confirm the 38M delta...")
    untied_cfg = GPTConfig(weight_tying=False)
    untied_model = GPTModel(untied_cfg)
    untied_params = count_parameters(untied_model)
    delta = untied_params - ref_params
    expected_delta = ref_cfg.vocab_size * ref_cfg.d_model
    print(f"  untied params            : {untied_params:,}")
    print(f"  delta                    : {delta:,}")
    print(f"  expected (vocab*d_model) : {expected_delta:,}")
    assert delta == expected_delta

    print("\nSingle forward through 124M reference, batch 1, seq 32...")
    tokens = torch.randint(0, ref_cfg.vocab_size, (1, 32))
    with torch.no_grad():
        logits = ref_model(tokens)
    print(f"  logits shape             : {tuple(logits.shape)}")
    assert logits.shape == (1, 32, ref_cfg.vocab_size)

    print("\nGenerating with a tiny model end to end (faster demo)...")
    tiny_cfg = GPTConfig(
        vocab_size=512,
        context_length=64,
        d_model=64,
        num_heads=4,
        num_layers=2,
        dropout=0.0,
    )
    tiny_model = GPTModel(tiny_cfg)
    tiny_params = count_parameters(tiny_model)
    print(f"  tiny params              : {tiny_params:,}")

    prompt = torch.tensor([[1, 2, 3, 4, 5]], dtype=torch.long)
    generated = generate(
        tiny_model,
        prompt,
        max_new_tokens=12,
        temperature=0.8,
        top_k=20,
        seed=42,
    )
    print(f"  prompt                   : {prompt.tolist()[0]}")
    print(f"  generated tokens         : {generated.tolist()[0]}")
    assert generated.shape == (1, prompt.shape[1] + 12)

    print("\nSliding window check: prompt longer than context...")
    long_prompt = torch.randint(0, tiny_cfg.vocab_size, (1, 80))
    generated_long = generate(tiny_model, long_prompt, max_new_tokens=4, temperature=1.0, top_k=10, seed=0)
    print(f"  long prompt shape        : {tuple(long_prompt.shape)}")
    print(f"  generated shape          : {tuple(generated_long.shape)}")
    assert generated_long.shape == (1, 84)
    print("\nModel assembly check passed.")


if __name__ == "__main__":
    demo()