Transformer Block from Scratch
One block is the unit of every modern decoder LLM. Layer norm, multi head attention, residual, MLP, residual. The pre-LN variant trains stably without warmup. The post-LN variant is what the original paper shipped. This lesson builds both, side by side, and shows which one survives a 12 layer stack at common learning rates.
Learning Objectives
- Build a transformer block in PyTorch from the four moving pieces: LayerNorm, multi head causal attention, residual connections, position wise MLP.
- Place the LayerNorms in two configurations (pre-LN and post-LN) and explain why one trains stably without warmup.
- Implement causal masking inside the multi head attention so token
icannot see tokensj > i. - Track gradient flow through both variants on a 12 layer stack and read the result without hand waving.
- Reuse the block as a drop-in unit when the next lesson assembles a 124 million parameter GPT.
The Problem
A transformer is one block repeated. Get the block wrong once, repeat it twelve times, and you ship a model that diverges in the first epoch or that needs warmup hacks the rest of the way. The two failure modes you will see in this lesson are not exotic. They show up the first time a learner stacks blocks naively. One is the attention layer attending to the future. The other is the LayerNorm placed where it cannot tame the residual signal at depth.
The fix is mechanical once you see it. The block has exactly two residual paths and exactly two normalization positions. Choose the positions correctly and the rest of the stack is just bookkeeping.
The Concept
Every decoder only transformer block is a function that takes a tensor of shape (batch, sequence, embedding) and returns a tensor of the same shape. Inside, two sublayers do the work.
This is the pre-LN variant. The LayerNorm sits inside the residual branch, before the sublayer. The residual connection carries the unnormalized signal forward.
The post-LN variant moves the LayerNorm to after the residual add.
Shape is identical. Training behavior is not. With post-LN, the gradient that flows back through the residual path must pass through the LayerNorm. At depth twelve and learning rate 3e-4that gradient shrinks fast enough to need a warmup schedule. Pre-LN leaves the residual path unnormalized, so gradients propagate cleanly to the embedding layer. Pre-LN is the configuration GPT-2 onward ships with for that reason.
Causal multi head attention
The attention sublayer projects the input three ways into query, key, value tensors. Each is reshaped from (B, T, D) to (B, H, T, D/H) where H is the head count. Scaled dot product attention computes softmax(Q K^T / sqrt(d_k)) per head, masks the upper triangle to negative infinity, applies the mask via softmax, then multiplies by V. Heads are concatenated back into a single (B, T, D) tensor and projected once more. The mask is the only piece that makes the model causal. Forget the mask and you train a model that cheats.
The MLP
The position wise MLP applies the same two layer network to every token independently. The hidden width is four times the embedding width, the activation is GELU, and a dropout follows the second linear. No tokens talk to each other inside the MLP. All token mixing happens in attention.
Residual connections do two things
They make the gradient path additive across depth, which keeps the gradient norm in scale through twelve layers. They also let each block learn an additive update to the running representation rather than a full replacement. Both effects are why the block scales.
Build It
code/main.py implements:
class LayerNormwith learnable scale and shift, biased eps, applied per token vector.class MultiHeadAttentionwithnum_headshead_dim = d_model // num_headsfused QKV projection, registered causal mask, attention and residual dropout.class FeedForwardwith two linear layers, GELU activation, dropout.class TransformerBlockwith apre_lnflag that toggles between the two variants.- A demo that builds a 6 layer pre-LN stack and a 6 layer post-LN stack with identical inputs and prints (a) output shape, (b) gradient norm at the embedding after one backward pass.
Run it:
python3 code/main.pyOutput: shape check on both stacks, gradient norms side by side. The pre-LN stack's embedding gradient is order of magnitude larger than the post-LN stack at the same learning rate, which is the empirical signal pre-LN trains without warmup.
Stack
torchfor the tensor math, autograd, andnn.Moduleplumbing.- No
transformersno pretrained weights. The block is implemented from primitives.
Production patterns in the wild
Three patterns turn the textbook block into something you can ship.
Fused QKV projection. Three separate linear layers cost three kernel launches and three matmuls. One linear layer of width 3 * d_model does the same work in one launch, then splits the output along the last axis. The fused path is faster on every accelerator and matches what reference implementations of GPT-2, LLaMA, and Mistral all ship.
Registered causal mask buffer. The mask depends only on the maximum context length. Allocate it once at construction with register_bufferslice the active window per forward pass, and skip the per-call allocation. Forgetting this turns the mask into an allocator hot spot at long context.
Dropout in two places, not three. Dropout belongs after the attention softmax (attention dropout) and after the second linear of the MLP (residual dropout). A dropout on the residual itself breaks the additive identity that lets the gradient flow at depth. Some early implementations got this wrong and paid for it with brittle training.
Use It
- The block in this lesson plugs straight into the GPT assembly in lesson 35 without modification.
- The pre-LN variant is what every modern open weights LLM uses. The post-LN variant is what the original 2017 attention paper used. Knowing both is enough to read any decoder architecture you will encounter.
- Swap the GELU for SiLU and you have the LLaMA family activation. Swap the LayerNorm for RMSNorm and you have the LLaMA family normalization. Same skeleton.
Exercises
- Add a
bias=Falseflag to every linear in the block. Modern open weights LLMs ship without biases on the linear layers. Measure how many parameters you save in a 12 layer 768 dim model. - Replace
nn.LayerNormwith a hand rolled RMSNorm and verify the output shape is unchanged. - Add a flag that returns the attention weights for the first head as a
(B, T, T)tensor. Plot the upper triangle to confirm it is zero after softmax. - Build a sanity check that feeds a
(2, 16, 384)tensor withH=6through both variants and asserts the forward outputs are different (for example,not torch.allclose) when weights are initialized identically and dropout is set to zero.
Key Terms
| Term | What people say | What it actually means |
|---|---|---|
| Pre-LN | "Pre norm" | LayerNorm inside the residual branch, before each sublayer; the residual carries the unnormalized signal |
| Post-LN | "Post norm" | LayerNorm after the residual add; what the 2017 paper shipped and what needs warmup |
| Causal mask | "Triangle mask" | The upper triangle of the attention logits set to negative infinity so token i cannot read token j when j is greater than i |
| Fused QKV | "Combined projection" | One linear of width 3D instead of three linears of width D; one kernel, one matmul |
| Residual stream | "Skip connection" | The unnormalized tensor that flows top to bottom through every block; what each block adds to |
Further Reading
- Phase 7 lesson 02 (self attention from scratch) for the attention math underneath this block.
- Phase 7 lesson 05 (full transformer) for the encoder decoder version of the same skeleton.
- Phase 10 lesson 04 (pre training mini GPT) for the training procedure that this block plugs into.
- Phase 19 lesson 35 (this track) which stacks twelve of these blocks into a GPT model.
Check what you learned
6 questions. Answer each one to see why.
Which two sublayers live inside a single decoder transformer block?
The full code
Every file for this lesson. Copy it, run it, then change it.
"""Transformer block from scratch: LayerNorm, multi head causal attention, residual, MLP, residual.
Implements both pre-LN and post-LN configurations behind a single flag. The demo
builds a six layer stack of each, sends a single forward and backward pass through,
and prints the gradient norm at the input embedding for each variant. The pre-LN
stack carries an order of magnitude larger gradient at the embedding than the
post-LN stack at identical learning rate, which is the mechanism that lets
modern decoder LLMs train without a warmup schedule.
Run: python3 code/main.py
"""
from __future__ import annotations
import math
from dataclasses import dataclass
import torch
import torch.nn as nn
import torch.nn.functional as F
@dataclass
class BlockConfig:
"""Hyperparameters shared across attention, MLP, and the wrapping block."""
d_model: int = 768
num_heads: int = 12
context_length: int = 1024
mlp_expansion: int = 4
attn_dropout: float = 0.1
residual_dropout: float = 0.1
use_bias: bool = True
pre_ln: bool = True
class LayerNorm(nn.Module):
"""Layer normalization with learnable scale and shift.
Normalizes over the last dimension (the embedding axis) for every token
independently. Equivalent to nn.LayerNorm(d_model) but spelled out so the
eps placement and the parameter shapes are visible.
"""
def __init__(self, d_model: int, eps: float = 1e-5) -> None:
super().__init__()
self.eps = eps
self.scale = nn.Parameter(torch.ones(d_model))
self.shift = nn.Parameter(torch.zeros(d_model))
def forward(self, x: torch.Tensor) -> torch.Tensor:
mean = x.mean(dim=-1, keepdim=True)
var = x.var(dim=-1, keepdim=True, unbiased=False)
return self.scale * (x - mean) / torch.sqrt(var + self.eps) + self.shift
class MultiHeadAttention(nn.Module):
"""Multi head causal self attention with a fused QKV projection.
Fused QKV: one linear of width 3 * d_model instead of three linears, one
kernel launch, one matmul. The causal mask is registered as a buffer so it
is allocated once at construction and sliced per forward.
"""
def __init__(self, cfg: BlockConfig) -> None:
super().__init__()
if cfg.d_model % cfg.num_heads != 0:
raise ValueError(
f"d_model ({cfg.d_model}) must be divisible by num_heads ({cfg.num_heads})"
)
self.d_model = cfg.d_model
self.num_heads = cfg.num_heads
self.head_dim = cfg.d_model // cfg.num_heads
self.context_length = cfg.context_length
self.qkv = nn.Linear(cfg.d_model, 3 * cfg.d_model, bias=cfg.use_bias)
self.out_proj = nn.Linear(cfg.d_model, cfg.d_model, bias=cfg.use_bias)
self.attn_dropout = nn.Dropout(cfg.attn_dropout)
self.resid_dropout = nn.Dropout(cfg.residual_dropout)
mask = torch.triu(
torch.ones(cfg.context_length, cfg.context_length, dtype=torch.bool),
diagonal=1,
)
self.register_buffer("causal_mask", mask, persistent=False)
def forward(self, x: torch.Tensor) -> torch.Tensor:
batch, seq, dim = x.shape
if seq > self.context_length:
raise ValueError(
f"sequence length {seq} exceeds context length {self.context_length}"
)
qkv = self.qkv(x)
q, k, v = qkv.split(self.d_model, dim=-1)
q = q.view(batch, seq, self.num_heads, self.head_dim).transpose(1, 2)
k = k.view(batch, seq, self.num_heads, self.head_dim).transpose(1, 2)
v = v.view(batch, seq, self.num_heads, self.head_dim).transpose(1, 2)
scores = q @ k.transpose(-2, -1) / math.sqrt(self.head_dim)
mask = self.causal_mask[:seq, :seq]
scores = scores.masked_fill(mask, float("-inf"))
attn = F.softmax(scores, dim=-1)
attn = self.attn_dropout(attn)
out = attn @ v
out = out.transpose(1, 2).contiguous().view(batch, seq, dim)
out = self.out_proj(out)
out = self.resid_dropout(out)
return out
class FeedForward(nn.Module):
"""Position wise MLP. No token mixing happens here; all of that lives in attention."""
def __init__(self, cfg: BlockConfig) -> None:
super().__init__()
hidden = cfg.mlp_expansion * cfg.d_model
self.fc1 = nn.Linear(cfg.d_model, hidden, bias=cfg.use_bias)
self.act = nn.GELU(approximate="tanh")
self.fc2 = nn.Linear(hidden, cfg.d_model, bias=cfg.use_bias)
self.dropout = nn.Dropout(cfg.residual_dropout)
def forward(self, x: torch.Tensor) -> torch.Tensor:
x = self.fc1(x)
x = self.act(x)
x = self.fc2(x)
x = self.dropout(x)
return x
class TransformerBlock(nn.Module):
"""One transformer block. Toggle pre_ln to switch between configurations.
Pre-LN: norm inside the residual branch before each sublayer. The residual
carries an unnormalized tensor through every block; gradients propagate
cleanly to the embedding layer without a warmup schedule.
Post-LN: norm after the residual add. Gradient must pass through the norm
on every block; deep stacks need warmup to avoid divergence.
"""
def __init__(self, cfg: BlockConfig) -> None:
super().__init__()
self.pre_ln = cfg.pre_ln
self.ln1 = LayerNorm(cfg.d_model)
self.attn = MultiHeadAttention(cfg)
self.ln2 = LayerNorm(cfg.d_model)
self.mlp = FeedForward(cfg)
def forward(self, x: torch.Tensor) -> torch.Tensor:
if self.pre_ln:
x = x + self.attn(self.ln1(x))
x = x + self.mlp(self.ln2(x))
else:
x = self.ln1(x + self.attn(x))
x = self.ln2(x + self.mlp(x))
return x
class BlockStack(nn.Module):
"""A small stack used by the demo. The lesson 35 GPT uses the same pattern with twelve blocks."""
def __init__(self, cfg: BlockConfig, depth: int) -> None:
super().__init__()
self.embed = nn.Embedding(num_embeddings=128, embedding_dim=cfg.d_model)
self.blocks = nn.ModuleList([TransformerBlock(cfg) for _ in range(depth)])
self.final_ln = LayerNorm(cfg.d_model)
def forward(self, tokens: torch.Tensor) -> torch.Tensor:
x = self.embed(tokens)
for block in self.blocks:
x = block(x)
x = self.final_ln(x)
return x
def gradient_norm_at_embedding(stack: BlockStack, tokens: torch.Tensor) -> float:
"""Send one forward and one backward through the stack, return the embedding gradient norm.
The loss is the sum of squares of the final tensor. The magnitude is unitless;
what matters is the ratio between pre-LN and post-LN at the same depth.
"""
stack.zero_grad(set_to_none=True)
out = stack(tokens)
loss = out.pow(2).sum()
loss.backward()
grad = stack.embed.weight.grad
if grad is None:
return 0.0
return float(grad.norm().item())
def _set_eval_mode(stack: BlockStack) -> None:
"""Disable dropout so the comparison between pre-LN and post-LN is deterministic."""
stack.eval()
def demo() -> None:
torch.manual_seed(0)
cfg_pre = BlockConfig(
d_model=192,
num_heads=6,
context_length=64,
attn_dropout=0.0,
residual_dropout=0.0,
pre_ln=True,
)
cfg_post = BlockConfig(
d_model=192,
num_heads=6,
context_length=64,
attn_dropout=0.0,
residual_dropout=0.0,
pre_ln=False,
)
depth = 6
pre_stack = BlockStack(cfg_pre, depth=depth)
post_stack = BlockStack(cfg_post, depth=depth)
post_stack.load_state_dict(pre_stack.state_dict())
_set_eval_mode(pre_stack)
_set_eval_mode(post_stack)
tokens = torch.randint(0, 128, (2, 32))
with torch.no_grad():
pre_out = pre_stack(tokens)
post_out = post_stack(tokens)
print("Pre-LN output shape :", tuple(pre_out.shape))
print("Post-LN output shape:", tuple(post_out.shape))
assert pre_out.shape == post_out.shape == (2, 32, 192)
pre_grad = gradient_norm_at_embedding(pre_stack, tokens)
post_grad = gradient_norm_at_embedding(post_stack, tokens)
print(f"Pre-LN embedding grad norm: {pre_grad:.6f}")
print(f"Post-LN embedding grad norm: {post_grad:.6f}")
if post_grad > 0:
ratio = pre_grad / post_grad
print(f"Pre-LN / Post-LN ratio : {ratio:.2f}x")
n_params = sum(p.numel() for p in pre_stack.parameters())
print(f"Stack parameter count : {n_params:,}")
print("Block check passed.")
if __name__ == "__main__":
demo()