Token and Positional Embeddings
Ids are integers. The model wants vectors. Two lookup tables sit between them, and the choice of the positional one shapes what the model can learn.
Learning Objectives
- Build a token-embedding lookup table that maps vocabulary ids to dense vectors.
- Build a learned positional-embedding lookup table indexed by position.
- Build a fixed sinusoidal positional embedding indexed by position with no parameters.
- Compose token and positional embeddings into a single input for a transformer block.
- Contrast learned and sinusoidal embeddings on length generalization and parameter count.
The frame
The model's first contact with a token id is a row lookup in the token-embedding matrix. The matrix has one row per vocabulary id and one column per model dimension. The lookup returns a vector that the rest of the model treats as the meaning of the id. Backprop updates the rows that were used in the forward pass. Over training the geometry of those rows learns to encode similarity in directions.
Token ids alone have no order. The model needs a second signal that tells it position one is different from position seventeen. The two dominant choices for that signal are a learned positional embedding (a second lookup table, one row per position) and a fixed sinusoidal positional embedding (a math formula with no parameters). The choice has consequences. A learned table is a parameter and is bounded by the maximum context length the model was trained on. A sinusoidal table is parameter-free in theory and the formula extends to any position, but this lesson's SinusoidalPositionalEmbedding precomputes a fixed table at max_context_length and its forward raises past that bound; both modules therefore enforce a maximum context length here. The model may still struggle past its training length even when the table is large enough to index.
This lesson builds both and composes them with the token embedding into a single input for the next lesson's attention block.
The shape contract
The input to the embedding stage is a batch of token ids of shape (B, T). The output is a tensor of shape (B, T, D) where D is the model dimension. Every batch element has the same context length T. Every position has the same vector dimension D.
The composition is a sum, not a concatenation. Summing keeps D constant through the network and lets the model decide on a per-feature basis whether the token meaning or the position dominates at each layer.
The token embedding matrix
The token embedding is a parameter tensor of shape (V, D) where V is the vocabulary size. PyTorch exposes it as nn.Embedding(V, D). At init the entries are drawn from a small Gaussian, traditionally with mean zero and standard deviation around 0.02 for transformer-scale models. The exact init matters less than that it stays consistent across runs.
The forward pass is a single indexing operation. PyTorch maps (B, T) int64 ids to (B, T, D) floats by gathering rows. The backward pass accumulates gradients only into the rows that were touched in the forward pass. Two rows that never appeared in the batch receive zero gradient on that step.
A subtle detail. The token embedding and the output projection at the end of the model often share weights (weight tying). When that happens, every backward pass touches every row of the embedding through the output side. The lesson here exposes both as separate modules but the same matrix could play both roles in a full model.
The learned positional embedding
The learned positional embedding is a second nn.Embedding of shape (max_context_length, D). The lookup is keyed by position id 0, 1, 2, ..., T-1. The forward pass broadcasts that position vector across the batch dimension.
The downside of the learned table is that it cannot be queried at position T if the model was only trained up to position T-1. The row does not exist. Production decoder-only models that use this scheme bake the maximum context length into the architecture and refuse to process longer inputs.
The sinusoidal positional embedding
The sinusoidal positional embedding is a function from position to vector. Position p and feature i produce
angle = p / (10000 ** (2 * (i // 2) / D))
emb[p, 2k] = sin(angle)
emb[p, 2k + 1] = cos(angle)The function has no parameters. Every position has a unique vector. The wavelength varies geometrically across feature dimensions, so the lower dimensions encode coarse position and the higher dimensions encode fine position.
The property that follows from the choice of sin and cos together is that the vector at position p + k is a linear function of the vector at position p. That gives the attention layer an easy path to learning relative-position offsets. The model does not need a separate parameter to express "look five tokens back."
The lesson computes the full sinusoidal table once at construction and indexes into it at forward time.
The composition
The input pipeline does three things in order. Read the token ids. Look up the token vectors. Add the positional vectors. Return the sum.
The broadcasting in the sum step replicates the (T, D) positional tensor along the batch dimension. PyTorch handles that automatically because the positional tensor has shape (1, T, D) after unsqueeze.
Contrastive analysis
The lesson runs both variants on the same inputs and prints two diagnostics.
The first is parameter count. The learned variant adds max_context_length * D parameters on top of the token embedding. The sinusoidal variant adds zero.
The second is the cosine similarity between embeddings at neighbouring positions. The sinusoidal variant has a smooth and predictable decay because the function is continuous. The learned variant at initialization has near-random similarity because the rows are drawn independently. After training, the learned variant typically develops a similar smooth structure, but it has to discover that structure from data.
What this lesson does not do
It does not build rotary positional encoding (RoPE) or AliBi. Those are the modern choices in production transformers. They both follow the same shape contract as the embeddings here (apply a position-dependent transformation to vectors of shape (B, T, D)) but they apply at the attention-projection step rather than at the input. The next lesson builds the attention block, and one of the optional extensions is to fold rotary into the query-key projections there.
It does not train the embedding. Training requires a loss, which requires a model output, which requires attention and an LM head. That is the next lesson and the one after.
How to read the code
main.py defines three modules. TokenEmbedding wraps nn.Embedding(V, D). LearnedPositionalEmbedding wraps nn.Embedding(L, D). SinusoidalPositionalEmbedding precomputes the table and exposes it as a buffer. EmbeddingComposer ties a token embedding and a positional embedding together. The demo at the bottom prints the shapes, the parameter counts, and the neighbour-position similarity diagnostic. The tests in code/tests/test_embeddings.py pin shape, broadcast behaviour, parameter count, and the sinusoidal formula.
Run the demo. Then change the model dimension D from 64 to 32 and watch how the sinusoidal wavelength bands change.
Check what you learned
6 questions. Answer each one to see why.
What shape does the embedding stage produce from (B, T) ids?
The full code
Every file for this lesson. Copy it, run it, then change it.
"""Token and positional embeddings.
Builds three modules:
- TokenEmbedding: vocab_size x d_model lookup
- LearnedPositionalEmbedding: max_context_length x d_model lookup
- SinusoidalPositionalEmbedding: parameter-free sin/cos table
Composes them via EmbeddingComposer for the transformer input.
Run: python3 code/main.py
"""
from __future__ import annotations
import math
from dataclasses import dataclass
import torch
import torch.nn as nn
DEFAULT_INIT_STD = 0.02
def _init_normal(weight: torch.Tensor, std: float = DEFAULT_INIT_STD) -> None:
"""Init a parameter tensor in place from a small Gaussian."""
with torch.no_grad():
weight.normal_(mean=0.0, std=std)
class TokenEmbedding(nn.Module):
"""Vocabulary-id to vector lookup."""
def __init__(self, vocab_size: int, d_model: int, init_std: float = DEFAULT_INIT_STD) -> None:
super().__init__()
if vocab_size < 1:
raise ValueError(f"vocab_size must be >= 1, got {vocab_size}")
if d_model < 1:
raise ValueError(f"d_model must be >= 1, got {d_model}")
self.vocab_size = vocab_size
self.d_model = d_model
self.embedding = nn.Embedding(vocab_size, d_model)
_init_normal(self.embedding.weight, std=init_std)
def forward(self, ids: torch.Tensor) -> torch.Tensor:
if ids.dtype != torch.long:
raise TypeError(f"ids must be long tensor, got {ids.dtype}")
if ids.dim() != 2:
raise ValueError(f"ids must be (B, T), got shape {tuple(ids.shape)}")
return self.embedding(ids)
class LearnedPositionalEmbedding(nn.Module):
"""Position-id to vector lookup with learned parameters."""
def __init__(
self,
max_context_length: int,
d_model: int,
init_std: float = DEFAULT_INIT_STD,
) -> None:
super().__init__()
if max_context_length < 1:
raise ValueError(f"max_context_length must be >= 1, got {max_context_length}")
if d_model < 1:
raise ValueError(f"d_model must be >= 1, got {d_model}")
self.max_context_length = max_context_length
self.d_model = d_model
self.embedding = nn.Embedding(max_context_length, d_model)
_init_normal(self.embedding.weight, std=init_std)
def forward(self, seq_len: int) -> torch.Tensor:
if seq_len < 1:
raise ValueError(f"seq_len must be >= 1, got {seq_len}")
if seq_len > self.max_context_length:
raise ValueError(
f"seq_len {seq_len} exceeds max_context_length {self.max_context_length}"
)
positions = torch.arange(seq_len, device=self.embedding.weight.device)
return self.embedding(positions)
class SinusoidalPositionalEmbedding(nn.Module):
"""Parameter-free position-to-vector mapping.
pe[p, 2k] = sin(p / 10000^(2k/d_model))
pe[p, 2k+1] = cos(p / 10000^(2k/d_model))
"""
def __init__(self, max_context_length: int, d_model: int, base: float = 10000.0) -> None:
super().__init__()
if max_context_length < 1:
raise ValueError(f"max_context_length must be >= 1, got {max_context_length}")
if d_model < 1:
raise ValueError(f"d_model must be >= 1, got {d_model}")
if d_model % 2 != 0:
raise ValueError(f"d_model must be even for sinusoidal, got {d_model}")
self.max_context_length = max_context_length
self.d_model = d_model
self.base = base
pe = self._build_table(max_context_length, d_model, base)
self.register_buffer("pe", pe, persistent=False)
@staticmethod
def _build_table(max_context_length: int, d_model: int, base: float) -> torch.Tensor:
pos = torch.arange(max_context_length, dtype=torch.float32).unsqueeze(1)
i = torch.arange(d_model // 2, dtype=torch.float32)
denom = base ** (2 * i / d_model)
angle = pos / denom
pe = torch.zeros(max_context_length, d_model, dtype=torch.float32)
pe[:, 0::2] = torch.sin(angle)
pe[:, 1::2] = torch.cos(angle)
return pe
def forward(self, seq_len: int) -> torch.Tensor:
if seq_len < 1:
raise ValueError(f"seq_len must be >= 1, got {seq_len}")
if seq_len > self.max_context_length:
raise ValueError(
f"seq_len {seq_len} exceeds max_context_length {self.max_context_length}"
)
return self.pe[:seq_len]
class EmbeddingComposer(nn.Module):
"""Sums a token embedding with a positional embedding.
The positional embedding may be learned or sinusoidal.
"""
def __init__(
self,
token_embedding: TokenEmbedding,
positional_embedding: nn.Module,
) -> None:
super().__init__()
if not isinstance(token_embedding, TokenEmbedding):
raise TypeError("token_embedding must be a TokenEmbedding")
if not isinstance(
positional_embedding,
(LearnedPositionalEmbedding, SinusoidalPositionalEmbedding),
):
raise TypeError(
"positional_embedding must be Learned or Sinusoidal Positional Embedding"
)
if token_embedding.d_model != getattr(positional_embedding, "d_model", None):
raise ValueError("token and positional embeddings must share d_model")
self.token_embedding = token_embedding
self.positional_embedding = positional_embedding
@property
def d_model(self) -> int:
return self.token_embedding.d_model
def forward(self, ids: torch.Tensor) -> torch.Tensor:
if ids.dim() != 2:
raise ValueError(f"ids must be (B, T), got shape {tuple(ids.shape)}")
seq_len = ids.shape[1]
tok = self.token_embedding(ids)
pos = self.positional_embedding(seq_len)
return tok + pos.unsqueeze(0)
def count_parameters(module: nn.Module) -> int:
return sum(p.numel() for p in module.parameters() if p.requires_grad)
def neighbour_cosine_curve(table: torch.Tensor, max_offset: int = 8) -> list[float]:
"""Average cosine similarity between row p and row p+k for k in 1..max_offset.
Returns a list of length max_offset.
"""
if table.dim() != 2:
raise ValueError("table must be (L, D)")
if max_offset < 1:
raise ValueError(f"max_offset must be >= 1, got {max_offset}")
if max_offset >= table.shape[0]:
raise ValueError(
f"max_offset {max_offset} must be < number of rows {table.shape[0]}"
)
rows = table.detach().to(torch.float32)
norms = rows.norm(dim=1, keepdim=True).clamp(min=1e-8)
unit = rows / norms
result: list[float] = []
for k in range(1, max_offset + 1):
a = unit[:-k]
b = unit[k:]
dot = (a * b).sum(dim=1).mean().item()
result.append(dot)
return result
@dataclass
class DemoConfig:
vocab_size: int = 320
d_model: int = 64
max_context_length: int = 128
batch_size: int = 4
seq_len: int = 32
seed: int = 11
def _print_section(title: str) -> None:
bar = "-" * len(title)
print(f"\n{title}\n{bar}")
def main() -> int:
cfg = DemoConfig()
torch.manual_seed(cfg.seed)
token_emb = TokenEmbedding(cfg.vocab_size, cfg.d_model)
learned_pos = LearnedPositionalEmbedding(cfg.max_context_length, cfg.d_model)
sinusoidal_pos = SinusoidalPositionalEmbedding(cfg.max_context_length, cfg.d_model)
learned_composer = EmbeddingComposer(token_emb, learned_pos)
sinusoidal_composer = EmbeddingComposer(token_emb, sinusoidal_pos)
ids = torch.randint(0, cfg.vocab_size, (cfg.batch_size, cfg.seq_len), dtype=torch.long)
_print_section("Shapes")
out_learned = learned_composer(ids)
out_sinusoidal = sinusoidal_composer(ids)
print(f"token_emb output : {tuple(token_emb(ids).shape)}")
print(f"learned composer : {tuple(out_learned.shape)}")
print(f"sinusoidal cmp. : {tuple(out_sinusoidal.shape)}")
assert out_learned.shape == (cfg.batch_size, cfg.seq_len, cfg.d_model)
assert out_sinusoidal.shape == (cfg.batch_size, cfg.seq_len, cfg.d_model)
_print_section("Parameter counts")
token_params = count_parameters(token_emb)
learned_params = count_parameters(learned_pos)
sinusoidal_params = count_parameters(sinusoidal_pos)
print(f"token embedding : {token_params:>7}")
print(f"learned positional : {learned_params:>7}")
print(f"sinusoidal positional : {sinusoidal_params:>7} (parameter-free)")
assert sinusoidal_params == 0
_print_section("Neighbour cosine similarity")
learned_curve = neighbour_cosine_curve(learned_pos.embedding.weight, max_offset=6)
sinusoidal_curve = neighbour_cosine_curve(sinusoidal_pos.pe, max_offset=6)
print("offset k | learned cos | sinusoidal cos")
for k, (a, b) in enumerate(zip(learned_curve, sinusoidal_curve), start=1):
print(f" {k:>4} | {a:>+7.4f} | {b:>+7.4f}")
_print_section("Sinusoidal property: smooth decay")
monotone_count = sum(
sinusoidal_curve[i] >= sinusoidal_curve[i + 1] - 1e-3
for i in range(len(sinusoidal_curve) - 1)
)
print(
f"sinusoidal curve monotone steps: {monotone_count}/{len(sinusoidal_curve) - 1}"
)
_print_section("Length extrapolation")
short = cfg.max_context_length // 2
long = cfg.max_context_length
print(f"sinusoidal at len {short:>3} : ok")
_ = sinusoidal_pos(short)
print(f"sinusoidal at len {long:>3} : ok")
_ = sinusoidal_pos(long)
print("learned bounded by max_context_length: must error past max")
try:
_ = learned_pos(long + 1)
except ValueError as e:
print(f" caught: {e}")
print("\nDemo OK.")
return 0
if __name__ == "__main__":
raise SystemExit(main())