Positional Encoding, Sinusoidal, RoPE, ALiBi
Attention is permutation-invariant. "The cat sat on the mat" and "mat the on sat cat the" produce the same output without positional signal. Three algorithms fix it, each with a different bet on what "position" means.
The Problem
Scaled dot-product attention is order-blind. The attention matrix softmax(Q K^T / √d) V is computed from pairwise similarities. Shuffle the rows of Xget the rows of the output shuffled the same way. Nothing inside attention cares about position.
That is not a bug in a bag-of-words model. For language, code, audio, video, anything where order carries meaning, it is fatal.
The fix is to inject position into the embeddings somehow. Three eras of answers:
- Absolute sinusoidal (Vaswani 2017). Add
sin/cosof position to the embedding. Simple, learnable-free, extrapolates poorly beyond trained lengths. - RoPE, Rotary Position Embeddings (Su 2021). Rotate Q and K vectors by an angle proportional to position. Encodes relative position directly in the dot product. Dominant in 2026.
- ALiBi, Attention with Linear Biases (Press 2022). Skip embeddings entirely; add a per-head linear penalty to attention scores based on distance. Excellent length extrapolation.
As of 2026, essentially every frontier open model uses RoPE: Llama 2/3/4, Qwen 2/3, Mistral, Mixtral, DeepSeek-V3, Kimi. A handful of long-context models use ALiBi or its modern variants. Absolute sinusoidal is historical.
The Concept
Absolute sinusoidal
Pre-compute a fixed matrix PE of shape (max_len, d_model):
Then X' = X + PE[:N] before attention. Each dimension is a sinusoid at a different frequency. The model learns to read position from the phase pattern. Fails beyond max_len: nothing told the model what happens at position 2048 when it only saw positions 0 to 2047.
RoPE
Rotate the Q and K vectors (not embeddings). For a pair of dimensions $(2i, 2i+1)$:
Apply the same rotation to keys with position pos_k. The dot product $q'_m \cdot k'_n$ becomes a function of $(m - n)$ alone. That is: the attention score depends only on the relative distanceeven though the rotation was keyed off absolute positions. Beautiful trick.
Extending RoPE: base can be scaled (NTK-aware, YaRN, LongRoPE) to extrapolate to longer contexts without retraining. Llama 3 extended from 8K to 128K context this way.
ALiBi
Skip the embedding trick. Bias the attention scores directly:
Where $m_h$ is a head-specific slope (e.g. $1 / 2^{8h/H}$). Closer tokens get boosted; far tokens get penalized. No training-time cost. The paper shows length extrapolation beats sinusoidal and matches RoPE on its original trained length.
What to pick in 2026
| Variant | Extrapolation | Training cost | Used by |
|---|---|---|---|
| Absolute sinusoidal | poor | free | original transformer, early BERT |
| Learned absolute | none | tiny | GPT-2, GPT-3 |
| RoPE | good with scaling | free | Llama 2/3/4, Qwen 2/3, Mistral, DeepSeek-V3, Kimi |
| RoPE + YaRN | excellent | fine-tune stage | Qwen2-1M, Llama 3.1 128K |
| ALiBi | excellent | free | BLOOM, MPT, Baichuan |
RoPE won because it slots into attention without changing the architecture, encodes relative position, and its base hyperparameter gives a clean knob for long-context fine-tuning.
Build It
Step 1: sinusoidal encoding
See code/main.py. A 4-line computation:
def sinusoidal(N, d):
pe = [[0.0] * d for _ in range(N)]
for pos in range(N):
for i in range(d // 2):
theta = pos / (10000 ** (2 * i / d))
pe[pos][2 * i] = math.sin(theta)
pe[pos][2 * i + 1] = math.cos(theta)
return peAdd this to the embedding matrix before the first attention layer.
Step 2: RoPE applied to Q, K
RoPE operates in-place on Q and K. For each pair of dims:
def apply_rope(x, pos, base=10000):
d = len(x)
out = list(x)
for i in range(d // 2):
theta = pos / (base ** (2 * i / d))
c, s = math.cos(theta), math.sin(theta)
a, b = x[2 * i], x[2 * i + 1]
out[2 * i] = a * c - b * s
out[2 * i + 1] = a * s + b * c
return outCrucial: apply the same function to Q at position $m$ and K at position $n$. Their dot product picks up a $\cos((m-n) \cdot \theta_i)$ factor on every coordinate pair. Attention learns relative position for free.
Step 3: ALiBi slopes and bias
def alibi_bias(n_heads, seq_len):
# slope_h = 2 ** (-8 * h / n_heads) for h = 1..n_heads
slopes = [2 ** (-8 * (h + 1) / n_heads) for h in range(n_heads)]
bias = []
for m in slopes:
row = [[-m * abs(i - j) for j in range(seq_len)] for i in range(seq_len)]
bias.append(row)
return bias # add to attention scores before softmaxAdd bias[h] to the (seq_len, seq_len) attention score matrix of head hthen softmax.
Step 4: verify relative-distance property of RoPE
Pick two random vectors a, b. Rotate by (pos_a, pos_b). Then by (pos_a + k, pos_b + k). Both dot products must match within floating-point error. That property is the whole point of RoPE, it is invariant to the absolute offset, only the relative gap matters.
Use It
PyTorch 2.5+ ships RoPE utilities in torch.nn.functional. Most production code uses flash_attn or xformers where RoPE is applied inside the attention kernel.
from transformers import AutoModel
model = AutoModel.from_pretrained("meta-llama/Llama-3.2-3B")
# model.config.rope_scaling → {"type": "yarn", "factor": 32.0, "original_max_position_embeddings": 8192}Long-context tricks in 2026:
- NTK-aware interpolation. Rescale
basetobase * (scale_factor)^(d/(d-2))when extending from 4K to 16K+. - YaRN. Smarter interpolation that preserves attention entropy on long contexts. Llama 3.1 128K uses it.
- LongRoPE. Microsoft's 2024 method that uses evolutionary search to pick per-dimension scale factors. Phi-3-Long uses it.
- Position interpolation + fine-tuning. Just shrink positions by the extension factor and fine-tune for 1 to 5B tokens. Surprisingly effective.
Ship It
See outputs/skill-positional-encoding-picker.md. The skill picks an encoding strategy for a new model given target context length, extrapolation needs, and training budget.
Exercises
- Easy. Plot the sinusoidal
PEmatrix as a heatmap formax_len=512, d=128. Confirm the "stripes get wider as dimension index grows" pattern. - Medium. Implement NTK-aware RoPE scaling. Train a tiny LM on sequences of length 256, then test on length 1024 with and without scaling. Measure perplexity.
- Hard. Implement ALiBi and RoPE in the same attention module. Train a 4-layer transformer on a copy task with sequences of length 512. Extrapolate to 2048 at test time. Compare degradation.
Key Terms
| Term | What people say | What it actually means |
|---|---|---|
| Positional encoding | "Tells attention about order" | Any signal added to embeddings or attention that encodes position. |
| Sinusoidal | "The original one" | sin/cos at geometric frequencies added to embeddings; doesn't extrapolate. |
| RoPE | "Rotary embeddings" | Rotate Q, K by position-dependent angle; dot product encodes relative distance. |
| ALiBi | "Linear bias trick" | Add $-m \cdot \lvert i-j \rvert$ to attention scores; no embedding needed, great extrapolation. |
| base | "RoPE's knob" | The frequency scaler in RoPE; increase to extend context at inference. |
| NTK-aware | "A RoPE scaling trick" | Rescale base so high-frequency dims aren't squeezed when context expands. |
| YaRN | "The fancy one" | Per-dimension interpolation+extrapolation that preserves attention entropy. |
| Extrapolation | "Works beyond trained length" | Can the position scheme serve correct output past max_len seen in training? |
Further Reading
- Vaswani et al. (2017). Attention Is All You Need §3.5original sinusoidal.
- Su et al. (2021). RoFormer: Enhanced Transformer with Rotary Position EmbeddingRoPE paper.
- Press, Smith, Lewis (2021). Train Short, Test Long: Attention with Linear Biases Enables Input Length ExtrapolationALiBi.
- Peng et al. (2023). YaRN: Efficient Context Window Extension of Large Language Modelsstate of the art RoPE scaling.
- Chen et al. (2023). Extending Context Window of Large Language Models via Positional InterpolationMeta's Llama 2 long-context paper.
- Ding et al. (2024). LongRoPE: Extending LLM Context Window Beyond 2 Million Tokensthe Microsoft method used by Phi-3-Long and cited in the Use It section.
- HuggingFace Transformers,
modeling_rope_utils.pyproduction-grade implementations of every RoPE scaling scheme (default, linear, dynamic, YaRN, LongRoPE, Llama-3).
The full code
Every file for this lesson. Copy it, run it, then change it.
# Positional encoding in Julia. Sinusoidal absolute positions, rotary
# positional embedding (RoPE), and ALiBi bias matrix. Verifies that
# RoPE dot products depend only on relative distance. Stdlib only. Sources:
# https://arxiv.org/abs/2104.09864
# https://arxiv.org/abs/2108.12409
# https://docs.julialang.org/en/v1/manual/mathematical-operations/
using Random
using Printf
function sinusoidal_pe(n::Int, d::Int; base::Float64=10000.0)::Matrix{Float64}
n > 0 || throw(ArgumentError("n must be > 0"))
d > 0 || throw(ArgumentError("d must be > 0"))
iseven(d) || throw(ArgumentError("d must be even for sinusoidal sin/cos pairs"))
pe = zeros(n, d)
for pos in 0:(n - 1)
for i in 0:(d ÷ 2 - 1)
theta = pos / (base ^ (2 * i / d))
pe[pos + 1, 2 * i + 1] = sin(theta)
pe[pos + 1, 2 * i + 2] = cos(theta)
end
end
return pe
end
function apply_rope(x::Vector{Float64}, pos::Int; base::Float64=10000.0)::Vector{Float64}
d = length(x)
iseven(d) || throw(ArgumentError("RoPE requires an even embedding dimension"))
out = copy(x)
for i in 0:(d ÷ 2 - 1)
theta = pos / (base ^ (2 * i / d))
c = cos(theta)
s = sin(theta)
a = x[2 * i + 1]
b = x[2 * i + 2]
out[2 * i + 1] = a * c - b * s
out[2 * i + 2] = a * s + b * c
end
return out
end
function dotprod(a::Vector{Float64}, b::Vector{Float64})::Float64
return sum(a .* b)
end
function alibi_slopes(n_heads::Int)::Vector{Float64}
n_heads > 0 || throw(ArgumentError("n_heads must be > 0"))
return [2.0 ^ (-8.0 * (h) / n_heads) for h in 1:n_heads]
end
function alibi_bias(n_heads::Int, seq_len::Int; causal::Bool=true)
slopes = alibi_slopes(n_heads)
out = Vector{Matrix{Float64}}()
for m in slopes
bias = fill(0.0, seq_len, seq_len)
for i in 1:seq_len
for j in 1:seq_len
if causal && j > i
bias[i, j] = -Inf
else
bias[i, j] = -m * abs(i - j)
end
end
end
push!(out, bias)
end
return out
end
function demo_sinusoidal()
println("=== sinusoidal positional encoding ===")
pe = sinusoidal_pe(8, 8)
println("first 4 positions, first 4 dims:")
for pos in 1:4
row_str = join([@sprintf("%+.3f", pe[pos, j]) for j in 1:4], " ")
@printf(" pos=%d: %s\n", pos - 1, row_str)
end
println()
end
function demo_rope_relative()
println("=== RoPE: dot product depends only on relative distance ===")
rng = MersenneTwister(0)
d = 16
q = randn(rng, d)
k = randn(rng, d)
pairs = [(3, 5), (7, 9), (100, 102), (1024, 1026)]
@printf("%6s %6s %4s %18s\n", "pos_q", "pos_k", "gap", "<q_rot, k_rot>")
for (pq, pk) in pairs
q_rot = apply_rope(q, pq)
k_rot = apply_rope(k, pk)
d_prod = dotprod(q_rot, k_rot)
@printf("%6d %6d %4d %18.6f\n", pq, pk, pk - pq, d_prod)
end
println("All rows with gap=2 should produce matching dot products.")
println()
end
function demo_rope_base_scaling()
println("=== RoPE base scaling (NTK-aware for long context) ===")
rng = MersenneTwister(1)
d = 8
q = randn(rng, d)
k = randn(rng, d)
for base in (10000.0, 100000.0, 1_000_000.0)
q_rot = apply_rope(q, 4096; base=base)
k_rot = apply_rope(k, 4098; base=base)
@printf(" base=%8d score=%+.6f\n", Int(base), dotprod(q_rot, k_rot))
end
println("Larger base = slower rotation = longer context without phase wrap.")
println()
end
function demo_alibi()
println("=== ALiBi bias matrix ===")
n_heads = 4
slopes = alibi_slopes(n_heads)
@printf("Slopes for %d heads: %s\n", n_heads,
join([@sprintf("%.4f", s) for s in slopes], ", "))
bias = alibi_bias(n_heads, 6; causal=false)
println("Head 1 bias (closer tokens get smaller penalty):")
for row in eachrow(bias[1])
println(" " * join([@sprintf("%+6.2f", v) for v in row], " "))
end
println()
end
function main()
demo_sinusoidal()
demo_rope_relative()
demo_rope_base_scaling()
demo_alibi()
println("takeaway: RoPE encodes relative position inside the dot product;")
println("ALiBi skips embeddings entirely. Sinusoidal is now a footnote.")
end
if abspath(PROGRAM_FILE) == @__FILE__
main()
end