BERT, Masked Language Modeling
GPT predicts the next word. BERT predicts a missing word. One sentence of difference, and half a decade of everything embedding-shaped.
The Problem
In 2018 every NLP task, sentiment, NER, QA, entailment, trained its own model from scratch on its own labeled data. There was no pre-trained "understand English" checkpoint you could fine-tune. ELMo (2018) showed you could pre-train contextual embeddings with a bidirectional LSTM; it helped but did not generalize.
BERT (Devlin et al. 2018) asked: what if we took a transformer encoder, trained it on every sentence on the internet, and forced it to predict missing words from context on both sides? Then you fine-tune one head on your downstream task. Parameter efficiency was a revelation.
The result: within 18 months BERT and its variants (RoBERTa, ALBERT, ELECTRA) dominated every NLP leaderboard that existed. By 2020 every search engine, content moderation pipeline, and semantic-search system on earth had a BERT inside.
In 2026 encoder-only models are still the right tool for classification, retrieval, and structured extraction, they run 5 to 10× faster per token than decoders and their embeddings are the backbone of every modern retrieval stack. ModernBERT (Dec 2024) pushed the architecture to 8K context with Flash Attention + RoPE + GeGLU.
The Concept
The training signal
Take a sentence: the quick brown fox jumps over the lazy dog.
Mask 15% of tokens randomly:
input: the [MASK] brown fox jumps [MASK] the lazy dog
target: the quick brown fox jumps over the lazy dogTrain the model to predict the original tokens at masked positions. Because the encoder is bidirectional, predicting [MASK] at position 1 can use brown fox jumps at positions 2+. That is the thing GPT cannot do.
The BERT mask rules
Of the 15% of tokens selected for prediction:
- 80% are replaced with
[MASK]. - 10% are replaced with a random token.
- 10% are left unchanged.
Why not always [MASK]? Because [MASK] never appears at inference time. Training the model to expect [MASK] at 100% of masked positions would create a distribution shift between pretraining and fine-tuning. The 10% random + 10% unchanged keeps the model honest.
Next Sentence Prediction (NSP), and why it was dropped
Original BERT also trained on NSP: given two sentences A and B, predict if B follows A. RoBERTa (2019) ablated it and showed NSP hurt, not helped. Modern encoders skip it.
What changed in 2026: ModernBERT
The 2024 ModernBERT paper rebuilt the block with 2026 primitives:
| Component | Original BERT (2018) | ModernBERT (2024) |
|---|---|---|
| Positional | Learned absolute | RoPE |
| Activation | GELU | GeGLU |
| Normalization | LayerNorm | Pre-norm RMSNorm |
| Attention | Full dense | Alternating local (128) + global |
| Context length | 512 | 8192 |
| Tokenizer | WordPiece | BPE |
And unlike the 2018 stack, it is Flash-Attention-native. Inference is 2 to 3× faster at sequence length 8K than DeBERTa-v3 with better GLUE scores.
Use cases that still pick an encoder in 2026
| Task | Why encoder beats decoder |
|---|---|
| Retrieval / semantic search embeddings | Bidirectional context = better embedding quality per token |
| Classification (sentiment, intent, toxicity) | One forward pass; no generation overhead |
| NER / token labeling | Per-position output, natively bidirectional |
| Zero-shot entailment (NLI) | Classifier head on top of encoder |
| Reranker for RAG | Cross-encoder scoring, 10x faster than LLM rerankers |
Build It
Step 1: masking logic
See code/main.py. The function create_mlm_batch takes a list of token IDs, a vocab size, and a mask probability. Returns input IDs (with masks applied) and labels (only at masked positions, -100 elsewhere, PyTorch's ignore index convention).
def create_mlm_batch(tokens, vocab_size, mask_prob=0.15, rng=None):
input_ids = list(tokens)
labels = [-100] * len(tokens)
for i, t in enumerate(tokens):
if rng.random() < mask_prob:
labels[i] = t
r = rng.random()
if r < 0.8:
input_ids[i] = MASK_ID
elif r < 0.9:
input_ids[i] = rng.randrange(vocab_size)
# else: keep original
return input_ids, labelsStep 2: run MLM prediction on a tiny corpus
Train a 2-layer encoder + MLM head on a vocabulary of 20 words, 200 sentences. No gradient, we do forward-pass sanity checks. Full training needs PyTorch.
Step 3: compare mask types
Show how the three-way rule keeps the model usable without [MASK]. Predict on an unmasked sentence and on a masked sentence. Both should produce reasonable token distributions because the model saw both patterns in training.
Step 4: fine-tune head
Replace the MLM head with a classification head on a toy sentiment dataset. Only the head trains; the encoder is frozen. This is the pattern every BERT application follows.
Use It
from transformers import AutoModel, AutoTokenizer
tok = AutoTokenizer.from_pretrained("answerdotai/ModernBERT-base")
model = AutoModel.from_pretrained("answerdotai/ModernBERT-base")
text = "Attention is all you need."
inputs = tok(text, return_tensors="pt")
out = model(**inputs).last_hidden_state # (1, N, 768)Embedding models are fine-tuned BERT. sentence-transformers models like all-MiniLM-L6-v2 are BERTs trained with contrastive loss. The encoder is the same. The loss changed.
Cross-encoder rerankers are also fine-tuned BERT. Pair-classification on [CLS] query [SEP] doc [SEP]. The bidirectional attention between query and doc is exactly what gives cross-encoders their quality edge over biencoders.
When not to pick BERT in 2026. Anything generative. The encoder has no sensible way to autoregressively produce tokens. Also: anything under 1B params where a small decoder can match quality with more flexibility (Phi-3-Mini, Qwen2-1.5B).
Ship It
See outputs/skill-bert-finetuner.md. The skill scopes a BERT fine-tune (backbone choice, head spec, data, eval, stopping) for a new classification or extraction task.
Exercises
- Easy. Run
code/main.pyand print the mask distribution across 10,000 tokens. Confirm ~15% are selected, and of those ~80% become[MASK]. - Medium. Implement whole-word masking: if a word is tokenized into subwords, mask all subwords together or none. Measure whether this improves MLM accuracy on a 500-sentence corpus.
- Hard. Train a tiny (2-layer, d=64) BERT on 10,000 sentences from a public dataset. Fine-tune the
[CLS]token for SST-2 sentiment. Compare against a decoder-only baseline at matched params, which wins?
Key Terms
| Term | What people say | What it actually means |
|---|---|---|
| MLM | "Masked language modeling" | Training signal: randomly replace 15% of tokens with [MASK]predict the originals. |
| Bidirectional | "Looks both ways" | Encoder attention has no causal mask, every position sees every other position. |
[CLS] | "The pooler token" | A special token prepended to every sequence; its final embedding is used as the sentence-level representation. |
[SEP] | "Segment separator" | Separates paired sequences (e.g. query/doc, sentence A/B). |
| NSP | "Next sentence prediction" | BERT's second pretraining task; shown to be useless in RoBERTa, dropped after 2019. |
| Fine-tuning | "Adapt to a task" | Keep the encoder mostly frozen; train a small head on top for the downstream task. |
| Cross-encoder | "A reranker" | A BERT that takes both query and doc as input, outputs a relevance score. |
| ModernBERT | "2024 refresh" | Encoder rebuilt with RoPE, RMSNorm, GeGLU, alternating local/global attention, 8K context. |
Further Reading
- Devlin et al. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingoriginal paper.
- Liu et al. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approachhow to train BERT right; kills NSP.
- Clark et al. (2020). ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generatorsreplaced-token detection beats MLM at matched compute.
- Warner et al. (2024). Smarter, Better, Faster, Longer: A Modern Bidirectional EncoderModernBERT paper.
- HuggingFace
modeling_bert.pycanonical encoder reference.
The full code
Every file for this lesson. Copy it, run it, then change it.
"""BERT-style masked language modeling — the masking rules demystified.
Pure stdlib. Shows the 80/10/10 rule, whole-word masking, and
distribution sanity checks over a large batch of tokens.
"""
import random
from collections import Counter
MASK_ID = 0 # reserve id 0 for [MASK] in this toy vocab
CLS_ID = 1
SEP_ID = 2
SPECIAL_IDS = {MASK_ID, CLS_ID, SEP_ID}
IGNORE_INDEX = -100
def create_mlm_batch(tokens, vocab_size, mask_prob=0.15, rng=None):
"""Apply BERT masking.
Returns (input_ids, labels). labels[i] = original token if position was
selected for prediction, IGNORE_INDEX otherwise.
"""
if rng is None:
rng = random.Random()
input_ids = list(tokens)
labels = [IGNORE_INDEX] * len(tokens)
for i, t in enumerate(tokens):
if t in SPECIAL_IDS:
continue
if rng.random() >= mask_prob:
continue
labels[i] = t
r = rng.random()
if r < 0.8:
input_ids[i] = MASK_ID
elif r < 0.9:
rand_id = t
while rand_id in SPECIAL_IDS or rand_id == t:
rand_id = rng.randrange(vocab_size)
input_ids[i] = rand_id
return input_ids, labels
def whole_word_mlm(tokens, word_spans, vocab_size, mask_prob=0.15, rng=None):
"""Mask whole words: if any subword in a span is selected, mask all.
word_spans: list of (start, end) half-open ranges into tokens.
"""
if rng is None:
rng = random.Random()
input_ids = list(tokens)
labels = [IGNORE_INDEX] * len(tokens)
for start, end in word_spans:
if any(tokens[i] in SPECIAL_IDS for i in range(start, end)):
continue
if rng.random() >= mask_prob:
continue
r = rng.random()
if r < 0.8:
for i in range(start, end):
labels[i] = tokens[i]
input_ids[i] = MASK_ID
elif r < 0.9:
for i in range(start, end):
labels[i] = tokens[i]
rand_id = tokens[i]
while rand_id in SPECIAL_IDS or rand_id == tokens[i]:
rand_id = rng.randrange(vocab_size)
input_ids[i] = rand_id
else:
for i in range(start, end):
labels[i] = tokens[i]
return input_ids, labels
def distribution_check(n_tokens, vocab_size, mask_prob=0.15, seed=42):
rng = random.Random(seed)
tokens = [rng.randrange(3, vocab_size) for _ in range(n_tokens)]
input_ids, labels = create_mlm_batch(tokens, vocab_size, mask_prob, rng)
selected = sum(1 for l in labels if l != IGNORE_INDEX)
masked = sum(1 for t, l in zip(input_ids, labels) if l != IGNORE_INDEX and t == MASK_ID)
randomized = sum(1 for t, l in zip(input_ids, labels) if l != IGNORE_INDEX and t != MASK_ID and t != l)
unchanged = sum(1 for t, l in zip(input_ids, labels) if l != IGNORE_INDEX and t == l)
return {
"tokens": n_tokens,
"selected": selected,
"selected_pct": 100 * selected / n_tokens,
"masked_of_selected_pct": 100 * masked / selected if selected else 0.0,
"random_of_selected_pct": 100 * randomized / selected if selected else 0.0,
"unchanged_of_selected_pct": 100 * unchanged / selected if selected else 0.0,
}
def toy_predict(masked_inputs, vocab):
"""Pretend MLM head: returns a uniform distribution over vocab.
Real BERT uses the encoder output at each position, projected to vocab.
"""
V = len(vocab)
return [[1.0 / V for _ in range(V)] for _ in masked_inputs]
def main():
vocab_words = [
"[MASK]", "[CLS]", "[SEP]",
"the", "quick", "brown", "fox", "jumps", "over",
"lazy", "dog", "a", "stitch", "in", "time",
"saves", "nine", "sat", "on", "mat",
]
vocab_size = len(vocab_words)
id_of = {w: i for i, w in enumerate(vocab_words)}
sentence = ["[CLS]", "the", "quick", "brown", "fox", "jumps", "over", "the", "lazy", "dog", "[SEP]"]
tokens = [id_of[w] for w in sentence]
rng = random.Random(42)
inp, labels = create_mlm_batch(tokens, vocab_size, mask_prob=0.5, rng=rng)
print("=== MLM masking demo (prob=0.5 so you can see it) ===")
print(f"{'idx':>4} {'word':>9} {'input_id':>9} {'input_word':>11} {'label':>6}")
for i, (t_in, t_orig, lab) in enumerate(zip(inp, tokens, labels)):
print(f"{i:>4} {vocab_words[t_orig]:>9} {t_in:>9} {vocab_words[t_in]:>11} {lab:>6}")
print()
print("=== 80/10/10 distribution over 100k random tokens ===")
stats = distribution_check(n_tokens=100_000, vocab_size=vocab_size, mask_prob=0.15)
print(f"selected: {stats['selected_pct']:.2f}% (target 15.0%)")
print(f" -> replaced with [MASK]: {stats['masked_of_selected_pct']:.2f}% (target 80.0%)")
print(f" -> replaced with random: {stats['random_of_selected_pct']:.2f}% (target 10.0%)")
print(f" -> left unchanged: {stats['unchanged_of_selected_pct']:.2f}% (target 10.0%)")
print()
print("=== whole-word masking demo ===")
# Treat "quick brown" and "lazy dog" as two-subword words for demo
tokens2 = [id_of[w] for w in sentence]
spans = [(0, 1), (1, 2), (2, 4), (4, 5), (5, 6), (6, 7), (7, 8), (8, 10), (10, 11)]
rng2 = random.Random(7)
inp2, labels2 = whole_word_mlm(tokens2, spans, vocab_size, mask_prob=0.5, rng=rng2)
print("spans: " + " ".join(f"[{s}:{e}]" for s, e in spans))
print("input words: " + " ".join(vocab_words[t] for t in inp2))
print("label mask: " + " ".join(("P" if l != IGNORE_INDEX else ".") for l in labels2))
print("P = position has a label, . = ignored")
if __name__ == "__main__":
main()