Chameleon and Early-Fusion Token-Only Multimodal Models
Every VLM we have seen so far keeps images and text separate. Visual tokens come from a vision encoder, flow into a projector, then meet text inside the LLM. The vision and text vocabularies never overlap. Chameleon (Meta, May 2024) asked: what if they did? Train a VQ-VAE that turns an image into a sequence of discrete tokens from a shared vocabulary. Every multimodal document is now one sequence, text tokens and image tokens interleaved, a single autoregressive loss. Side effect: the model can generate mixed-modality outputs, alternating text and image tokens in a single inference call. This lesson reads the early-fusion thesis and builds a toy version end to end.
Learning Objectives
- Explain why a shared vocabulary + single loss changes what the model can do.
- Describe how a VQ-VAE tokenizes an image into a discrete sequence compatible with a transformer's next-token objective.
- Name Chameleon's training-stability tricks: QK-Norm, dropout placement, LayerNorm ordering.
- Compare Chameleon vs BLIP-2's Q-Former approach and describe when each is the right choice.
The Problem
Adapter-based VLMs (LLaVA, BLIP-2, Qwen-VL) treat text and image as two different things. A text token goes through embed(text_token); an image goes through visual_encoder(image) → projector → ... pseudo_tokens. The model has two input paths that merge partway in.
Three consequences:
- The LLM can only consume images, not emit them. Output is text only.
- Mixed-modality documents (alternating paragraphs and images, as in an article) are awkward, you either parse the multimodal input outside the model or chain generations.
- Distributional mismatch. Visual tokens and text tokens live in different regions of the hidden space, creating subtle alignment issues.
Chameleon rejects the premise: images are just sequences of discrete tokens from a shared vocabulary. Train the model on interleaved documents, one loss, one autoregressive decoder, and you unlock mixed-modality generation for free.
The Concept
VQ-VAE as image tokenizer
The tokenizer is a vector-quantized variational autoencoder. The architecture:
- Encoder: CNN + ViT that maps image to a spatial feature map, say 32x32 features of dim 256.
- Codebook: a learned vocabulary of K vectors (Chameleon uses 8192), also dim 256.
- Quantization: for each spatial feature, look up the nearest codebook entry by L2 distance. Replace the continuous feature with the integer index.
- Decoder: CNN that takes quantized features back to pixels.
Training: VAE reconstruction loss + commitment loss + codebook loss. The codebook indices form a discrete alphabet for images.
For Chameleon: one image becomes 32*32 = 1024 tokens drawn from a vocabulary of 8192. Concatenate with text tokens (from the LLM's BPE vocabulary, say 32000). Final vocabulary: 40192. The transformer sees one sequence, one loss.
The shared vocabulary
Chameleon's vocabulary combines text tokens, image tokens, and modality separators. Each token has a single ID. The input embedding layer maps every ID to a D-dim hidden vector. The output projection maps hidden back to vocab logits. Softmax picks the next token, whatever modality.
Separators matter: <image> and </image> tags bracket the image-token sequence. At generation time, if the model emits <image>downstream software knows the next 1024 tokens are VQ indices to send to the decoder for pixel rendering.
Mixed-modality generation
Inference is next-token prediction in the shared vocabulary. Example prompt: "Draw a cat and describe it." Chameleon emits:
<image> 4821 1029 2891 ... (1024 image tokens) </image>
The cat is orange, sitting on a windowsill...The model picks the order autonomously, it may produce image then text, text then image, or interleave. Same decoder, same loss.
Compare to adapter VLMs where generation is text-only. Chameleon reopens the question of model output modalities.
Training stability, QK-Norm, dropout, LayerNorm ordering
Early-fusion training is unstable at scale. Chameleon's paper documents three tricks:
- QK-Norm. Apply LayerNorm to the query and key projections inside attention, before the dot product. Prevents logit magnitude explosion at depth. Used by multiple post-2024 large models.
- Dropout placement. Dropout after every residual-add, not just after attention and MLP. More regularization required when gradients from image tokens can dominate.
- LayerNorm ordering. Pre-LN on the residual branch (standard), plus an extra LN on the skip connection of the last block. Stabilizes final-layer gradient flow.
Without these tricks, 34B-param Chameleon training diverged at multiple checkpoints. With them, it converges. The training recipe is as much of the contribution as the architecture.
The tokenizer's reconstruction ceiling
VQ-VAE is lossy. At 8192 codebook entries and 1024 tokens per 512x512 image, reconstruction PSNR caps around 26-28 dB. This is enough for recognizable image gen but visibly worse than continuous-space diffusion (Stable Diffusion 3 achieves 32+ dB).
The tokenizer is the bottleneck. Better tokenizers (MAGVIT-v2, IBQ, SBER-MoVQGAN) lift the ceiling. Emu3 (Lesson 12.12) achieves SDXL-quality generation via a better tokenizer alone.
Chameleon vs BLIP-2 / LLaVA
Chameleon (early fusion, shared vocab):
- One loss, one decoder.
- Generates mixed-modality output.
- Tokenizer is the quality ceiling.
- Expensive: VQ-VAE decoder per generated image on inference path.
BLIP-2 / LLaVA (late fusion, separate towers):
- Vision in, text out only.
- Reuses pretrained LLM.
- No tokenizer bottleneck for understanding.
- Cheap: single forward pass.
Pick by task. If you need image generation, Chameleon family. If you only need understanding, adapter-VLM is simpler and reuses more pretrained compute.
Fuyu and AnyGPT
Fuyu (Adept, 2023) is a related approach: skip the separate vision encoder entirely, feed raw image patches through the LLM's input projection as if they were tokens, no tokenizer. Simpler than Chameleon, loses the shared-vocab output generation.
AnyGPT (Zhan et al., 2024) extends Chameleon to four modalities: text, image, speech, music. Same VQ-VAE trick for each, shared transformer. Any-to-any generation. Covered more in Lesson 12.16.
Use It
code/main.py builds a toy end-to-end early-fusion model:
- A tiny VQ-VAE-style quantizer that maps 8x8 patches to codebook indices (K=16).
- A shared vocabulary of (text ids 0..31) + (image ids 32..47) + (separators 48, 49).
- A toy autoregressive decoder (bigram table) trained on synthetic captions + image-token sequences.
- Sampling loop that emits alternating text + image tokens given a prompt.
The code intentionally keeps the transformer tiny (bigrams) so you can trace the signal flow end to end.
Ship It
This lesson produces outputs/skill-tokenizer-vs-adapter-picker.md. Given a product spec (understand only vs understand + generate, required image quality, cost budget), it picks between Chameleon-family (early fusion) and LLaVA-family (late fusion) and justifies with quantitative rules of thumb.
Exercises
- Chameleon uses K=8192 codebook entries and 1024 tokens per 512x512 image. Estimate the compression ratio vs a 24-bit RGB image. Is it lossy? How lossy?
- A 4K image (3840x2160) at the same VQ-VAE density produces how many image tokens? Can a Chameleon-style model generate a 4K image in one inference call? What breaks first, context, tokenizer quality, or KV cache?
- Implement QK-Norm in pure Python. Given a 64-dim query and key, show the dot product before and after LayerNorm. Why is magnitude control important at depth?
- Read Chameleon Section 2.3 on training stability. Describe the exact failure mode the paper observed at 34B without QK-Norm. What was the "norm explosion" signature?
- Extend the toy decoder to emit a mixed-modality response given a text-only prompt. Measure how often the model picks image-first vs text-first given training-data distribution 60% text-first / 40% image-first.
Key Terms
| Term | What people say | What it actually means |
|---|---|---|
| Early fusion | "Unified tokens" | Images converted to discrete tokens sharing the transformer's vocabulary from step one |
| VQ-VAE | "Image tokenizer" | CNN + ViT + codebook that maps images to integer indices the transformer can predict |
| Shared vocabulary | "One dictionary" | A single token ID space covering text + image + modality separators |
| QK-Norm | "Attention stabilizer" | LayerNorm applied to query and key before their dot product, prevents norm blowup |
| Mixed-modality generation | "Text + image output" | Inference that autonomously produces interleaved text and image tokens in one pass |
| Codebook size | "K entries" | Number of discrete vectors the VQ-VAE can quantize to; trades compression for fidelity |
| Tokenizer ceiling | "Reconstruction limit" | Best PSNR achievable by decoding VQ tokens; bounds the model's image quality |
Further Reading
The full code
Every file for this lesson. Copy it, run it, then change it.
"""Chameleon-style early-fusion: toy VQ quantizer + shared-vocab autoregressive decoder.
End-to-end pipeline:
1. VQ-VAE-ish quantizer: 8x8 grayscale patch -> integer codebook index, K=16.
2. Shared vocab: text ids 0..31, image ids 32..47, separators 48 (<image>), 49 (</image>).
3. Bigram decoder trained on synthetic (text + <image> codes </image>) pairs.
4. Sampling loop that emits mixed-modality output.
Stdlib only. The transformer is a bigram count table — the point is to see the
shared-vocabulary loop in miniature, not to get image quality.
"""
from __future__ import annotations
import math
import random
from collections import defaultdict
random.seed(42)
VOCAB_TEXT = 32
VOCAB_IMG = 16
IMG_OFFSET = VOCAB_TEXT
SEP_OPEN = VOCAB_TEXT + VOCAB_IMG
SEP_CLOSE = SEP_OPEN + 1
VOCAB_SIZE = SEP_CLOSE + 1
CODEBOOK = [[(i * 7 + 3 * j) % 8 for j in range(4)] for i in range(VOCAB_IMG)]
def quantize_patch(patch: list[int]) -> int:
"""Nearest-codebook lookup by L2 distance."""
best = 0
best_d = float("inf")
for k, code in enumerate(CODEBOOK):
d = sum((p - c) ** 2 for p, c in zip(patch, code))
if d < best_d:
best_d = d
best = k
return best + IMG_OFFSET
def image_to_tokens(img: list[list[int]]) -> list[int]:
"""8x8 grayscale -> 4 patches of 4 floats (downsampled). Return token IDs."""
patches = []
for pr in range(0, 8, 4):
for pc in range(0, 8, 4):
flat = []
for r in range(2):
for c in range(2):
s = 0
for dr in range(2):
for dc in range(2):
s += img[pr + 2 * r + dr][pc + 2 * c + dc]
flat.append(s // 4)
patches.append(flat)
return [quantize_patch(p) for p in patches]
def synthesize_caption(kind: str) -> list[int]:
"""Pick a short synthetic text token sequence."""
if kind == "red":
return [1, 5, 3, 7]
if kind == "blue":
return [2, 5, 3, 8]
if kind == "green":
return [1, 5, 3, 9]
return [1, 5, 3, 10]
def synth_image(kind: str) -> list[list[int]]:
shade = {"red": 7, "blue": 2, "green": 4, "gray": 5}[kind]
return [[(shade + (r + c) % 3) for c in range(8)] for r in range(8)]
def make_dataset(n: int = 40) -> list[list[int]]:
kinds = ["red", "blue", "green", "gray"]
corpus = []
for _ in range(n):
k = random.choice(kinds)
tokens = synthesize_caption(k) + [SEP_OPEN] + image_to_tokens(synth_image(k)) + [SEP_CLOSE]
if random.random() < 0.4:
tokens = [SEP_OPEN] + image_to_tokens(synth_image(k)) + [SEP_CLOSE] + synthesize_caption(k)
corpus.append(tokens)
return corpus
def train_bigram(corpus: list[list[int]]) -> dict:
counts: dict = defaultdict(lambda: defaultdict(int))
for seq in corpus:
for a, b in zip(seq, seq[1:]):
counts[a][b] += 1
return counts
def sample_next(bigram: dict, prev: int) -> int:
dist = bigram.get(prev, {})
if not dist:
return random.randrange(VOCAB_SIZE)
total = sum(dist.values())
r = random.random() * total
acc = 0
for tok, c in dist.items():
acc += c
if r <= acc:
return tok
return next(iter(dist))
def generate(bigram: dict, prompt: list[int], max_len: int = 40) -> list[int]:
out = list(prompt)
while len(out) < max_len:
nxt = sample_next(bigram, out[-1])
out.append(nxt)
if nxt == SEP_CLOSE and any(t < VOCAB_TEXT for t in out):
break
return out
def render(tokens: list[int]) -> str:
parts = []
for t in tokens:
if t == SEP_OPEN:
parts.append("<image>")
elif t == SEP_CLOSE:
parts.append("</image>")
elif t < VOCAB_TEXT:
parts.append(f"w{t}")
else:
parts.append(f"i{t - IMG_OFFSET}")
return " ".join(parts)
def main() -> None:
print("=" * 60)
print("CHAMELEON EARLY-FUSION TOY (Phase 12, Lesson 11)")
print("=" * 60)
print("\n1. VQ tokenizer — 8x8 grayscale -> 4 patches -> 4 image tokens")
print("-" * 60)
for kind in ["red", "blue", "green", "gray"]:
img = synth_image(kind)
codes = image_to_tokens(img)
print(f" {kind:<6} -> codes {codes}")
print("\n2. Shared vocabulary layout")
print("-" * 60)
print(f" text tokens : 0..{VOCAB_TEXT - 1}")
print(f" image tokens : {IMG_OFFSET}..{IMG_OFFSET + VOCAB_IMG - 1}")
print(f" <image> : {SEP_OPEN}")
print(f" </image> : {SEP_CLOSE}")
print(f" vocab total : {VOCAB_SIZE}")
print("\n3. Dataset (40 sequences of interleaved text + image tokens)")
print("-" * 60)
corpus = make_dataset(40)
for seq in corpus[:4]:
print(" " + render(seq))
print("\n4. Train bigram, sample mixed-modality output")
print("-" * 60)
bigram = train_bigram(corpus)
for _ in range(3):
out = generate(bigram, [1, 5], max_len=30)
print(" " + render(out))
print("\nTAKEAWAY")
print("-" * 60)
print(" one model, one vocab, one loss -> mixed-modality output for free")
print(" tokenizer quality caps image fidelity (lesson 12.12 on Emu3)")
print(" at scale you need QK-Norm + careful dropout for stable training")
if __name__ == "__main__":
main()