Get Started

Multi-Token Prediction (MTP)

Every autoregressive LLM from GPT-2 to Llama 3 trains on one loss per position: predict the next token. DeepSeek-V3 added a second loss per position: predict the token after that. The extra 14B of parameters (on a 671B model) got distilled back into the main model through gradient flow, and the trained MTP heads were repurposed at inference as speculative-decoding drafters with 80%+ acceptance. 1.8× generation throughput came for free. This lesson builds the sequential MTP module from the DeepSeek technical report, computes the loss and the shared-head parameter layout, and explains why MTP keeps the causal chain while Gloeckle et al.'s original parallel MTP broke it.

60 minutesBuildPython (stdlib)Members

Learning Objectives

  • State the MTP training objective and derive the joint loss across prediction depths.
  • Explain the difference between Gloeckle et al.'s parallel MTP heads (2024) and DeepSeek-V3's sequential MTP modules and why the sequential design preserves the causal chain.
  • Compute the parameter and memory overhead of adding MTP modules to a pre-training run.
  • Implement one MTP module from scratch: the shared embedding, the per-depth transformer block, the projection, and the shared output head.