Native Sparse Attention (DeepSeek NSA)
At 64k tokens, attention eats 70-80% of decode latency. Every open-model lab has a plan to fix it. DeepSeek's NSA (ACL 2025 best paper) is the one that stuck: three parallel attention branches, compressed coarse-grained tokens, selectively retained fine-grained tokens, and sliding windows for local context, combined through a learned gate. It is hardware-aligned (kernel-friendly), natively trainable (works in pre-training, not bolted on at inference), and on 64k decodes it runs faster than FlashAttention while matching or beating full attention quality. This lesson builds the three branches end-to-end and shows why the sparsity is end-to-end differentiable.
Learning Objectives
- State the three NSA attention branches and what each one captures.
- Explain why NSA is "natively trainable" where prior sparse-attention methods were inference-only.
- Compute the attention compute savings of NSA versus full attention at 64k context as a function of compression block size and selection top-k.
- Implement the three-branch combination in stdlib Python on a short synthetic sequence and verify the gating weights behave.