Jamba, Hybrid SSM-Transformer
State space models (SSMs) and transformers want different things. Transformers buy quality via attention at quadratic cost. SSMs buy linear-time inference and constant memory via a recurrence but lag quality. AI21's Jamba (March 2024) and Jamba 1.5 (August 2024) put them in the same model: 1 Transformer layer for every 7 Mamba layers, MoE on every other block, and a 256k context window that fits on a single 80GB GPU. Mamba-3 (ICLR 2026) tightens the SSM side with complex-valued state spaces and MIMO projections. This lesson reads both architectures end to end and explains why the hybrid recipe has survived three years of scaling when pure-SSM and pure-Transformer long-context attempts have not.
Learning Objectives
- Explain the three primitives in a Jamba block, Transformer layers, Mamba layers, MoE, and the 1:7:even interleaving recipe.
- State what an SSM's recurrence looks like at a high level and why it enables constant-memory inference.
- Compute the KV cache footprint of a Jamba model at 256k context and compare to what a pure-Transformer model would need.
- Name the three Mamba-3 innovations (exponential-trapezoidal discretization, complex-valued state update, MIMO) and the problem each one targets.