Transformers Deep Dive
The architecture that changed everything.
16 lessonsAbout 17 hoursExpert level8 free lessonsBy AI engineering expert Rohit Ghumare
Start this phase0 of 16 done
- 01Why Transformers, The Problems with RNNsRNNs process tokens one at a time. Transformers process all tokens at once. That single architectural bet changed every scaling curve in deep learning after 2017.
- 02Self-Attention from ScratchAttention is a lookup table where every word asks "who matters to me?" - and learns the answer.
- 03Multi-Head AttentionOne attention head learns one relation at a time. Eight heads learn eight. Heads are free. Take more of them.
- 04Positional Encoding, Sinusoidal, RoPE, ALiBiAttention is permutation-invariant. "The cat sat on the mat" and "mat the on sat cat the" produce the same output without positional signal. Three algorithms fix it, each with a different bet on what "position" means.
- 05The Full Transformer, Encoder + DecoderAttention is the star. Everything else, residuals, normalization, feed-forward, cross-attention, is the scaffolding that lets you stack it deep.
- 06BERT, Masked Language ModelingGPT predicts the next word. BERT predicts a missing word. One sentence of difference, and half a decade of everything embedding-shaped.
- 07GPT, Causal Language ModelingBERT sees both sides. GPT sees only the past. The triangle mask is the most consequential single line of code in modern AI.
- 08T5, BART, Encoder-Decoder ModelsEncoders understand. Decoders generate. Put them back together and you get a model built for input → output tasks: translate, summarize, rewrite, transcribe.
- 09Vision Transformers (ViT)An image is a grid of patches. A sentence is a grid of tokens. The same transformer eats both.
- 10Audio Transformers, Whisper ArchitectureAudio is an image of frequency over time. Whisper is a ViT that eats mel spectrograms and speaks back.
- 11Mixture of Experts (MoE)A dense 70B transformer activates every parameter for every token. A 671B MoE activates only 37B per token and beats it on every benchmark. Sparsity is the most important scaling idea of the decade.
- 12KV Cache, Flash Attention & Inference OptimizationTraining is parallel and FLOP-bound. Inference is serial and memory-bound. Different bottleneck, different tricks.
- 13Scaling LawsThe 2020 Kaplan paper said: bigger model, lower loss. The 2022 Hoffmann paper said: you were under-training. Compute goes into two buckets, parameters and tokens, and the split is not obvious.
- 14Build a Transformer from Scratch, The CapstoneThirteen lessons. One model. No shortcuts.
- 15Attention Variants, Sliding Window, Sparse, DifferentialFull attention is a circle. Every token sees every token, and memory pays the price. Four variants bend the shape of the circle and recover half the cost.
- 16Speculative Decoding, Draft, Verify, RepeatAutoregressive decoding is serial. Each token waits for the previous one. Speculative decoding breaks the chain: a cheap model drafts N tokens, the expensive model verifies all N in one forward pass. When the draft is right you paid one big forward for N generations.