Get Started

Vision Transformer Encoder

Patches alone do not see. A 12-layer pre-LN transformer with 12 attention heads turns the sequence of patch tokens into a sequence of contextual tokens, with the CLS token pooling whole-image features in its final hidden state. This lesson is the engine room of every modern vision-language model.

90 minutesBuildPythonMembers

Learning Objectives

  • Implement a pre-LN transformer block with multi-head self-attention and a feed-forward sub-layer.
  • Stack 12 blocks with 12 heads to form a ViT-Base encoder.
  • Wire the patch front end from lesson 58 into the encoder and run a forward pass.
  • Verify that the CLS token aggregates information from every patch.