Get Started

Projection Layer for Modality Alignment

A vision encoder produces image tokens. A text decoder consumes text tokens. The two live in different vector spaces. A small two-layer MLP projects image tokens into the text embedding space, and a cosine alignment loss against a paired caption pulls the two spaces into agreement. That projection is the smallest piece of a vision-language model and the one that matters most for transfer.

90 minutesBuildPythonMembers

Learning Objectives

  • Build a two-layer MLP projection that maps image features into the text embedding space.
  • Construct a mock text embedding table (no pretrained tokenizer, no real corpus).
  • Compute a cosine alignment loss between projected image tokens and a paired caption embedding.
  • Train the projection alone with a frozen vision encoder and a frozen text table.