Projection Layer for Modality Alignment
A vision encoder produces image tokens. A text decoder consumes text tokens. The two live in different vector spaces. A small two-layer MLP projects image tokens into the text embedding space, and a cosine alignment loss against a paired caption pulls the two spaces into agreement. That projection is the smallest piece of a vision-language model and the one that matters most for transfer.
90 minutesBuildPythonMembers
Learning Objectives
- Build a two-layer MLP projection that maps image features into the text embedding space.
- Construct a mock text embedding table (no pretrained tokenizer, no real corpus).
- Compute a cosine alignment loss between projected image tokens and a paired caption embedding.
- Train the projection alone with a frozen vision encoder and a frozen text table.