Self-Supervised Vision, SimCLR, DINO, MAE
Labels are the bottleneck of supervised vision. Self-supervised pretraining removes them: learn visual features from 100M unlabelled images, fine-tune on 10k labelled ones.
75 minutesLearn + BuildPythonMembers
Learning Objectives
- Trace the three major self-supervised families, contrastive (SimCLR), teacher-student (DINO), masked reconstruction (MAE), and state what each one optimises
- Implement an InfoNCE loss from scratch and explain why a batch of 512 works but a batch of 32 fails
- Explain why MAE's 75% masking ratio is not arbitrary and how it differs from BERT's 15% for text
- Use DINOv2 or MAE ImageNet checkpoints for linear probing and zero-shot retrieval