Get Started

Video-Language Models: Temporal Tokens and Grounding

Video is not a stack of photos. A 5-second clip has causal ordering, action verbs, and event timing that an image model cannot represent. Video-LLaMA (Zhang et al., June 2023) shipped the first open video-LLM with audio-visual grounding. VideoChat and Video-LLaVA scaled the pattern. By 2025 Qwen2.5-VL's TMRoPE closed the gap with frontier proprietary models. Each system solved temporal tokens differently, Q-former per clip, concat-pool per frame, TMRoPE per token. This lesson reads the patterns, builds a uniform-vs-dynamic frame sampler, and evaluates on temporal grounding tasks.

180 minutesBuildPython (stdlib, frame sampler + temporal-grounding evaluator)Members

Learning Objectives

  • Explain why temporal positional encoding changes video VLM performance independently of the vision encoder.
  • Compare uniform, dynamic-FPS, and event-driven frame sampling on tokens-per-second vs grounding accuracy.
  • Describe Q-former-per-clip (Video-LLaMA) vs pooled-per-frame (Video-LLaVA) vs M-RoPE-per-token (Qwen2.5-VL) designs.
  • Name the four video benchmarks: VideoMME, TempCompass, EgoSchema, Video-MMMU.