Get Started

Capstone 12, Video Understanding Pipeline (Scene, QA, Search)

Twelve Labs productized Marengo + Pegasus. VideoDB shipped the CRUD-for-video API. AI2's Molmo 2 published open VLM checkpoints. Gemini long-context handles hours of video natively. TimeLens-100K defined temporal grounding at scale. The 2026 pipeline is settled: scene segmentation, per-scene caption + embedding, transcript alignment, multi-vector index, and a query that answers with (start, end) timestamps plus frame previews. The capstone is ingesting 100 hours, hitting public benchmarks, and measuring hallucination on counting and action questions.

30 hoursCapstonePython (pipeline), TypeScript (UI)Free lesson

Problem

Long-form video QA is the most bandwidth-hungry multimodal problem at 2026 scale. Gemini 2.5 Pro can read a 2-hour video natively, but ingesting 100 hours of video into a queryable corpus still requires a scene-level index. The production shape combines scene segmentation (TransNetV2 or PySceneDetect), per-scene captioning with a VLM (Gemini 2.5, Qwen3-VL-Max, or Molmo 2), transcript alignment (Whisper-v3-turbo with word timestamps), and a multi-vector index that stores caption, frame embedding, and transcript side by side. The query pipeline answers with (start, end) timestamps plus frame previews.

Benchmarks are public (ActivityNet-QA, NeXT-GQA) plus your own 100-query custom set. Hallucination on counting and action-type questions is the known-hard failure class; the capstone explicitly measures it.

Concept

Three pipelines run in parallel at ingest. Scene segmentation cuts the video into scenes. VLM captioning generates a caption per scene and a frame embedding from a keyframe. ASR alignment produces word-level timestamps. The three streams are joined by (scene_id, time range). Each scene gets three vector types in a multi-vector index (Qdrant): caption embedding, keyframe embedding, transcript embedding.

At query time, the natural-language question fires against all three vectors; results merge with RRF; a temporal-grounding adapter (TimeLens-style) refines the (start, end) window within the top scene. The VLM synthesizer (Gemini 2.5 Pro or Qwen3-VL-Max) takes query + top scenes + cropped frames and answers with cited timestamps and a frame preview.

The hallucination measurement matters. Counting ("how many people enter the room?") and action-type ("does the chef pour before stirring?") questions are notoriously unreliable. Report accuracy separately from descriptive questions.

Architecture

Code
video file / URL
      |
      v
PySceneDetect / TransNetV2  (scene segmentation)
      |
      +--- per-scene keyframe --- VLM caption + frame embedding
      |                            (Gemini 2.5 Pro / Qwen3-VL-Max / Molmo 2)
      |
      +--- audio channel --- Whisper-v3-turbo ASR + word timestamps
      |
      v
multi-vector Qdrant: {caption_emb, keyframe_emb, transcript_emb}
      |
query:
  dense queries against all three -> RRF merge -> top-k scenes
      |
      v
TimeLens / VideoITG temporal grounding (refine start/end within scene)
      |
      v
VLM synth: query + top scenes + frame previews
      |
      v
answer + (start, end) timestamps + frame thumbs + citations

Stack

  • Scene segmentation: TransNetV2 (state-of-the-art 2024-26) or PySceneDetect
  • ASR: Whisper-v3-turbo via faster-whisper with word timestamps
  • VLM captioner + answerer: Gemini 2.5 Pro or Qwen3-VL-Max or Molmo 2
  • Temporal grounding: TimeLens-100K-trained adapter or VideoITG
  • Index: Qdrant with multi-vector support (caption / frame / transcript)
  • UI: Next.js 15 with HTML5 video player and scene thumbnails
  • Eval: ActivityNet-QA, NeXT-GQA, custom 100-question hand-labeled set
  • Hallucination benchmark: counting and action-type subsets with hand labels

Build It

  1. Ingest walker. Accept YouTube URLs or local MP4s. Downscale to 720p if needed. Persist {video_id, file_path}.
  2. Scene segmentation. Run TransNetV2 or PySceneDetect to produce [{scene_id, start_ms, end_ms, keyframe_path}]. Target 100 hours: ~6k-8k scenes.
  3. ASR pass. Run Whisper-v3-turbo on audio; export word-level timestamps; split into per-scene transcript slices.
  4. VLM captioning. Per scene, call Gemini 2.5 Pro (or Qwen3-VL-Max) with the keyframe and a short caption template. Produce caption + frame embedding.
  5. Multi-vector index. Qdrant collection with three named vectors. Payload: {video_id, scene_id, start_ms, end_ms, keyframe_url}.
  6. Query. Natural-language question fires three dense queries; merge with reciprocal rank fusion; top-k=5 scenes.
  7. Temporal grounding. Run TimeLens-style adapter on the top scene to refine the (start, end) window within the scene.
  8. VLM synth. Call Gemini 2.5 Pro with query + top-3 scene clips (as images or short clips) + transcripts. Require (video_id, start_ms, end_ms) citations.
  9. Eval. Run ActivityNet-QA and NeXT-GQA. Build a 100-query custom set. Report overall accuracy + per-class breakdown (counting, action, descriptive).

Use It

Code
$ video-qa ask --url=https://youtube.com/watch?v=X "how many cars pass the intersection in the first minute?"
[scene]    23 scenes detected
[asr]      transcript complete, 4m12s
[index]    69 vectors written (23 scenes x 3)
[query]    top scene: scene 3 [01:32-01:54], confidence 0.84
[ground]   refined window: [00:12-00:58]
[synth]    gemini 2.5 pro, 1.4s
answer:    5 cars pass the intersection between 00:12 and 00:58.
citations: [scene 3: 00:12-00:58]
          [frame preview at 00:14, 00:27, 00:44, 00:51, 00:57]

Ship It

outputs/skill-video-qa.md is the deliverable. Given a YouTube URL or uploaded video, the pipeline indexes scenes and answers questions with timestamped citations.

WeightCriterionHow it is measured
25Temporal grounding IoUIntersection-over-union on held-out grounding set
20QA accuracyNeXT-GQA and custom 100-query
20Ingest throughputHours of video per dollar spent
20UI and citation UXTimestamp links, thumbnail strip, jump-to-frame
15Hallucination rateCounting and action-type accuracy separately
100

Exercises

  1. Swap Gemini 2.5 Pro for Qwen3-VL-Max on the captioning pass. Report caption quality delta on a human-rated 50-scene sample.
  2. Reduce per-scene frame embedding to one pooled vector instead of multi-vector. Measure the retrieval regression.
  3. Build a "counting strict" mode: the synthesizer extracts each counted instance with a timestamp and the user clicks to verify. Measure whether user-verification reduces hallucination.
  4. Benchmark ingest cost: hours-of-video-per-dollar across three VLM choices. Pick the sweet spot.
  5. Add speaker-diarized transcript: run pyannote speaker diarization on the audio and embed per-speaker transcripts. Demonstrate "what did Alice say about X?" queries.

Key Terms

TermWhat people sayWhat it actually means
Scene segmentation"Shot detection"Cutting video into scenes at shot boundaries
Multi-vector index"Caption + frame + transcript"Qdrant collection with named vectors per representation
Temporal grounding"When exactly did it happen"Refining the (start, end) window for a query answer
Frame embedding"Visual representation"A vector embedding of a keyframe; used for scene-visual similarity
RRF fusion"Reciprocal rank fusion"Merge strategy across multiple ranked lists; a classic hybrid-retrieval trick
Counting hallucination"Miscount"Known failure mode of VLMs on "how many X" questions
ActivityNet-QA"Video-QA benchmark"Long-form video QA accuracy benchmark

Further Reading

Check what you learned

7 questions. Answer each one to see why.

Question 1 of 70 correct

Why is a scene-level index still needed even when a long-context VLM can read a 2-hour video natively?

The full code

Every file for this lesson. Copy it, run it, then change it.

main.py
"""Video understanding pipeline — multi-vector scene index scaffold.

The hard architectural primitive is a multi-vector-per-scene index with
three representations (caption, frame, transcript), queried in parallel and
merged with reciprocal rank fusion, then refined by a temporal-grounding
step that picks a sub-window inside the best scene. This scaffold implements
the index shape, the triple-query fusion, and the sub-window grounding.

Run:  python main.py
"""

from __future__ import annotations

import math
import random
import re
from collections import defaultdict
from dataclasses import dataclass, field


EMB_DIM = 24


def tokenize(s: str) -> list[str]:
    return re.findall(r"\w+", s.lower())


def fake_embed(text: str) -> list[float]:
    v = [0.0] * EMB_DIM
    for tok in tokenize(text):
        h = hash(tok)
        v[h % EMB_DIM] += 1.0
        v[(h >> 8) % EMB_DIM] += 0.5
    n = math.sqrt(sum(x * x for x in v)) or 1.0
    return [x / n for x in v]


def cosine(a: list[float], b: list[float]) -> float:
    return sum(x * y for x, y in zip(a, b))


# ---------------------------------------------------------------------------
# scene record  --  multi-vector: caption / frame / transcript
# ---------------------------------------------------------------------------

@dataclass
class Scene:
    video_id: str
    scene_id: int
    start_ms: int
    end_ms: int
    caption: str
    transcript: str
    frame_tags: str              # stand-in for frame embedding features
    caption_emb: list[float] = field(default_factory=list)
    frame_emb: list[float] = field(default_factory=list)
    transcript_emb: list[float] = field(default_factory=list)

    def embed(self) -> None:
        self.caption_emb = fake_embed(self.caption)
        self.frame_emb = fake_embed(self.frame_tags)
        self.transcript_emb = fake_embed(self.transcript)


SAMPLE = [
    Scene("vid_001", 0,       0,  32_000, "sunrise over skyline, drone footage",
          "we start here in tokyo",
          "skyline buildings dawn orange sky haze"),
    Scene("vid_001", 1,  32_000,  68_000, "busy intersection with pedestrians",
          "shibuya crossing after sunrise",
          "street people walking cars traffic signal"),
    Scene("vid_001", 2,  68_000, 132_000, "cars stopped at a red light",
          "let me count the vehicles approaching",
          "cars red light queue crossing lanes"),
    Scene("vid_001", 3, 132_000, 170_000, "kitchen scene chef pouring then stirring",
          "first we pour then we stir it slowly",
          "chef pan stove pour stir ingredient"),
    Scene("vid_001", 4, 170_000, 210_000, "chef plating the finished dish",
          "plated presentation of the dish",
          "plate garnish spoon finishing dish"),
    Scene("vid_002", 0,       0,  40_000, "ocean waves at sunset",
          "beautiful evening at the shore",
          "ocean waves sunset sky shore"),
]


# ---------------------------------------------------------------------------
# triple-vector query + RRF merge
# ---------------------------------------------------------------------------

def multi_vector_search(query: str, scenes: list[Scene], k: int = 5) -> list[tuple[Scene, float]]:
    qv = fake_embed(query)
    scored_caption = sorted(scenes, key=lambda s: -cosine(qv, s.caption_emb))
    scored_frame = sorted(scenes, key=lambda s: -cosine(qv, s.frame_emb))
    scored_transcript = sorted(scenes, key=lambda s: -cosine(qv, s.transcript_emb))

    fused: dict[tuple[str, int], float] = defaultdict(float)
    index: dict[tuple[str, int], Scene] = {}
    for ranks, stream in ((scored_caption, "cap"),
                          (scored_frame, "frm"),
                          (scored_transcript, "trn")):
        for rank, sc in enumerate(ranks):
            key = (sc.video_id, sc.scene_id)
            fused[key] += 1.0 / (60 + rank + 1)
            index[key] = sc

    ranked = sorted(fused.items(), key=lambda x: -x[1])
    return [(index[k_], s) for k_, s in ranked[:k]]


# ---------------------------------------------------------------------------
# temporal grounding stub  --  refine start/end within the best scene
# ---------------------------------------------------------------------------

def ground_window(query: str, scene: Scene) -> tuple[int, int]:
    """Stand-in: pick a sub-window of the scene based on query keyword position."""
    q = set(tokenize(query))
    t_tokens = tokenize(scene.transcript)
    if not q or not t_tokens:
        return scene.start_ms, scene.end_ms
    positions = [i for i, w in enumerate(t_tokens) if w in q]
    if not positions:
        return scene.start_ms, scene.end_ms
    span = scene.end_ms - scene.start_ms
    start_frac = min(positions) / max(1, len(t_tokens))
    end_frac = (max(positions) + 1) / max(1, len(t_tokens))
    start = int(scene.start_ms + span * max(0.0, start_frac - 0.05))
    end = int(scene.start_ms + span * min(1.0, end_frac + 0.05))
    return start, end


# ---------------------------------------------------------------------------
# demo
# ---------------------------------------------------------------------------

def fmt_ms(ms: int) -> str:
    s = ms // 1000
    return f"{s // 60:02d}:{s % 60:02d}"


def main() -> None:
    scenes = SAMPLE
    for s in scenes:
        s.embed()

    queries = [
        ("how many cars pass through the intersection", False),
        ("what happened first pour or stir", False),
        ("plating of the dish", True),
        ("ocean at sunset", True),
    ]

    for q, descriptive in queries:
        print(f"\nQ: {q}  (descriptive={descriptive})")
        hits = multi_vector_search(q, scenes, k=3)
        for sc, score in hits:
            print(f"  scene {sc.video_id}/{sc.scene_id} @ [{fmt_ms(sc.start_ms)}-{fmt_ms(sc.end_ms)}] "
                  f"score={score:.4f}  cap='{sc.caption[:40]}'")
        top = hits[0][0]
        start, end = ground_window(q, top)
        print(f"  grounded window: [{fmt_ms(start)}-{fmt_ms(end)}] "
              f"(narrowed from {fmt_ms(top.start_ms)}-{fmt_ms(top.end_ms)})")


if __name__ == "__main__":
    main()