Get Started

Capstone 04, Multimodal Document QA (Vision-First PDF, Tables, Charts)

The 2026 document-QA frontier moved away from OCR-then-text and toward vision-first late interaction. ColPali, ColQwen2.5, and ColQwen3-omni treat each PDF page as an image, embed it with multi-vector late interaction, and let the query attend to patches directly. On financial 10-Ks, scientific papers, and handwritten notes this pattern beats OCR-first by a large margin. Build the pipeline end to end on 10k pages and publish the side-by-side against OCR-then-text.

30 hoursCapstonePython (pipeline), TypeScript (viewer UI)Free lesson

Problem

Enterprises sit on PDFs that OCR pipelines mangle: scanned 10-Ks with rotated tables, scientific papers dense with equations, charts that only make sense as images, handwritten annotations. Treating these as text-first means losing half the signal. The 2026 answer is late-interaction multi-vector retrieval on raw page images. ColPali (Illuin Tech) introduced it; ColQwen2.5-v0.2 and ColQwen3-omni pushed accuracy. On ViDoRe v3, vision-first retrieval scores above OCR-then-text by meaningful margins, and the gap widens on charts, tables, and handwriting.

The trade-off is storage and latency. A ColQwen embedding is ~2048 patch vectors per page, not a single 1024-dim vector. Raw storage balloons. DocPruner (2026) brings 50% pruning without measurable accuracy loss. You will index 10k pages, measure ViDoRe v3 nDCG@5, serve answers under 2s, and compare directly against an OCR-then-text baseline.

Concept

Late interaction means every query token scores against every patch token, and the maximum score per query token is summed. You get fine-grained matching without needing a single pooled vector. A multi-vector index (Vespa, Qdrant multi-vector, or AstraDB) stores the per-patch embeddings and runs MaxSim at retrieval time.

The answerer is a vision-language model that takes the query plus the top-k retrieved pages as images and writes an answer with evidence regions (bounding boxes or page references). Qwen3-VL-30B, Gemini 2.5 Pro, and InternVL3 are the 2026 frontier choices. For equations and scientific notation, an OCR fallback (Nougat, dots.ocr) is spliced in as an optional text channel.

Evaluation is a two-dimensional matrix. One axis: content type (plain text paragraphs, dense tables, bar/line charts, handwritten notes, equations). Other axis: retrieval approach (vision-first late interaction vs OCR-then-text vs hybrid). Each cell gets nDCG@5 and answer accuracy. The report is the deliverable.

Architecture

Code
PDFs -> page renderer (PyMuPDF, 180 DPI)
           |
           v
  ColQwen2.5-v0.2 embed (multi-vector per page, ~2048 patches)
           |
           +------> DocPruner 50% compression
           |
           v
   multi-vector index (Vespa or Qdrant multi-vector)
           |
query ----+----> retrieve top-k pages (MaxSim)
           |
           v
  VLM answerer: Qwen3-VL-30B | Gemini 2.5 Pro | InternVL3
    inputs: query + top-k page images + optional OCR text
           |
           v
  answer with cited page numbers + evidence regions
           |
           v
  Streamlit / Next.js viewer: highlighted boxes on source page

Stack

  • Page rendering: PyMuPDF (fitz) at 180 DPI, portrait-normalized
  • Late-interaction model: ColQwen2.5-v0.2 or ColQwen3-omni (vidore team on Hugging Face)
  • Index: Vespa with multi-vector field, or Qdrant multi-vector, or AstraDB with MaxSim
  • Pruning: DocPruner 2026 policy (keep high-variance patches, 50% compression at < 0.5% accuracy loss)
  • OCR fallback (equations / dense tables): dots.ocr or Nougat
  • VLM answerer: Qwen3-VL-30B self-hosted or Gemini 2.5 Pro hosted; InternVL3 as fallback
  • Evaluation: ViDoRe v3 benchmark, M3DocVQA for multi-page reasoning
  • Viewer UI: Next.js 15 with canvas overlay for evidence regions

Build It

  1. Ingest. Walk a corpus of 10k PDF pages across 10-Ks, scientific papers, and scanned documents. Render each page to a 1536x2048 PNG. Persist {doc_id, page_num, image_path}.
  2. Embed. Run ColQwen2.5-v0.2 on each page image. Output shape ~2048 patch embeddings of dim 128. Apply DocPruner to keep the highest-signal half. Write to Vespa multi-vector field or Qdrant multi-vector.
  3. Query. For each incoming query, embed with the query tower (token-level embeddings). Run MaxSim against the index: for every query token, take the max dot-product over page patch embeddings, sum. Return top-k pages.
  4. Synthesize. Call Qwen3-VL-30B with the query and the top-5 page images. Prompt: "Answer using only the supplied pages. Cite each claim by (doc_id, page) and name the region (figure, table, paragraph)."
  5. Evidence regions. Post-process the answer to extract cited regions. If the VLM emits bounding boxes (Qwen3-VL does), render them as overlays in the viewer.
  6. OCR fallback. For pages identified as equation-dense (heuristic on image variance), run Nougat or dots.ocr and pass the OCR text as an extra channel alongside the image.
  7. Eval. Run ViDoRe v3 (retrieval nDCG@5) and M3DocVQA (multi-page QA accuracy). Also run OCR-then-text pipeline on the same corpus with the same synthesizer. Produce a content-type × approach matrix.
  8. UI. Streamlit prototype first; Next.js 15 production viewer with page-by-page evidence-region overlay.

Use It

Code
$ doc-qa ask "what was the 2024 operating margin change for segment EMEA?"
[retrieve]   top-5 pages in 320ms (ColQwen2.5, MaxSim, Vespa)
[synth]      qwen3-vl-30b, 1.4s, cited (form-10k-2024, p. 88) + (..., p. 92)
answer:
  EMEA operating margin moved from 18.2% to 16.8%, a 140bp decline.
  cited: 10-K-2024.pdf p.88 (Table 4, Segment Operating Margin)
         10-K-2024.pdf p.92 (MD&A, Operating Performance)
[viewer]     open with highlighted bounding boxes overlaid on p.88 Table 4

Ship It

outputs/skill-doc-qa.md describes the deliverable: a vision-first multimodal document QA system tuned to a specific corpus and evaluated against an OCR-then-text baseline on ViDoRe v3.

WeightCriterionHow it is measured
25ViDoRe v3 / M3DocVQA accuracyBenchmark numbers vs OCR-text baseline and published leaderboard
20Evidence-region groundingFraction of cited regions that actually contain the answer span
20Storage and latency engineeringDocPruner compression ratio, index p95, answer p95
20Multi-page reasoningAccuracy on a hand-labeled 100-question multi-page set
15Source-inspection UXViewer clarity, overlay fidelity, side-by-side comparison tools
100

Exercises

  1. Measure ColQwen2.5-v0.2 vs ColQwen3-omni on the same corpus. Which pages does one get right and the other miss? Add a "content class" tag to the index to route by type.
  2. Prune embeddings aggressively (75%, 90%). Find the compression cliff: the point where ViDoRe nDCG@5 drops below the OCR baseline.
  3. Build a hybrid: run OCR-then-text and ColQwen in parallel, fuse with RRF, rerank with a cross-encoder. Does the hybrid beat either alone? Where does it help most?
  4. Swap Qwen3-VL-30B for a smaller VLM (Qwen2.5-VL-7B). Measure the accuracy-per-dollar curve.
  5. Add handwritten-note support. Render the handwriting corpus, embed with ColQwen, measure retrieval. Compare against a handwriting OCR pipeline.

Key Terms

TermWhat people sayWhat it actually means
Late interaction"ColPali-style retrieval"Query tokens score against page patches independently; MaxSim aggregates
Multi-vector"Per-patch embedding"Each document has many vectors, not one pooled vector
MaxSim"Late-interaction scoring"For every query token, take max similarity over document vectors; sum
DocPruner"Patch compression"2026 pruning that keeps 50% of patches with negligible accuracy loss
ViDoRe v3"Document-retrieval benchmark"The 2026 standard for measuring visual-document retrieval
Evidence region"Cited bounding box"A bbox on the source page that localizes the answer span
OCR fallback"Equation channel"Text pipeline used alongside vision for equation- or table-heavy pages

Further Reading

Check what you learned

7 questions. Answer each one to see why.

Question 1 of 70 correct

Why does the 2026 frontier prefer vision-first late interaction over OCR-then-text on financial PDFs and scientific papers?

The full code

Every file for this lesson. Copy it, run it, then change it.

main.py
"""Multimodal document QA — ColPali-style late interaction scaffold.

The hard architectural primitive is late-interaction retrieval: every query
token scores against every document patch, the MaxSim per query token is
summed, the top-k pages are returned. This scaffold implements MaxSim end to
end on synthetic patch embeddings so the algorithm is observable without
loading a real ColQwen model. Includes DocPruner-style patch pruning.

Run:  python main.py
"""

from __future__ import annotations

import math
import random
import re
from dataclasses import dataclass, field


# ---------------------------------------------------------------------------
# patch embeddings  --  fake 16-dim patch vectors per page
# ---------------------------------------------------------------------------

EMB_DIM = 16


def tokenize(text: str) -> list[str]:
    return re.findall(r"\w+", text.lower())


def hash_embed(tok: str) -> list[float]:
    rnd = random.Random(hash(tok) & 0xFFFFFFFF)
    v = [rnd.gauss(0, 1) for _ in range(EMB_DIM)]
    n = math.sqrt(sum(x * x for x in v)) or 1.0
    return [x / n for x in v]


@dataclass
class Page:
    doc_id: str
    page_num: int
    content_tokens: list[str]          # stand-in for page contents
    patches: list[list[float]] = field(default_factory=list)

    def embed_patches(self) -> None:
        """Multi-vector: each content token becomes a patch vector."""
        self.patches = [hash_embed(t) for t in self.content_tokens]


# ---------------------------------------------------------------------------
# DocPruner  --  keep top-fraction patches by norm variance
# ---------------------------------------------------------------------------

def doc_prune(patches: list[list[float]], keep_fraction: float = 0.5) -> list[list[float]]:
    """Keep patches with highest per-patch norm (poor proxy for info density
    but matches the DocPruner intuition: drop low-signal patches)."""
    scored = [(sum(abs(x) for x in p), p) for p in patches]
    scored.sort(key=lambda x: -x[0])
    keep_n = max(1, int(len(scored) * keep_fraction))
    return [p for _, p in scored[:keep_n]]


# ---------------------------------------------------------------------------
# MaxSim late interaction  --  the algorithmic core of ColPali / ColQwen
# ---------------------------------------------------------------------------

def dot(a: list[float], b: list[float]) -> float:
    return sum(x * y for x, y in zip(a, b))


def max_sim_score(query_tokens: list[list[float]],
                  doc_patches: list[list[float]]) -> float:
    """For every query token embedding, take max dot product against any
    doc patch; sum across query tokens. This is MaxSim / late interaction."""
    total = 0.0
    for q in query_tokens:
        best = -1e9
        for p in doc_patches:
            s = dot(q, p)
            if s > best:
                best = s
        total += best
    return total


# ---------------------------------------------------------------------------
# index + retrieval  --  ranked top-k by MaxSim
# ---------------------------------------------------------------------------

@dataclass
class Index:
    pages: list[Page] = field(default_factory=list)

    def add(self, p: Page) -> None:
        self.pages.append(p)

    def retrieve(self, query: str, k: int = 5) -> list[tuple[Page, float]]:
        q_tokens = [hash_embed(t) for t in tokenize(query)]
        scored = [(pg, max_sim_score(q_tokens, pg.patches)) for pg in self.pages]
        scored.sort(key=lambda x: -x[1])
        return scored[:k]


# ---------------------------------------------------------------------------
# synthetic corpus  --  ten pages spanning tables, charts, handwriting, text
# ---------------------------------------------------------------------------

CORPUS = [
    ("10k-2024", 88, "segment EMEA operating margin 18.2 to 16.8 decline 140bp table four"),
    ("10k-2024", 92, "MDA operating performance EMEA macro headwinds FX impact narrative"),
    ("10k-2024", 14, "executive summary revenue growth 7 percent consolidated totals"),
    ("paper-vidore-v3", 3, "late interaction multi vector retrieval ColPali ColQwen benchmark"),
    ("paper-vidore-v3", 7, "nDCG results table vision first vs OCR then text columns"),
    ("paper-m3docrag", 2, "M3DocVQA multi page reasoning evaluation protocol"),
    ("handwritten-lab", 5, "experiment notes circuit board pH readings handwritten"),
    ("handwritten-lab", 6, "graph with annotated error bars figure 3 caption"),
    ("chart-report", 11, "line chart revenue by segment EMEA americas APAC Q1 Q4"),
    ("chart-report", 12, "bar chart operating margin by segment with 2023 2024 comparison"),
]


def build_index(prune: bool = True) -> Index:
    idx = Index()
    for doc, page, text in CORPUS:
        p = Page(doc_id=doc, page_num=page, content_tokens=tokenize(text))
        p.embed_patches()
        if prune:
            p.patches = doc_prune(p.patches, keep_fraction=0.5)
        idx.add(p)
    return idx


def main() -> None:
    print("=== build index with DocPruner (50% patches) ===")
    idx = build_index(prune=True)
    print(f"pages indexed: {len(idx.pages)}")

    queries = [
        "what was the 2024 operating margin change for EMEA",
        "late interaction retrieval vs OCR",
        "handwritten experimental figures with error bars",
        "bar chart comparing segment margins",
    ]

    for q in queries:
        print(f"\nQ: {q}")
        hits = idx.retrieve(q, k=3)
        for pg, score in hits:
            print(f"  score={score:+.3f}  {pg.doc_id} p.{pg.page_num}")

    # pruning ablation
    print("\n=== ablation: pruning off vs on ===")
    full = build_index(prune=False)
    pruned = build_index(prune=True)
    q = "chart comparing segment margins"
    full_top = [(p.doc_id, p.page_num) for p, _ in full.retrieve(q, 3)]
    prn_top = [(p.doc_id, p.page_num) for p, _ in pruned.retrieve(q, 3)]
    print(f"  full    top-3 : {full_top}")
    print(f"  pruned  top-3 : {prn_top}")
    print(f"  overlap       : {len(set(full_top) & set(prn_top))}/3")


if __name__ == "__main__":
    main()