Capstone 02, RAG over Codebase (Cross-Repo Semantic Search)
Every serious engineering org in 2026 runs an internal code search that understands meaning, not just strings. Sourcegraph Amp, Cursor's codebase answers, Augment's enterprise graph, Aider's repomap, Pinterest's internal MCP, same shape. Ingest many repos, parse with tree-sitter, embed function- and class-level chunks, hybrid-search, re-rank, answer with citations. This capstone asks you to build one that handles 2M lines of code across 10 repos and survives incremental re-indexing on every git push.
Problem
By 2026 every frontier coding agent ships with a codebase retrieval layer because context windows alone do not solve cross-repo questions. Claude's 1M-token context helps; it does not eliminate the need for ranked retrieval. Naive cosine search over raw chunks poisons results on generated code, on monorepo duplication, and on the long tail of rarely-imported symbols. The production answer is a hybrid (dense + BM25) search over AST-aware chunks with a re-ranker, backed by a graph of symbol references.
You learn this by indexing a real fleet, not one tutorial repo, and measuring MRR@10, citation faithfulness, and incremental freshness. The failure modes are infrastructural: a 100k-file monorepo, a push that retouches half the files, a query that needs to cross four repos to answer correctly.
Concept
An AST-aware ingestion pipeline parses each file with tree-sitter, extracts function and class nodes, and chunks at node boundaries rather than fixed token windows. Each chunk gets three representations: a dense embedding (Voyage-code-3 or nomic-embed-code), sparse BM25 terms, and a short natural-language summary. The summary adds a third retrievable modality, users ask "how is X authorized" and the summary mentions "authz", even if the code only has check_permission.
Retrieval is hybrid. A query fires both dense and BM25 searches, merges top-k, and hands the union to a cross-encoder re-ranker (Cohere rerank-3 or bge-reranker-v2-gemma-2b). The re-ranked list goes to a long-context synthesizer (Claude Sonnet 4.7 with prompt caching, or Llama 3.3 70B self-hosted) with instructions to cite every claim by file and line range. Answers without citations are rejected by a post-filter.
Incremental freshness is the infrastructure problem. Git push triggers a diff: which files changed, which symbols changed. Only affected chunks re-embed. Affected cross-file symbol edges (imports, method calls) get recomputed. The index stays consistent without reprocessing 2M lines each commit.
Architecture
git push --> webhook --> ingest worker (LlamaIndex Workflow)
|
v
tree-sitter parse + AST chunk
|
+--------------+----------------+
v v v
dense BM25 index summary (LLM)
(Voyage / bge) (Tantivy) (Haiku 4.5)
| | |
+------> Qdrant / pgvector <----+
|
v
symbol graph (Neo4j / kuzu)
|
query --> LangGraph agent (retrieve -> rerank -> synth)
|
v
Claude Sonnet 4.7 1M context
|
v
answer + file:line citationsStack
- Parsing: tree-sitter with 17 language grammars (Python, TS, Rust, Go, Java, C++, etc.)
- Dense embeddings: Voyage-code-3 (hosted) or nomic-embed-code-v1.5 (self-host), bge-code-v1 fallback
- Sparse index: Tantivy (Rust) with BM25F, field-weighted on symbol name vs body
- Vector DB: Qdrant 1.12 with hybrid search, or pgvector + pgvectorscale for teams under 50M vectors
- Chunk summary model: Claude Haiku 4.5 or Gemini 2.5 Flash, prompt-cached
- Re-ranker: Cohere rerank-3 or bge-reranker-v2-gemma-2b self-hosted
- Orchestration: LlamaIndex Workflows for ingestion, LangGraph for query agent
- Synthesizer: Claude Sonnet 4.7 (1M context) with prompt caching
- Symbol graph: Neo4j (managed) or kuzu (embedded) for import and call edges
- Observability: Langfuse spans per retrieval + synthesis step
Build It
- Ingestion walker. Iterate git history on every push hook. Collect changed files. For each file, parse with tree-sitter, extract function and class nodes with their full source span. Emit chunk records
{repo, path, start_line, end_line, symbol, body}. - Chunk summarizer. Batch chunks into Haiku 4.5 calls with prompt caching on the system preamble. Prompt: "Summarize this function in one sentence, naming its public contract and side effects." Store summary alongside the chunk.
- Embedding pool. Two parallel queues: dense (Voyage-code-3 batch 128) and summary (same model, but on the summary string). Write vectors to Qdrant with payload
{repo, path, start_line, end_line, symbol, kind}. - BM25 index. Field-weighted Tantivy index: symbol name weight 4, symbol body weight 1, summary weight 2. Enables "find the function named X" queries alongside "find the function that does X".
- Symbol graph. For each chunk, record edges: imports (this file uses symbol Y from repo Z), calls (this function calls method M on class C), inheritance. Store in kuzu. Used at query time to expand retrieval across repo boundaries.
- Query agent. LangGraph with three nodes.
retrievefires dense + BM25 in parallel, deduplicates by (repo, path, symbol).rerankruns the cross-encoder on top-50 and keeps top-10.synthcalls Claude Sonnet 4.7 with the reranked chunks in context, caches the system prompt, requires file:line citations. - Citation enforcement. Parse the model output; any claim without a
(repo/path:start-end)anchor gets flagged for re-ask or dropped. Return cited-only answer to the user. - Incremental re-index. On each webhook, compute the symbol-level diff. Only re-embed chunks whose text changed. Recompute symbol edges for chunks whose imports changed. Measure: a 50-file push re-indexed in under 60 seconds for a 2M-LOC fleet.
- Eval. Label 100 cross-repo questions with gold file:line answers. Measure MRR@10, nDCG@10, citation faithfulness (fraction of claims with verifiable anchors), and p50/p99 latency.
Use It
$ code-rag ask "how is S3 multipart abort wired into our retry budget?"
[retrieve] 12 chunks dense + 7 chunks bm25, 16 unique after dedup
[rerank] top-5 kept (cohere rerank-3)
[synth] claude-sonnet-4.7, cache hit rate 68%, 2.1s
answer:
Multipart aborts are triggered by `AbortMultipartOnFail` in
services/uploader/retry.go:122-148, which decrements the per-bucket
retry budget defined in config/budgets.yaml:34-51 ...
citations: [services/uploader/retry.go:122-148, config/budgets.yaml:34-51,
libs/s3client/multipart.ts:44-61]Ship It
Deliverable skill outputs/skill-codebase-rag.md. Given a corpus of repos, it stands up the ingestion pipeline, the hybrid index, and the query agent, and returns a cited answer for any cross-repo question. Rubric:
| Weight | Criterion | How it is measured |
|---|---|---|
| 25 | Retrieval quality | MRR@10 and nDCG@10 on a 100-question held-out set |
| 20 | Citation faithfulness | Fraction of answer claims with verifiable file:line anchors |
| 20 | Latency and scale | p95 query latency at 10k QPS on the indexed corpus size |
| 20 | Incremental indexing correctness | Time from git push to searchable on a 50-file commit |
| 15 | UX and answer formatting | Citation clickability, snippet previews, follow-up affordance |
| 100 |
Exercises
- Swap Voyage-code-3 for nomic-embed-code self-hosted. Measure the MRR@10 delta. Report whether the gap closes with re-ranking enabled.
- Inject 20% generated code (LLM-produced boilerplate) into the corpus and re-evaluate. Observe retrieval poisoning. Add a "generated" flag to the payload and down-weight those hits.
- Benchmark Qdrant hybrid search vs pgvector + pgvectorscale at your corpus size. Report p99 at batch size 1.
- Add a sampling-based drift check: weekly, rerun the 100-question eval. Alert on MRR@10 drop > 5%.
- Extend to cross-language symbol resolution: a Python function that calls a Go service over gRPC. Use the symbol graph to link them.
Key Terms
| Term | What people say | What it actually means |
|---|---|---|
| AST-aware chunking | "Function-level splits" | Cutting code at tree-sitter node boundaries instead of fixed token windows |
| Hybrid search | "Dense + sparse" | Run BM25 and vector search in parallel, merge top-k, rerank |
| Cross-encoder rerank | "Second-stage rank" | Model that scores each (query, candidate) pair together, more accurate than cosine |
| Prompt caching | "Cached system prompt" | 2026 Claude / OpenAI feature that discounts repeat prefix tokens up to 90% |
| Symbol graph | "Code graph" | Edges for imports, calls, inheritance across files and repos |
| Citation faithfulness | "Grounded answer rate" | Fraction of claims a user can verify by clicking the anchor and reading the referenced span |
| Incremental re-index | "Push-to-search time" | Wall-clock from git push to the changed symbols being queryable |
Further Reading
- Sourcegraph Ampproduction cross-repo code intelligence
- Sourcegraph Cody RAG architecturethe reference deep-dive for this capstone
- Aider repo-maptree-sitter ranked repo view
- Augment Code enterprise graphcommercial symbol-graph RAG
- Qdrant hybrid search docsreference implementation
- Voyage AI code embeddingsVoyage-code-3 details
- Cohere rerank-3cross-encoder reference
- Pinterest MCP internal searchinternal-platform reference
Check what you learned
7 questions. Answer each one to see why.
Why is naive cosine search over raw chunks insufficient for cross-repo code retrieval?
The full code
Every file for this lesson. Copy it, run it, then change it.
"""Code RAG — AST-aware chunking + hybrid retrieval scaffold.
The hard architectural primitive here is hybrid retrieval with rank fusion:
two index structures (dense vector, BM25) run in parallel, results are merged
with reciprocal rank fusion, then a re-ranker picks the final top-k. This
scaffold implements both halves with stdlib: a naive dense index (hash-based
fake embeddings so the loop runs deterministically offline) and a real BM25
from scratch. The fusion + rerank logic is the part that matters.
Run: python main.py
"""
from __future__ import annotations
import math
import re
from collections import Counter, defaultdict
from dataclasses import dataclass, field
# ---------------------------------------------------------------------------
# chunk shape -- AST-aware function-level chunks
# ---------------------------------------------------------------------------
@dataclass
class Chunk:
repo: str
path: str
start_line: int
end_line: int
symbol: str
body: str
summary: str = ""
def anchor(self) -> str:
return f"{self.repo}/{self.path}:{self.start_line}-{self.end_line}"
SAMPLE_CORPUS = [
Chunk("uploader", "services/retry.go", 122, 148, "AbortMultipartOnFail",
"if ctx.Err() != nil { return abort() }; decrement bucket budget; retry with backoff",
"aborts an in-flight S3 multipart upload and decrements the per-bucket retry budget"),
Chunk("uploader", "config/budgets.yaml", 34, 51, "bucket_budget",
"per_bucket_budget: 64; backoff_ms: [100, 500, 2500]; abort_threshold: 3",
"declares the retry budget and exponential backoff schedule per S3 bucket"),
Chunk("client", "libs/s3client/multipart.ts", 44, 61, "abortUpload",
"await s3.abortMultipartUpload({Bucket, Key, UploadId}); metrics.inc('s3.abort')",
"client-side S3 multipart abort with metrics instrumentation"),
Chunk("auth", "services/authz/check.py", 12, 38, "check_permission",
"def check_permission(user, resource, action): return policy.evaluate(user, resource, action)",
"central authorization gateway evaluating an OPA policy for user-resource-action"),
Chunk("auth", "libs/policy/opa.py", 88, 110, "evaluate",
"def evaluate(user, resource, action): return self.engine.query('authz', input=...)",
"OPA policy engine query wrapper for authorization checks"),
Chunk("catalog", "services/search/query.rs", 200, 240, "rank_fusion",
"pub fn rank_fusion(dense: Vec<Hit>, sparse: Vec<Hit>) -> Vec<Hit>",
"reciprocal rank fusion of dense and sparse retrieval results"),
]
# ---------------------------------------------------------------------------
# naive dense index -- deterministic fake embeddings for scaffold testing
# ---------------------------------------------------------------------------
def fake_embed(text: str, dim: int = 64) -> list[float]:
"""Hash-based deterministic embedding; stands in for Voyage-code-3."""
vec = [0.0] * dim
for tok in re.findall(r"\w+", text.lower()):
h = hash(tok)
vec[h % dim] += 1.0
vec[(h >> 8) % dim] += 0.5
norm = math.sqrt(sum(v * v for v in vec)) or 1.0
return [v / norm for v in vec]
def cosine(a: list[float], b: list[float]) -> float:
return sum(x * y for x, y in zip(a, b))
@dataclass
class DenseIndex:
vectors: list[tuple[Chunk, list[float]]] = field(default_factory=list)
def add(self, chunk: Chunk) -> None:
text = f"{chunk.symbol}\n{chunk.summary}\n{chunk.body}"
self.vectors.append((chunk, fake_embed(text)))
def search(self, query: str, k: int = 10) -> list[tuple[Chunk, float]]:
qv = fake_embed(query)
scored = [(c, cosine(qv, v)) for c, v in self.vectors]
scored.sort(key=lambda x: -x[1])
return scored[:k]
# ---------------------------------------------------------------------------
# BM25 from scratch -- the real algorithm, documents are Chunks
# ---------------------------------------------------------------------------
def tokenize(text: str) -> list[str]:
return re.findall(r"\w+", text.lower())
@dataclass
class BM25Index:
k1: float = 1.5
b: float = 0.75
docs: list[Chunk] = field(default_factory=list)
doc_lens: list[int] = field(default_factory=list)
df: Counter = field(default_factory=Counter)
tf: list[Counter] = field(default_factory=list)
avgdl: float = 0.0
def add(self, chunk: Chunk) -> None:
# field-weighted: symbol x4, summary x2, body x1
tokens = (tokenize(chunk.symbol) * 4 +
tokenize(chunk.summary) * 2 +
tokenize(chunk.body))
counts = Counter(tokens)
self.docs.append(chunk)
self.doc_lens.append(len(tokens))
self.tf.append(counts)
for term in counts:
self.df[term] += 1
self.avgdl = sum(self.doc_lens) / len(self.doc_lens)
def search(self, query: str, k: int = 10) -> list[tuple[Chunk, float]]:
q_terms = tokenize(query)
n = len(self.docs)
scores: list[float] = [0.0] * n
for term in q_terms:
if term not in self.df:
continue
idf = math.log((n - self.df[term] + 0.5) / (self.df[term] + 0.5) + 1.0)
for i, counts in enumerate(self.tf):
if term not in counts:
continue
f = counts[term]
dl = self.doc_lens[i]
denom = f + self.k1 * (1 - self.b + self.b * dl / self.avgdl)
scores[i] += idf * f * (self.k1 + 1) / denom
ranked = sorted(zip(self.docs, scores), key=lambda x: -x[1])
return [(c, s) for c, s in ranked[:k] if s > 0]
# ---------------------------------------------------------------------------
# reciprocal rank fusion -- the merge step of hybrid retrieval
# ---------------------------------------------------------------------------
def rrf(dense: list[tuple[Chunk, float]], sparse: list[tuple[Chunk, float]],
k_rrf: int = 60) -> list[tuple[Chunk, float]]:
score: dict[str, float] = defaultdict(float)
by_anchor: dict[str, Chunk] = {}
for rank, (c, _) in enumerate(dense):
score[c.anchor()] += 1.0 / (k_rrf + rank + 1)
by_anchor[c.anchor()] = c
for rank, (c, _) in enumerate(sparse):
score[c.anchor()] += 1.0 / (k_rrf + rank + 1)
by_anchor[c.anchor()] = c
fused = sorted(score.items(), key=lambda x: -x[1])
return [(by_anchor[a], s) for a, s in fused]
# ---------------------------------------------------------------------------
# stub reranker -- cross-encoder stand-in; rerank by query-symbol overlap
# ---------------------------------------------------------------------------
def rerank(query: str, candidates: list[tuple[Chunk, float]],
top_k: int = 5) -> list[tuple[Chunk, float]]:
q_toks = set(tokenize(query))
out: list[tuple[Chunk, float]] = []
for c, prior in candidates:
symbol_overlap = len(q_toks & set(tokenize(c.symbol))) * 3
summary_overlap = len(q_toks & set(tokenize(c.summary)))
out.append((c, prior + 0.3 * symbol_overlap + 0.1 * summary_overlap))
out.sort(key=lambda x: -x[1])
return out[:top_k]
# ---------------------------------------------------------------------------
# orchestrator -- the full retrieve -> fuse -> rerank flow
# ---------------------------------------------------------------------------
def answer(query: str, dense: DenseIndex, bm25: BM25Index) -> dict[str, object]:
dense_hits = dense.search(query, k=10)
sparse_hits = bm25.search(query, k=10)
fused = rrf(dense_hits, sparse_hits)
top = rerank(query, fused, top_k=5)
citations = [c.anchor() for c, _ in top]
return {
"query": query,
"dense_top": [c.anchor() for c, _ in dense_hits[:3]],
"sparse_top": [c.anchor() for c, _ in sparse_hits[:3]],
"fused_top": [c.anchor() for c, _ in fused[:5]],
"rerank_top": citations,
}
def main() -> None:
dense = DenseIndex()
bm25 = BM25Index()
for ch in SAMPLE_CORPUS:
dense.add(ch)
bm25.add(ch)
for q in ("how is S3 multipart abort wired into retry budget",
"where is authorization centralized",
"how does rank fusion work"):
result = answer(q, dense, bm25)
print(f"Q: {result['query']}")
print(f" dense : {result['dense_top']}")
print(f" sparse : {result['sparse_top']}")
print(f" fused : {result['fused_top']}")
print(f" rerank : {result['rerank_top']}")
print()
if __name__ == "__main__":
main()