Multi-Agent Debate and Collaboration
Du et al. (ICML 2024, "Society of Minds") run N model instances that independently propose answers, then iteratively critique each other over R rounds to converge. Improves factuality, rule-following, reasoning. Sparse topology beats full mesh on token cost.
Learning Objectives
- Explain the debate protocol: N proposers, R rounds, converge on a shared answer.
- Describe why debate improves factuality, rule-following, and reasoning.
- Explain sparse topology: not every debater needs to see every other.
- Implement a stdlib debate over a scripted LLM with full-mesh and sparse variants; measure token cost vs accuracy.
The Problem
Self-Refine (Lesson 05) is one model critiquing itself, risks groupthink. CRITIC (Lesson 05) grounds critique in external tools, not always available. Debate introduces a third mode: multiple instances, cross-critique, convergence by disagreement.
The Concept
Society of Minds (Du et al., ICML 2024)
- N model instances independently propose answers to the same question.
- Over R rounds, each model reads the others' proposals and critiques them.
- Models update their answers based on the critiques.
- After R rounds, return the convergent answer.
Original experiments used N=3, R=2 due to cost. Accuracy improves with more agents and more rounds on hard problems (MMLU, GSM8K, Chess Move Validity, biography generation).
Cross-model combinations beat single-model debates: ChatGPT + Bard together > either alone.
Sparse topology
"Improving Multi-Agent Debate with Sparse Communication Topology" (arXiv:2406.11776, 2024-2025) showed full-mesh debate is not always optimal. Sparse topologies (star, ring, hub-and-spoke) can match accuracy at lower token cost. Each debater sees only a subset of peers.
Implications:
- Full mesh N=5, R=3 = 5 × 3 = 15 proposals, each reading 4 peers = 60 critique ops.
- Star N=5, R=3 (one hub + 4 spokes) = 15 proposals, spokes read only the hub = 12 critique ops.
When debate helps
- Factuality. N independent proposals, cross-check reduces hallucination.
- Rule-following. Chess move validity, one model misses a rule, others catch it.
- Open-ended reasoning. Multiple framings narrow in on the right answer.
When debate hurts
- Latency-sensitive UX. N × R serial rounds is latency you may not have.
- Cost-sensitive scale. N × R tokens per question.
- Simple factual lookups. One lookup is cheaper than five debates.
2026 practical instantiations
- Anthropic orchestrator-workers (Lesson 12), one variant of debate with a synthesis step.
- LangGraph supervisor (Lesson 13), central router + specialist agents can implement debate as a node.
- OpenAI Agents SDK (Lesson 16), agents handoff back and forth for iterative critique.
- Multi-agent evalspair debate + evaluator-optimizer for eval signal.
Where this pattern goes wrong
- Convergence collapse. All agents converge on the first wrong answer. Mitigate with required disagreement rounds.
- Hub failure. In a star topology, a bad hub corrupts everyone. Rotate or use multiple hubs.
- Prompt homogenization. All agents use the same prompt; they produce the same answers. Use diverse prompts and/or models.
Build It
code/main.py implements stdlib debate:
Debaterclass (scripted LLM with per-debater opinion drift).FullMeshDebateandSparseDebaterunners.- Three questions: one factual, one rule-based, one reasoning.
- Metrics: convergent answer, rounds to convergence, total critique ops.
Run it:
python3 code/main.pyOutput: per-protocol accuracy and cost; sparse matches full mesh on 2/3 questions at lower cost.
Use It
- Anthropic orchestrator-workers for simple 2-3-worker debates.
- LangGraph for stateful multi-round debate with checkpointing.
- Custom for research or specialized correctness guarantees.
Ship It
outputs/skill-debate.md scaffolds a multi-agent debate with configurable topology, N, R, and a convergence rule.
Exercises
- Implement a "forced disagreement" rule: in round 1, every debater must produce a distinct proposal. Measure effect on convergence speed.
- Add a confidence-weighted aggregation: debaters return (answer, confidence); aggregator weights by confidence. Does it help?
- Swap one "agent" for a different scripted LLM with different opinions. Does heterogeneity improve accuracy?
- Measure token cost for full mesh vs sparse on your 3 questions. Plot cost vs accuracy.
- Read the Society of Minds paper. Port your toy to N=5, R=3. What breaks? What gets better?
Key Terms
| Term | What people say | What it actually means |
|---|---|---|
| Debate | "Multi-agent critique" | N proposers, R rounds of cross-critique, converge |
| Full mesh | "Everyone reads everyone" | Every debater reads every peer each round |
| Sparse topology | "Limited peer view" | Debaters read only a subset of peers |
| Hub-and-spoke | "Star topology" | One central debater, N-1 spokes read only the hub |
| Convergence | "Agreement" | Debaters converge on a shared answer |
| Society of Minds | "Du et al. debate paper" | ICML 2024 multi-agent debate method |
Further Reading
- Du et al., Society of Minds (arXiv:2305.14325)canonical multi-agent debate
- Sparse Communication Topology (arXiv:2406.11776)sparse topology results
- Anthropic, Building Effective Agentsorchestrator-workers as a debate variant
- Madaan et al., Self-Refine (arXiv:2303.17651)single-model self-critique counterpart
Check what you learned
7 questions. Answer each one to see why.
What does the Society of Minds protocol have N model instances do?
The full code
Every file for this lesson. Copy it, run it, then change it.
"""Multi-agent debate: full mesh vs sparse (star) topology.
Scripted debaters with different opinion drifts. Measures convergent answer,
rounds to consensus, and total critique ops (as a cost proxy).
"""
from __future__ import annotations
from collections import Counter
from dataclasses import dataclass, field
from typing import Any, Callable
@dataclass
class Debater:
name: str
drift: Callable[[str, list[str]], str]
def _make_debater(name: str, bias: str,
corrections: dict[str, str]) -> Debater:
def drift(question: str, peer_answers: list[str]) -> str:
current = corrections.get(question, bias)
if peer_answers:
common = Counter(peer_answers).most_common(1)[0][0]
if common != current and common != bias:
return common
return current
return Debater(name=name, drift=drift)
def full_mesh_round(debaters: list[Debater], question: str,
prior: dict[str, str]) -> tuple[dict[str, str], int]:
new_answers: dict[str, str] = {}
ops = 0
for debater in debaters:
peers = [prior[d.name] for d in debaters if d.name != debater.name]
new_answers[debater.name] = debater.drift(question, peers)
ops += len(peers)
return new_answers, ops
def sparse_star_round(hub: Debater, spokes: list[Debater], question: str,
prior: dict[str, str]) -> tuple[dict[str, str], int]:
new_answers: dict[str, str] = {}
ops = 0
spoke_names = [s.name for s in spokes]
new_answers[hub.name] = hub.drift(
question, [prior[n] for n in spoke_names]
)
ops += len(spoke_names)
for spoke in spokes:
new_answers[spoke.name] = spoke.drift(
question, [prior[hub.name]]
)
ops += 1
return new_answers, ops
def run_debate(debaters: list[Debater], question: str, rounds: int,
topology: str) -> tuple[str, int, int]:
prior: dict[str, str] = {}
for debater in debaters:
prior[debater.name] = debater.drift(question, [])
total_ops = 0
converged_round = -1
hub = debaters[0]
spokes = debaters[1:]
for r in range(rounds):
if topology == "full_mesh":
new, ops = full_mesh_round(debaters, question, prior)
else:
new, ops = sparse_star_round(hub, spokes, question, prior)
total_ops += ops
if all(v == list(new.values())[0] for v in new.values()) and converged_round == -1:
converged_round = r + 1
prior = new
votes = Counter(prior.values()).most_common(1)[0][0]
return votes, converged_round, total_ops
def main() -> None:
print("=" * 70)
print("MULTI-AGENT DEBATE — Phase 14, Lesson 25")
print("=" * 70)
questions_and_truth = {
"capital_of_portugal": "Lisbon",
"is_2_plus_2_equal_4": "yes",
"chess_legal_e4": "legal",
}
debaters = [
_make_debater(
"alpha", bias="Lisbon",
corrections={"is_2_plus_2_equal_4": "yes",
"chess_legal_e4": "legal"},
),
_make_debater(
"beta", bias="Madrid",
corrections={"capital_of_portugal": "Lisbon",
"is_2_plus_2_equal_4": "yes",
"chess_legal_e4": "legal"},
),
_make_debater(
"gamma", bias="Porto",
corrections={"capital_of_portugal": "Lisbon",
"is_2_plus_2_equal_4": "yes",
"chess_legal_e4": "legal"},
),
]
for q, truth in questions_and_truth.items():
print(f"\n--- {q} (truth: {truth}) ---")
for topology in ("full_mesh", "sparse_star"):
answer, converged, ops = run_debate(
debaters, q, rounds=3, topology=topology,
)
correct = "CORRECT" if answer == truth else "WRONG"
print(f" {topology:12} answer={answer:10} "
f"converged_round={converged} ops={ops} {correct}")
print()
print("sparse star matches full mesh on accuracy with fewer critique ops.")
print("debate helps factual and rule-based tasks; adds latency and cost.")
if __name__ == "__main__":
main()