Get Started

Multi-Agent Debate and Collaboration

Du et al. (ICML 2024, "Society of Minds") run N model instances that independently propose answers, then iteratively critique each other over R rounds to converge. Improves factuality, rule-following, reasoning. Sparse topology beats full mesh on token cost.

60 minutesLearn + BuildPython (stdlib)Free lesson

Learning Objectives

  • Explain the debate protocol: N proposers, R rounds, converge on a shared answer.
  • Describe why debate improves factuality, rule-following, and reasoning.
  • Explain sparse topology: not every debater needs to see every other.
  • Implement a stdlib debate over a scripted LLM with full-mesh and sparse variants; measure token cost vs accuracy.

The Problem

Self-Refine (Lesson 05) is one model critiquing itself, risks groupthink. CRITIC (Lesson 05) grounds critique in external tools, not always available. Debate introduces a third mode: multiple instances, cross-critique, convergence by disagreement.

The Concept

Society of Minds (Du et al., ICML 2024)

  • N model instances independently propose answers to the same question.
  • Over R rounds, each model reads the others' proposals and critiques them.
  • Models update their answers based on the critiques.
  • After R rounds, return the convergent answer.

Original experiments used N=3, R=2 due to cost. Accuracy improves with more agents and more rounds on hard problems (MMLU, GSM8K, Chess Move Validity, biography generation).

Cross-model combinations beat single-model debates: ChatGPT + Bard together > either alone.

Sparse topology

"Improving Multi-Agent Debate with Sparse Communication Topology" (arXiv:2406.11776, 2024-2025) showed full-mesh debate is not always optimal. Sparse topologies (star, ring, hub-and-spoke) can match accuracy at lower token cost. Each debater sees only a subset of peers.

Implications:

  • Full mesh N=5, R=3 = 5 × 3 = 15 proposals, each reading 4 peers = 60 critique ops.
  • Star N=5, R=3 (one hub + 4 spokes) = 15 proposals, spokes read only the hub = 12 critique ops.

When debate helps

  • Factuality. N independent proposals, cross-check reduces hallucination.
  • Rule-following. Chess move validity, one model misses a rule, others catch it.
  • Open-ended reasoning. Multiple framings narrow in on the right answer.

When debate hurts

  • Latency-sensitive UX. N × R serial rounds is latency you may not have.
  • Cost-sensitive scale. N × R tokens per question.
  • Simple factual lookups. One lookup is cheaper than five debates.

2026 practical instantiations

  • Anthropic orchestrator-workers (Lesson 12), one variant of debate with a synthesis step.
  • LangGraph supervisor (Lesson 13), central router + specialist agents can implement debate as a node.
  • OpenAI Agents SDK (Lesson 16), agents handoff back and forth for iterative critique.
  • Multi-agent evalspair debate + evaluator-optimizer for eval signal.

Where this pattern goes wrong

  • Convergence collapse. All agents converge on the first wrong answer. Mitigate with required disagreement rounds.
  • Hub failure. In a star topology, a bad hub corrupts everyone. Rotate or use multiple hubs.
  • Prompt homogenization. All agents use the same prompt; they produce the same answers. Use diverse prompts and/or models.

Build It

code/main.py implements stdlib debate:

  • Debater class (scripted LLM with per-debater opinion drift).
  • FullMeshDebate and SparseDebate runners.
  • Three questions: one factual, one rule-based, one reasoning.
  • Metrics: convergent answer, rounds to convergence, total critique ops.

Run it:

Code
python3 code/main.py

Output: per-protocol accuracy and cost; sparse matches full mesh on 2/3 questions at lower cost.

Use It

  • Anthropic orchestrator-workers for simple 2-3-worker debates.
  • LangGraph for stateful multi-round debate with checkpointing.
  • Custom for research or specialized correctness guarantees.

Ship It

outputs/skill-debate.md scaffolds a multi-agent debate with configurable topology, N, R, and a convergence rule.

Exercises

  1. Implement a "forced disagreement" rule: in round 1, every debater must produce a distinct proposal. Measure effect on convergence speed.
  2. Add a confidence-weighted aggregation: debaters return (answer, confidence); aggregator weights by confidence. Does it help?
  3. Swap one "agent" for a different scripted LLM with different opinions. Does heterogeneity improve accuracy?
  4. Measure token cost for full mesh vs sparse on your 3 questions. Plot cost vs accuracy.
  5. Read the Society of Minds paper. Port your toy to N=5, R=3. What breaks? What gets better?

Key Terms

TermWhat people sayWhat it actually means
Debate"Multi-agent critique"N proposers, R rounds of cross-critique, converge
Full mesh"Everyone reads everyone"Every debater reads every peer each round
Sparse topology"Limited peer view"Debaters read only a subset of peers
Hub-and-spoke"Star topology"One central debater, N-1 spokes read only the hub
Convergence"Agreement"Debaters converge on a shared answer
Society of Minds"Du et al. debate paper"ICML 2024 multi-agent debate method

Further Reading

Check what you learned

7 questions. Answer each one to see why.

Question 1 of 70 correct

What does the Society of Minds protocol have N model instances do?

The full code

Every file for this lesson. Copy it, run it, then change it.

main.py
"""Multi-agent debate: full mesh vs sparse (star) topology.

Scripted debaters with different opinion drifts. Measures convergent answer,
rounds to consensus, and total critique ops (as a cost proxy).
"""

from __future__ import annotations

from collections import Counter
from dataclasses import dataclass, field
from typing import Any, Callable


@dataclass
class Debater:
    name: str
    drift: Callable[[str, list[str]], str]


def _make_debater(name: str, bias: str,
                  corrections: dict[str, str]) -> Debater:
    def drift(question: str, peer_answers: list[str]) -> str:
        current = corrections.get(question, bias)
        if peer_answers:
            common = Counter(peer_answers).most_common(1)[0][0]
            if common != current and common != bias:
                return common
        return current
    return Debater(name=name, drift=drift)


def full_mesh_round(debaters: list[Debater], question: str,
                    prior: dict[str, str]) -> tuple[dict[str, str], int]:
    new_answers: dict[str, str] = {}
    ops = 0
    for debater in debaters:
        peers = [prior[d.name] for d in debaters if d.name != debater.name]
        new_answers[debater.name] = debater.drift(question, peers)
        ops += len(peers)
    return new_answers, ops


def sparse_star_round(hub: Debater, spokes: list[Debater], question: str,
                      prior: dict[str, str]) -> tuple[dict[str, str], int]:
    new_answers: dict[str, str] = {}
    ops = 0
    spoke_names = [s.name for s in spokes]
    new_answers[hub.name] = hub.drift(
        question, [prior[n] for n in spoke_names]
    )
    ops += len(spoke_names)
    for spoke in spokes:
        new_answers[spoke.name] = spoke.drift(
            question, [prior[hub.name]]
        )
        ops += 1
    return new_answers, ops


def run_debate(debaters: list[Debater], question: str, rounds: int,
               topology: str) -> tuple[str, int, int]:
    prior: dict[str, str] = {}
    for debater in debaters:
        prior[debater.name] = debater.drift(question, [])

    total_ops = 0
    converged_round = -1
    hub = debaters[0]
    spokes = debaters[1:]
    for r in range(rounds):
        if topology == "full_mesh":
            new, ops = full_mesh_round(debaters, question, prior)
        else:
            new, ops = sparse_star_round(hub, spokes, question, prior)
        total_ops += ops
        if all(v == list(new.values())[0] for v in new.values()) and converged_round == -1:
            converged_round = r + 1
        prior = new

    votes = Counter(prior.values()).most_common(1)[0][0]
    return votes, converged_round, total_ops


def main() -> None:
    print("=" * 70)
    print("MULTI-AGENT DEBATE — Phase 14, Lesson 25")
    print("=" * 70)

    questions_and_truth = {
        "capital_of_portugal": "Lisbon",
        "is_2_plus_2_equal_4": "yes",
        "chess_legal_e4": "legal",
    }

    debaters = [
        _make_debater(
            "alpha", bias="Lisbon",
            corrections={"is_2_plus_2_equal_4": "yes",
                         "chess_legal_e4": "legal"},
        ),
        _make_debater(
            "beta", bias="Madrid",
            corrections={"capital_of_portugal": "Lisbon",
                         "is_2_plus_2_equal_4": "yes",
                         "chess_legal_e4": "legal"},
        ),
        _make_debater(
            "gamma", bias="Porto",
            corrections={"capital_of_portugal": "Lisbon",
                         "is_2_plus_2_equal_4": "yes",
                         "chess_legal_e4": "legal"},
        ),
    ]

    for q, truth in questions_and_truth.items():
        print(f"\n--- {q}  (truth: {truth}) ---")
        for topology in ("full_mesh", "sparse_star"):
            answer, converged, ops = run_debate(
                debaters, q, rounds=3, topology=topology,
            )
            correct = "CORRECT" if answer == truth else "WRONG"
            print(f"  {topology:12}  answer={answer:10}  "
                  f"converged_round={converged}  ops={ops}  {correct}")

    print()
    print("sparse star matches full mesh on accuracy with fewer critique ops.")
    print("debate helps factual and rule-based tasks; adds latency and cost.")


if __name__ == "__main__":
    main()