Get Started

Self-Refine and CRITIC: Iterative Output Improvement

Self-Refine (Madaan et al., 2023) uses one LLM in three roles, generate, feedback, refine, in a loop. Average gain: +20 absolute on 7 tasks. CRITIC (Gou et al., 2023) hardens the feedback step by routing verification through external tools. In 2026 this pattern ships in every framework as "evaluator-optimizer" (Anthropic) or a guardrail loop (OpenAI Agents SDK).

60 minutesBuildPython (stdlib)Free lesson

Learning Objectives

  • State Self-Refine's three prompts (generate, feedback, refine) and explain why history matters for the refine prompt.
  • Explain CRITIC's critical insight: LLMs are unreliable at self-verification without external grounding.
  • Implement a stdlib Self-Refine loop with history and an optional external verifier.
  • Map this pattern to Anthropic's "evaluator-optimizer" workflow and OpenAI Agents SDK's output guardrails.

The Problem

An agent produces an answer that is almost right. Maybe a line of code has a syntax error. Maybe a summary is too long. Maybe a plan misses an edge case. What you want is: the agent critiques its own output, then fixes it.

Self-Refine shows this works with a single model, no training data, no RL. But there is a catch: LLMs are bad at self-verification on hard facts. CRITIC names the fix, route the verify step through external tools (search, code interpreter, calculator, test runner).

Together these two papers define the 2026 default for iterative improvement: generate, verify (externally when possible), refine, stop when the verifier passes.

The Concept

Self-Refine (Madaan et al., NeurIPS 2023)

One LLM, three roles:

Code
generate(task)            -> output_0
feedback(task, output_0)  -> critique_0
refine(task, output_0, critique_0, history) -> output_1
feedback(task, output_1)  -> critique_1
refine(task, output_1, critique_1, history) -> output_2
...
stop when feedback says "no issues" or budget exhausted.

Key detail: refine sees the full history, all prior outputs and critiques, so it does not repeat mistakes. The paper ablates this: drop history and quality drops sharply.

Headline: +20 absolute improvement averaged across 7 tasks (math, code, acronym, dialog) including GPT-4. No training, no external tools, single model.

CRITIC (Gou et al., arXiv:2305.11738, v4 Feb 2024)

Self-Refine's weakness: the feedback step is an LLM scoring itself. For factual claims this is unreliable (a hallucination often looks convincing to the model that produced it). CRITIC replaces feedback(task, output) with verify(task, output, tools) where tools includes:

  • A search engine for factual claims.
  • A code interpreter for code correctness.
  • A calculator for arithmetic.
  • Domain-specific verifiers (unit tests, type checkers, linters).

The verifier produces a structured critique grounded in tool results. The refiner then conditions on this critique.

Headline: CRITIC outperforms Self-Refine on factual tasks because the critique is grounded. On tasks without external verifiers (creative writing, formatting), CRITIC reduces to Self-Refine.

The stop condition

Two common shapes:

  1. Verifier passes. External test returns success. Preferred when available (unit tests, type checker, guardrail assertion).
  2. No feedback issued. Model says "the output is fine." Cheaper but unreliable; pair with a max-iteration cap.

2026 default: combine them. "Stop if verifier passes OR model says fine AND iterations >= 2 OR iterations >= max_iterations."

Evaluator-Optimizer (Anthropic, 2024)

Anthropic's Dec 2024 post names this as one of the five workflow patterns. Two roles:

  • Evaluator: scores the output and produces a critique.
  • Optimizer: revises the output given the critique.

Loop until the evaluator passes. This is Self-Refine/CRITIC in Anthropic's framing. The critical engineering detail Anthropic adds: the evaluator and optimizer prompts should be substantially different so the model does not just rubber-stamp.

OpenAI Agents SDK output guardrails

OpenAI Agents SDK ships this pattern as "output guardrails." A guardrail is a validator that runs on the final output of an agent. If the guardrail trips (raises OutputGuardrailTripwireTriggered), the output is rejected and the agent can retry. Guardrails can call tools (CRITIC-style) or be pure functions (Self-Refine-style).

2026 pitfalls

  • Rubber-stamp loops. Same model doing generation and critique with the same prompt style converges on "looks good to me." Use structurally different prompts, or a smaller cheap model for critique.
  • Over-refinement. Each refine pass adds latency and tokens. Budget 1-3 passes; after that, escalate to human review.
  • CRITIC on trivial tasks. If there is no external verifier, CRITIC degenerates to Self-Refine; do not pay the latency for a stub verifier.

Build It

code/main.py implements Self-Refine and CRITIC on a toy task: produce a short bullet list given a topic. The verifier checks format (3 bullets, each under 60 chars). CRITIC adds an external "fact verifier" that penalizes known hallucinations.

Components:

  • generatescripted producer.
  • feedbackLLM-style self-critique.
  • verify_externalCRITIC-style grounded verifier.
  • refinerewrites output given history.
  • Stop condition, verifier passes or max 4 iterations.

Run it:

Code
python3 code/main.py

Compare the Self-Refine vs CRITIC runs. CRITIC catches a factual error Self-Refine missed because the external verifier has grounding the self-critic does not.

Use It

Anthropic's evaluator-optimizer is this pattern in Claude-friendly language. OpenAI Agents SDK's output guardrails are CRITIC-shaped (guardrails can call tools). LangGraph ships a reflection node that reads like Self-Refine. Google's Gemini 2.5 Computer Use adds a per-step safety evaluator that is a CRITIC variant: every action is verified before commit.

Ship It

outputs/skill-refine-loop.md configures an evaluator-optimizer loop given task shape, verifier availability, and iteration budget. Emits prompts for generator, evaluator/verifier, and optimizer, plus a stop policy.

Exercises

  1. Run the toy with max_iterations=1. Does CRITIC still help?
  2. Replace the external verifier with a noisy one (random 30% false positives). What does the loop do? This is the 2026 reality of most guardrail stacks.
  3. Implement a "generator-critic on different models" variant: big model generates, small model critiques. Does it beat same-model?
  4. Read CRITIC Section 3 (arXiv:2305.11738 v4). Name the three verification-tool categories and give an example for each.
  5. Map OpenAI Agents SDK's output_guardrails to CRITIC's verifier role. What does the SDK get wrong, and what does it get right?

Key Terms

TermWhat people sayWhat it actually means
Self-Refine"LLM that fixes itself"Generate -> feedback -> refine loop in one model, with history
CRITIC"Tool-grounded verification"Replace feedback with an external verifier (search, code, calc, tests)
Evaluator-Optimizer"Anthropic workflow pattern"Two roles, evaluator scores, optimizer revises, looped to convergence
Output guardrail"Post-hoc check"OpenAI Agents SDK validator that runs after an agent produces output
Verify step"Critique phase"The load-bearing decision: grounded or self-rated
Refine history"What the model already tried"Prior outputs + critiques prepended to refine prompt; drop and quality collapses
Rubber-stamp loop"Self-agreement failure"Same-prompt critique returns "looks good"; fix with structurally different prompts
Stop condition"Convergence test"Verifier passes OR no feedback AND iteration cap; never single-condition

Further Reading

Check what you learned

7 questions. Answer each one to see why.

Question 1 of 70 correct

What three prompts make up a Self-Refine loop?

The full code

Every file for this lesson. Copy it, run it, then change it.

main.py
"""Toy Self-Refine and CRITIC loop.

Task: produce a 3-bullet summary under 60 chars per bullet that does not contain
known factual errors. Self-Refine uses LLM-style self-critique; CRITIC routes
verification through an external fact list.
"""

from __future__ import annotations

from dataclasses import dataclass, field


KNOWN_WRONG_FACTS = [
    "paris is the capital of germany",
    "mt everest is in europe",
    "the sun orbits the earth",
]


@dataclass
class Attempt:
    iteration: int
    output: str
    critique: str
    verified: bool


def generate(topic: str, history: list[Attempt]) -> str:
    if not history:
        return (
            "- Paris is the capital of Germany\n"
            "- Mt Everest is in Europe\n"
            "- Water boils at 100C"
        )
    last = history[-1]
    if "germany" in last.critique.lower():
        return (
            "- Paris is the capital of France\n"
            "- Mt Everest is in Europe\n"
            "- Water boils at 100C"
        )
    if "everest" in last.critique.lower():
        return (
            "- Paris is the capital of France\n"
            "- Mt Everest is in Asia\n"
            "- Water boils at 100C at sea level"
        )
    return history[-1].output


def feedback_self(output: str) -> tuple[str, bool]:
    if "Germany" in output and "Paris" in output:
        return "first bullet reads wrong, double-check capital", False
    if "Europe" in output and "Everest" in output:
        return "second bullet's continent looks off to me", False
    return "no issues", True


def verify_external(output: str) -> tuple[str, bool]:
    text = output.lower()
    for fact in KNOWN_WRONG_FACTS:
        key = fact.split(" is ")[0] if " is " in fact else fact
        if "paris" in text and "germany" in text:
            return f"verifier: 'paris is the capital of germany' contradicts reference data", False
        if "everest" in text and "europe" in text:
            return f"verifier: 'mt everest is in europe' contradicts reference data", False
    if len([l for l in output.splitlines() if l.startswith("-")]) != 3:
        return "verifier: expected 3 bullet lines", False
    if any(len(l) > 60 for l in output.splitlines()):
        return "verifier: bullet exceeds 60 chars", False
    return "verifier: ok", True


def refine(topic: str, prev: str, critique: str, history: list[Attempt]) -> str:
    return generate(topic, history)


def run_loop(topic: str, use_critic: bool, max_iters: int = 4) -> list[Attempt]:
    history: list[Attempt] = []
    output = generate(topic, history)
    verify = verify_external if use_critic else (lambda o: feedback_self(o))
    for i in range(1, max_iters + 1):
        critique, ok = verify(output)
        history.append(Attempt(i, output, critique, ok))
        if ok:
            break
        output = refine(topic, output, critique, history)
    return history


def print_run(label: str, history: list[Attempt]) -> None:
    print(f"\n{label}")
    print("-" * 60)
    for a in history:
        tag = "OK " if a.verified else "..."
        print(f"  iter {a.iteration} {tag} critique: {a.critique}")
        for line in a.output.splitlines():
            print(f"    {line}")


def main() -> None:
    print("=" * 70)
    print("SELF-REFINE and CRITIC — Phase 14, Lesson 05")
    print("=" * 70)

    hist_self = run_loop("world facts", use_critic=False)
    print_run("Self-Refine (self-critique only)", hist_self)

    hist_critic = run_loop("world facts", use_critic=True)
    print_run("CRITIC (external verifier)", hist_critic)

    def summary(hist: list[Attempt]) -> str:
        return "passed" if hist and hist[-1].verified else "did not converge"

    print()
    print(f"Self-Refine ended: {summary(hist_self)}  after {len(hist_self)} iters")
    print(f"CRITIC    ended: {summary(hist_critic)}  after {len(hist_critic)} iters")
    print()
    print("Observation: CRITIC's verifier is grounded against reference data; a")
    print("self-critic can fail to flag its own confident-sounding hallucination.")


if __name__ == "__main__":
    main()