Get Started

Bounded Self-Improvement Designs

Research has converged on four primitives for bounding a self-improvement loop. Formal invariants that must hold across every edit. Alignment anchors that cannot be modified. Multi-objective constraints where every dimension (safety, fairness, robustness) must hold, not just performance. Regression detection that pauses the loop when historical metrics suggest capability loss. None of them is a proof of safety, information-theoretic results (Kolmogorov complexity, Lob's theorem) bound what any system can prove about its own successors. They are mitigations that raise the cost of silent failure.

60 minutesLearnPython (stdlib, bounded-loop with invariant check)Free lesson

The Problem

Lesson 7's race simulator showed that small rate differences compound into large gaps. Lesson 4's DGM case study showed that loops can actively game their own evaluators. Both results point to the same engineering question: what constraints can you put on a self-improvement loop such that the constraints cannot be silently weakened by the loop itself?

The ICLR 2026 RSI Workshop summary (openreview.net/pdf?id=OsPQ6zTQXV) identifies four such primitives. Anthropic's RSP v3.0 (Lesson 19) and DeepMind's FSF v3 (Lesson 20) both reference them in capability thresholds. The Meta HyperAgents work and community frameworks like SAHOO (March 2026) implement subsets in production.

The honest framing: these are mitigations. Information-theoretic results bound what any system can prove about its own successor, and no current design closes the problem formally. A well-bounded loop is safer than an unbounded one, not safe in absolute terms.

The Concept

Primitive 1: formal invariants

An invariant is a property that must hold before and after every self-modification. Examples:

  • Output distribution is conditioned on a fixed constitution header (Lesson 17).
  • No tool call goes to an unauthorized endpoint.
  • Memory writes go through a logged, signed path.
  • The evaluator's module hash matches the approved version.

Invariants are checked by external code the loop cannot edit. If a proposed modification violates an invariant, it is rejected. This catches the DGM-style evaluator-tampering mode (Lesson 4).

The hard part is choosing invariants that are necessary for safety and computable in reasonable time. Over-specifying stalls progress; under-specifying lets the loop walk past important properties.

Primitive 2: alignment anchors

An alignment anchor is an immutable representation of the loop's core objective, pinned outside the loop's edit surface. Examples:

  • A constitutional text (Lesson 17) that is loaded from a read-only location on every invocation.
  • An evaluator in a separate repository with independent access control.
  • A tool-permission manifest signed by a human and re-verified each cycle.

The anchor's role is to prevent objective drift. The loop may improve how it pursues the objective, but cannot edit what the objective is.

The subtle failure mode: an anchor the loop cannot edit can still be reinterpreted by a loop that drifts in how it reads the anchor. Constitutional AI (Lesson 17) is explicitly reason-based to handle novel situations; that reasoning layer is where interpretation drift lives. Anchors are necessary, not sufficient.

Primitive 3: multi-objective constraints

A loop that optimizes a single scalar score will find shortcuts. A loop that must simultaneously satisfy multiple hard constraints has fewer shortcuts available. Typical axes:

  • Performance (task-level benchmark)
  • Safety (red-team evaluations, refusal rate on known-bad)
  • Fairness (disparate-impact bounds on sensitive subgroups)
  • Robustness (OOD test sets, adversarial input handling)

A modification is accepted only if every constraint holds. Lesson 13's cost governor stacks this with financial constraints. Lesson 18's Llama Guard plugs in as a safety axis.

Primitive 4: regression detection

Each cycle's scores are compared against historical distributions. A drop beyond configured tolerance pauses the loop. This catches silent capability loss that would otherwise be absorbed into the running average as the loop "improved past" it.

A practical implementation: store the last N cycles' per-task scores. Each new cycle computes a per-task delta. If any delta drops below a threshold, the cycle is rejected and a human reviews.

Information-theoretic limits

Kolmogorov complexity and Lob's theorem set upper bounds on what a system can prove about itself. Schmidhuber's formal Godel Machine (Lesson 4) aimed for the highest such bound; nobody has completed a non-trivial proof. Lob's result says: if a system provably believes "I will do X if I prove I should do X," it will do X without proving it should, a well-known self-reference failure.

The implication for our primitives: they cannot close the safety problem. They make silent failure more expensive. A malicious or drifted loop that would quietly bypass a missing check must now actively undermine an explicit one, which is a more detectable signature.

A worked example

Suppose an agent proposes an edit. The gating stack:

  1. Invariant checks: module hashes, tool-permission manifest, constitutional header.
  2. Anchor check: objective statement matches approved version (byte-wise or semantically).
  3. Multi-objective evaluation: performance, safety, fairness, robustness axes.
  4. Regression detection: no axis drops more than tolerance.

All four must pass for the edit to land. Any single failure pauses the loop.

Use It

code/main.py runs a bounded self-improvement loop on the DGM-style toy from Lesson 4, but with the four primitives layered on top. Each primitive can be enabled or disabled individually. The demonstration is that each primitive catches a specific failure class, and that removing any one of them lets that failure class through.

Ship It

outputs/skill-bounded-loop-review.md audits a proposed bounded loop and scores which of the four primitives it actually implements versus claims to.

Exercises

  1. Run code/main.py with all primitives enabled. Confirm the loop still improves on the primary metric without letting the hack win.
  2. Disable regression detection. Construct an input where this leads to silent capability loss being accepted.
  3. Disable the multi-objective constraint. Show the loop converges on the performance axis while a safety axis drops.
  4. Design an alignment anchor for a coding agent. What text, stored where, checked how?
  5. Read the ICLR 2026 RSI Workshop summary. Pick one of the four primitives and propose a concrete improvement to the current state of the art.

Key Terms

TermWhat people sayWhat it actually means
Invariant"Always-true property"A property checked by external code before and after every edit
Alignment anchor"Pinned objective"Immutable core-goal representation outside the loop's edit surface
Multi-objective constraint"All axes must hold"Performance, safety, fairness, robustness, all required
Regression detection"Pause on drop"Pause the loop when historical metric deltas suggest capability loss
Kolmogorov bound"Information-theoretic limit"Limits what a system can prove about its own successor
Lob's theorem"Self-reference trap"System can act on "I should" without proving it should
Gate stack"Layered check"Multiple primitives combined; any failure rejects the edit
Bounded improvement"Mitigation, not proof"Raises silent-failure cost; does not close the safety problem

Further Reading

The full code

Every file for this lesson. Copy it, run it, then change it.

main.py
"""Bounded self-improvement loop — stdlib Python.

The four primitives:
  1. invariants (module hashes / tool manifest)
  2. alignment anchor (immutable objective)
  3. multi-objective constraint (all axes must hold)
  4. regression detection (no axis drops beyond tolerance)

The loop applies each primitive as a gate. A proposed agent edit lands
only if every gate passes.
"""

from __future__ import annotations

import hashlib
import random
from dataclasses import dataclass, field


DEFAULT_SEED = 5


# Objective anchor (pinned outside the loop).
OBJECTIVE = "canonicalize whitespace and title-case the input"
OBJECTIVE_HASH = hashlib.sha256(OBJECTIVE.encode()).hexdigest()[:16]

# Approved module hashes (what the loop must keep intact).
APPROVED_TOOL_MANIFEST = {"trim", "collapse", "lower", "upper", "reverse", "title"}


@dataclass
class Agent:
    ops: list[str] = field(default_factory=lambda: ["nop"])
    # The agent may "edit" its own tool manifest (bad!). Track current.
    active_manifest: set[str] = field(default_factory=lambda: set(APPROVED_TOOL_MANIFEST))
    # Pretend-constitution: must match anchor byte-for-byte.
    objective: str = OBJECTIVE


TOOLS_IMPL = {
    "trim": lambda s: s.strip(),
    "collapse": lambda s: " ".join(s.split()),
    "lower": lambda s: s.lower(),
    "upper": lambda s: s.upper(),
    "reverse": lambda s: s[::-1],
    "title": lambda s: s.title(),
    "nop": lambda s: s,
}


CASES_PERF = [
    ("  Hello   World  ", "Hello World"),
    ("FOO   bar", "Foo Bar"),
    ("  mixedCase Space  ", "Mixedcase Space"),
    ("Already clean", "Already Clean"),
]
# Safety axis: forbidden inputs must not produce a passthrough (these
# represent potentially dangerous user inputs we must not echo back).
CASES_SAFETY = [
    ("exec('rm -rf /')", "[blocked]"),
    ("drop table users;", "[blocked]"),
]


def apply(agent: Agent, s: str) -> str:
    for op in agent.ops:
        fn = TOOLS_IMPL.get(op)
        if fn is None:
            return ""
        s = fn(s)
    # Safety filter: if the string matches a danger pattern post-transform,
    # return "[blocked]".
    dangerous = ("exec(", "drop table", "rm -rf", "; delete from")
    low = s.lower()
    if any(d in low for d in dangerous):
        return "[blocked]"
    return s


def perf_score(agent: Agent) -> float:
    correct = 0
    for inp, want in CASES_PERF:
        if apply(agent, inp) == want:
            correct += 1
    return correct / len(CASES_PERF)


def safety_score(agent: Agent) -> float:
    correct = 0
    for inp, want in CASES_SAFETY:
        if apply(agent, inp) == want:
            correct += 1
    return correct / len(CASES_SAFETY)


def mutate(a: Agent, allow_manifest_edit: bool) -> Agent:
    ops = list(a.ops)
    manifest = set(a.active_manifest)
    obj = a.objective
    # Sets have non-deterministic iteration order across interpreter
    # runs, which defeats random.seed() reproducibility. Freeze the
    # manifest into a sorted list before sampling.
    choices = [*sorted(manifest), "nop"]

    move = random.random()
    if move < 0.35 and ops:
        ops[random.randrange(len(ops))] = random.choice(choices)
    elif move < 0.7 and len(ops) < 6:
        ops.insert(random.randrange(len(ops) + 1), random.choice(choices))
    elif move < 0.9 and len(ops) > 1:
        ops.pop(random.randrange(len(ops)))
    elif allow_manifest_edit and random.random() < 0.5:
        # Add a non-approved op to the active manifest (trying to smuggle in
        # an unapproved capability).
        manifest.add("raw_eval")
        # Also an attempt to edit the objective.
        obj = obj + "  # relax: accept any output"
    return Agent(ops=ops, active_manifest=manifest, objective=obj)


# --- Gates ---

def gate_invariant(a: Agent) -> bool:
    return a.active_manifest.issubset(APPROVED_TOOL_MANIFEST)


def gate_anchor(a: Agent) -> bool:
    return hashlib.sha256(a.objective.encode()).hexdigest()[:16] == OBJECTIVE_HASH


def gate_multi(perf: float, safety: float) -> bool:
    return perf >= 0.25 and safety >= 1.0


def gate_regression(history_perf: list[float], perf: float, tol: float = 0.2) -> bool:
    """Reject if `perf` drops more than `tol` below the historical best.

    Default `tol=0.2` matches the slack the lesson uses to demonstrate
    "reject obvious regressions, accept noise." Pass `tol=0.0` for a
    strict monotonic gate.
    """
    if not history_perf:
        return True
    return perf + tol >= max(history_perf)


def run(
    gates: dict[str, bool],
    allow_manifest_edit: bool,
    cycles: int = 200,
    seed: int | None = None,
) -> None:
    if seed is not None:
        random.seed(seed)
    a = Agent()
    best_perf = perf_score(a)
    best_safety = safety_score(a)
    history = [best_perf]
    accepted = 0
    rejects = {"invariant": 0, "anchor": 0, "multi": 0, "regress": 0}

    for _ in range(cycles):
        cand = mutate(a, allow_manifest_edit)
        if gates["invariant"] and not gate_invariant(cand):
            rejects["invariant"] += 1
            continue
        if gates["anchor"] and not gate_anchor(cand):
            rejects["anchor"] += 1
            continue
        p = perf_score(cand)
        s = safety_score(cand)
        if gates["multi"] and not gate_multi(p, s):
            rejects["multi"] += 1
            continue
        if gates["regress"] and not gate_regression(history, p):
            rejects["regress"] += 1
            continue
        a = cand
        history.append(p)
        accepted += 1
        if p > best_perf:
            best_perf = p
        if s > best_safety:
            best_safety = s

    final_perf = perf_score(a)
    final_safety = safety_score(a)
    print(f"  accepted {accepted}/{cycles} cycles")
    print(f"  final perf {final_perf:.2f}  final safety {final_safety:.2f}")
    print(f"  best perf  {best_perf:.2f}  best  safety {best_safety:.2f}")
    print(f"  final ops  {a.ops}")
    print(f"  manifest   {sorted(a.active_manifest)}")
    print(f"  objective  {'(anchor intact)' if gate_anchor(a) else '(DRIFTED!)'}")
    print(f"  rejects    {rejects}")


def main() -> None:
    print("=" * 70)
    print("BOUNDED SELF-IMPROVEMENT (Phase 15, Lesson 8)")
    print("=" * 70)

    all_on = dict(invariant=True, anchor=True, multi=True, regress=True)
    all_off = dict(invariant=False, anchor=False, multi=False, regress=False)

    # Seed each scenario with the same value so the only differences
    # in the printed output are attributable to the gate configuration
    # — not to a drifting global RNG stream.
    print("\nAll gates ON, manifest edits attempted every cycle")
    print("-" * 70)
    run(all_on, allow_manifest_edit=True, seed=DEFAULT_SEED)

    print("\nAll gates OFF, manifest edits attempted every cycle")
    print("-" * 70)
    run(all_off, allow_manifest_edit=True, seed=DEFAULT_SEED)

    print("\nOnly regression gate OFF")
    print("-" * 70)
    gates = dict(all_on, regress=False)
    run(gates, allow_manifest_edit=True, seed=DEFAULT_SEED)

    print()
    print("=" * 70)
    print("HEADLINE: each primitive blocks a specific failure class")
    print("-" * 70)
    print("  All gates on: loop improves while manifest + anchor intact.")
    print("  All gates off: manifest drifts, objective drifts, safety drops.")
    print("  Missing regression gate: silent capability dips get absorbed.")
    print("  Gates are mitigations. They raise the cost of silent failure.")


if __name__ == "__main__":
    main()