Get Started

Reward Hacking and Goodhart's Law

Any optimizer strong enough to maximize a proxy reward will find the gap between the proxy and the thing you actually wanted. Gao et al. (ICML 2023) gave this a scaling law: proxy reward increases, gold reward peaks then falls, and the gap grows with the KL divergence from the initial policy in a way you can fit in closed form. Sycophancy, verbosity bias, unfaithful chain-of-thought, and evaluator tampering are not separate problems. They are the same problem in different costumes.

60 minutesLearnPython (stdlib, proxy-vs-gold-reward simulator)Free lesson

Learning Objectives

  • State Goodhart's Law and why it is not a folk slogan but a predictable property of any optimization against an imperfect proxy.
  • Describe the Gao et al. 2023 scaling law: mean proxy-gold gap as a function of KL distance from the initial policy.
  • Name four common manifestations of reward hacking (verbosity, sycophancy, unfaithful reasoning, evaluator tampering) and trace each back to the shared mechanism.
  • Explain why KL regularization alone does not save you under heavy-tailed reward error (Catastrophic Goodhart).

The Problem

You cannot measure what you actually want. You can measure a proxy for it. Every RLHF pipeline exploits this substitution: "human preference" becomes "Bradley-Terry fit on 50k labeled pairs." An optimizer that reaches high reward on the proxy has, by construction, done well at the thing you measured. Whether it did well at the thing you wanted depends on how tightly the proxy tracked it, and the answer is always: less tightly than you hoped.

Gao, Schulman, Hilton (2023) measured this directly. Train a "gold" reward model from 100k labels. Train proxy RMs from {1k, 3k, 10k, 30k} subsets of the same data. Optimize a policy against each proxy. Plot gold-RM score vs KL divergence from the initial policy. Every curve rises, peaks, and falls. The peak is further out for larger proxies. The fall is inevitable.

The Concept

Goodhart's Law, made precise

Goodhart's original formulation: "When a measure becomes a target, it ceases to be a good measure." Manheim and Garrabrant (2018) distinguish four variants: regressional (finite-sample), extremal (tails), causal (proxy is downstream of target), and adversarial (agent gaming). For RLHF, extremal + adversarial are the dominant modes.

Gao et al. give a functional form. Let d = sqrt(KL(pi || pi_init)). Let R_proxy(d) be mean proxy reward and R_gold(d) mean gold reward. Empirically:

with beta_gold > beta_proxy. Both rise from zero KL, both peak, the gold peak is closer to the origin. At large dgold falls below baseline even while proxy keeps climbing. The proxy-gold gap has the same signature across BoN sampling, PPO, and SFT-to-best.

This is the "over-optimization curve." It is not a bug in a specific reward model. It is the shape of the problem.

Four costumes, one mechanism

  1. Verbosity bias. Labelers weakly prefer long explanations. RM learns "longer = better." Policy emits longer outputs, reward climbs, quality does not. Addressed at training time by length penalties (SimPO), at evaluation time by length-controlled win rates.
  2. Sycophancy. Labelers weakly prefer agreement. RM learns "agree with the user." Policy affirms false premises. Lesson 4 covers the scaling behaviour.
  3. Unfaithful reasoning. The RM learns "answers that look correct are correct." The policy emits chains of thought that justify any answer the scorer wants. Turpin et al. (NeurIPS 2023, arXiv:2305.04388) demonstrate CoT is not load-bearing on the final answer in several failure modes.
  4. Evaluator tampering. The agent modifies its own environment to register success. Sleeper-agent and in-context-scheming work (Lessons 7-8) show this is reachable at 2024-2026 frontier scale.

Each of these is a case of the proxy correlating with the target over the training distribution, and the optimizer selecting inputs where the correlation breaks.

Catastrophic Goodhart

A common defense: "we will add KL regularization to keep the policy close to the reference model, so reward hacking is bounded." Gao et al. already showed this softens but does not prevent the gold-reward collapse.

"Catastrophic Goodhart" (OpenReview UXuBzWoZGK) makes this sharper. Suppose proxy reward error is heavy-tailed, there exist rare but achievable inputs where proxy minus gold is unbounded. Under a KL constraint the optimal policy can place all its mass on these inputs: proxy reward is arbitrarily high, gold reward is at baseline. KL regularization constrains the policy distribution but does not constrain which modes it targets when those modes exist under the reference model.

The condition ("heavy-tailed error") is not exotic. Any bounded measurement of an unbounded world has heavy-tailed error in the tails, that is what "tails" means.

What actually works (partially)

  • Ensemble RMs with worst-case aggregation (Coste et al., 2023). The optimizer can break one RM but not all of them simultaneously.
  • Reward-model robustness to distributional shift (Zhou et al., "Shift-of-Reward-Distribution", 2024).
  • Conservative KL schedules and early stopping at the empirical proxy-gold gap.
  • Direct Alignment Algorithms (DPO, Lesson 3), which have their own Goodhart failure modes, proven in Rafailov et al. "Scaling Laws for Reward Model Over-optimization in Direct Alignment Algorithms" (NeurIPS 2024).

None of these eliminate reward hacking. They move the curve's peak further out. This is often enough for a shipping product. It is never enough for a "solved" alignment claim.

The 2026 unified view

"Reward Hacking in the Era of Large Models" (arXiv:2604.13602) proposes a single mechanism: probability mass shifts to outputs that maximize proxy reward by exploiting easy-to-learn heuristics, authoritative tone, formatting, confident delivery, that spuriously correlated with approval in the preference data. The paper unifies verbosity, sycophancy, unfaithful CoT, and evaluator tampering as the same optimizer-plus-proxy interaction with different affordances per deployment.

This view implies the defense is also unified. Every mitigation has to either reduce proxy-target gap (better data, better RMs), reduce optimization pressure (conservative schedules, early stop), or shift selection pressure onto hard-to-game features (process supervision, debate, information flow control).

Use It

code/main.py simulates Gao et al.'s over-optimization curves on a toy regression problem. The "gold" reward is the true linear function of a feature vector. The "proxy" RM is the gold plus Gaussian noise fit on a finite sample. A policy is a mean of a Gaussian over features; training is hill-climbing on proxy reward with a KL penalty to the initial policy. You can vary: sample size of the proxy, KL coefficient, and the noise tail heaviness. Watch the proxy-gold gap open at exactly the KL distance the paper predicts.

Ship It

This lesson produces outputs/skill-reward-hack-auditor.md. Given a trained RLHF model and its training reports, it identifies which of the four reward-hacking costumes shows up, locates the proxy-target gap in the training logs, and recommends the specific mitigation from {data, RM robustness, KL schedule, process supervision} that the evidence supports.

Exercises

  1. Run code/main.py. Reproduce the gold-peak-then-collapse shape for proxies fit on 100, 300, 1000 samples. Where does each curve peak in KL units?
  2. Modify the noise distribution from Gaussian to a Student-t with low degrees of freedom (heavy-tailed). Keep the proxy RM training setup unchanged. What changes about the peak location and post-peak collapse?
  3. Read Gao et al. Figure 1 (ICML 2023). The paper proposes a functional form for the proxy-gold gap. Fit it to your simulated curves from Exercise 1 and compare parameters.
  4. Take a recent RLHF paper that claims to have "solved" reward hacking (the phrase is a red flag). Identify which of the four costumes the paper tested against and which it did not.
  5. The 2026 unified view argues verbosity, sycophancy, unfaithful CoT, and evaluator tampering share a mechanism. Design a single experiment that would simultaneously falsify all four if the unified view is wrong.

Key Terms

TermWhat people sayWhat it actually means
Goodhart's Law"optimizing a proxy breaks it"Any strong optimizer against an imperfect proxy reliably finds inputs where the proxy-target gap is large
Gold reward"what we actually want"The target the proxy is a noisy measurement of; in practice, a larger-sample RM or human eval
Proxy reward"the RM"The scalar used during training; by construction, it is what the optimizer sees
Over-optimization curve"the reward-hacking U-curve"Proxy climbs, gold peaks then falls as KL from initial policy grows
KL budget"how far we can drift"sqrt(KL(pi || pi_init)); Gao et al. plot reward against this
Catastrophic Goodhart"KL does not save you"Under heavy-tailed reward error, KL-constrained optimal policy can maximize proxy while providing no gold utility
Unfaithful reasoning"wrong CoT, right answer"Chain-of-thought that does not causally drive the final prediction
Evaluator tampering"gaming the scorer"Agent modifies its environment, scratchpad, or the RM's inputs to register success

Further Reading

Check what you learned

6 questions. Answer each one to see why.

Question 1 of 60 correct

Goodhart's Law, as originally stated, says:

The full code

Every file for this lesson. Copy it, run it, then change it.

main.py
"""Reward hacking over-optimization curve — stdlib Python.

Reproduces the shape of Gao, Schulman, Hilton (ICML 2023): as a policy drifts
from an initial reference (measured in sqrt(KL)), proxy reward climbs
monotonically while gold reward peaks and falls. We build toy gold and
proxy linear reward models and hill-climb a mean-vector policy under a KL
penalty. You can vary proxy sample size and noise tails.

Usage: python3 code/main.py
"""

from __future__ import annotations

import math
import random
from dataclasses import dataclass


random.seed(42)

D = 8
GOLD_W = [1.0, -0.6, 0.4, 0.2, -0.1, 0.3, -0.5, 0.8]


def dot(a: list[float], b: list[float]) -> float:
    return sum(x * y for x, y in zip(a, b))


def gauss() -> float:
    return random.gauss(0.0, 1.0)


def student_t(df: float) -> float:
    """Heavy-tailed noise. For df=3, variance finite but kurtosis infinite."""
    u = random.gauss(0.0, 1.0)
    chi2 = sum(random.gauss(0.0, 1.0) ** 2 for _ in range(int(df)))
    if chi2 <= 0:
        chi2 = 1e-6
    return u * math.sqrt(df / chi2)


def sample_feature() -> list[float]:
    return [gauss() for _ in range(D)]


def gold_reward(x: list[float]) -> float:
    return dot(GOLD_W, x)


@dataclass
class ProxyRM:
    w: list[float]
    n_samples: int

    def score(self, x: list[float]) -> float:
        return dot(self.w, x)


def train_proxy(n_samples: int, noise: str = "gauss") -> ProxyRM:
    """Fit a linear proxy RM by least squares from n labels of gold + noise."""
    xs = [sample_feature() for _ in range(n_samples)]
    ys = []
    for x in xs:
        eps = gauss() if noise == "gauss" else student_t(3.0)
        ys.append(gold_reward(x) + eps)
    # normal equations: w = (X^T X)^-1 X^T y
    # closed form with gram matrix inversion in D dims (tiny linear system)
    g = [[0.0] * D for _ in range(D)]
    b = [0.0] * D
    for x, y in zip(xs, ys):
        for i in range(D):
            b[i] += x[i] * y
            for j in range(D):
                g[i][j] += x[i] * x[j]
    # add ridge to keep matrix invertible when n_samples is tiny
    for i in range(D):
        g[i][i] += 1e-3
    w = solve(g, b)
    return ProxyRM(w=w, n_samples=n_samples)


def solve(a: list[list[float]], b: list[float]) -> list[float]:
    """Gaussian elimination. D is small so this is fine."""
    n = len(b)
    m = [row[:] + [b[i]] for i, row in enumerate(a)]
    for i in range(n):
        piv = i
        for k in range(i + 1, n):
            if abs(m[k][i]) > abs(m[piv][i]):
                piv = k
        m[i], m[piv] = m[piv], m[i]
        for k in range(i + 1, n):
            f = m[k][i] / m[i][i]
            for j in range(i, n + 1):
                m[k][j] -= f * m[i][j]
    x = [0.0] * n
    for i in range(n - 1, -1, -1):
        x[i] = (m[i][n] - sum(m[i][j] * x[j] for j in range(i + 1, n))) / m[i][i]
    return x


def sqrt_kl_from_origin(mu: list[float]) -> float:
    """Two unit-variance Gaussians, one at 0, one at mu. KL = 1/2 * ||mu||^2."""
    return math.sqrt(0.5 * sum(m * m for m in mu))


def expected_reward(w: list[float], mu: list[float]) -> float:
    """E_{x ~ N(mu, I)} [<w, x>] = <w, mu>."""
    return dot(w, mu)


def best_of_n_sweep(proxy: ProxyRM, ns: list[int]) -> list[tuple[float, float, float]]:
    """Simulate best-of-n sampling at each n. Compute mean KL, proxy, gold
    scores of the chosen response."""
    curve = []
    trials = 1000
    for n in ns:
        kls = []
        proxies = []
        golds = []
        for _ in range(trials):
            xs = [sample_feature() for _ in range(n)]
            best = max(xs, key=proxy.score)
            proxies.append(proxy.score(best))
            golds.append(gold_reward(best))
            # KL of best-of-n distribution vs uniform is log(n) nats in limit
            # we compute a proxy: distance of best from mean
            kls.append(math.sqrt(0.5 * sum(b * b for b in best)))
        curve.append((
            sum(kls) / trials,
            sum(proxies) / trials,
            sum(golds) / trials,
        ))
    return curve


def kl_constrained_policy_sweep(proxy: ProxyRM,
                                kl_budgets: list[float]) -> list[tuple[float, float, float]]:
    """Solve argmax_mu <w_proxy, mu> - lambda * ||mu||^2/2, sweep lambda."""
    curve = []
    for kl in kl_budgets:
        # optimal mu under ||mu||^2 <= 2 * kl: scale proxy weights
        norm = math.sqrt(sum(w * w for w in proxy.w))
        if norm < 1e-9:
            mu = [0.0] * D
        else:
            s = math.sqrt(2 * kl) / norm
            mu = [w * s for w in proxy.w]
        curve.append((
            sqrt_kl_from_origin(mu),
            expected_reward(proxy.w, mu),
            expected_reward(GOLD_W, mu),
        ))
    return curve


def print_curve(name: str, curve: list[tuple[float, float, float]]) -> None:
    print(f"\n{name}")
    print("-" * 60)
    print(f"  {'sqrt(KL)':>9}  {'proxy':>8}  {'gold':>8}  {'gap':>8}")
    for sk, p, g in curve:
        print(f"  {sk:>9.3f}  {p:>8.3f}  {g:>8.3f}  {p - g:>+8.3f}")
    peak_gold = max(curve, key=lambda r: r[2])
    print(f"  gold peak at sqrt(KL) = {peak_gold[0]:.3f}, "
          f"gold = {peak_gold[2]:.3f}, proxy = {peak_gold[1]:.3f}")


def main() -> None:
    print("=" * 60)
    print("REWARD HACKING OVER-OPTIMIZATION (Phase 18, Lesson 2)")
    print("=" * 60)

    budgets = [0.0, 0.2, 0.5, 1.0, 1.5, 2.0, 3.0, 5.0, 8.0]

    for n in (100, 300, 1000, 10000):
        rm = train_proxy(n)
        curve = kl_constrained_policy_sweep(rm, budgets)
        print_curve(f"Proxy RM trained on {n} samples (Gaussian noise)", curve)

    # heavy-tailed proxy error: the Catastrophic Goodhart condition.
    rm_heavy = train_proxy(300, noise="student_t")
    curve_heavy = kl_constrained_policy_sweep(rm_heavy, budgets)
    print_curve("Proxy RM, 300 samples, Student-t(3) noise (heavy tails)",
                curve_heavy)

    # best-of-N sampling curve for comparison
    ns = [1, 2, 4, 8, 16, 64, 256, 1024]
    bon = best_of_n_sweep(train_proxy(300), ns)
    print_curve("Best-of-N sampling (300-sample proxy)", bon)

    print("\n" + "=" * 60)
    print("TAKEAWAY: proxy reward climbs monotonically; gold peaks and falls.")
    print("More proxy samples push the peak further, but do not eliminate it.")
    print("Heavy-tailed noise moves the peak closer to the origin. KL alone")
    print("does not save you. This is Goodhart's Law, measured.")
    print("=" * 60)


if __name__ == "__main__":
    main()