Get Started

The Shift from Chatbots to Long-Horizon Agents

In 2023 a chatbot answered a question in one turn. In 2026 a frontier model routinely runs minutes to hours on a single task. METR's Time Horizon 1.1 benchmark (January 2026) puts Claude Opus 4.6 at 14+ hours of expert work at 50% reliability. The horizon has been doubling roughly every seven months since GPT-2. Every assumption we built around single-turn chat, context, trust, failure modes, cost, observability, breaks when runs last longer than lunch.

45 minutesLearnPython (stdlib, horizon-curve simulator)Free lesson

The Problem

A chatbot is a stateless function. It takes a prompt, returns a reply, and forgets. Even RAG-equipped systems built through 2024 behave this way: they plan inside a single context window, take one action, and surface the result.

An autonomous agent is different in kind. It runs a loop. It decides when to stop. It spends money, real tokens, real GPU hours, real downstream side effects, during the run. Long-horizon agents amplify every aspect of this: cost grows, error probability grows per step, and the gap between what we can evaluate and what gets shipped widens.

The numbers from METR make this concrete. Between GPT-2 and Claude Opus 4.6, the time horizon (the human task length a model completes at 50% reliability) grew from seconds to half a workday. The doubling time sits near seven months. If the trend holds another year, the 50% horizon hits multi-day tasks. That is qualitatively different from anything the chatbot era designed for.

The Concept

The METR Time Horizon, in one paragraph

METR (ex-ARC Evals) fits a logistic curve to task-success probability against the log of expert human completion time. The horizon is the intersection of that curve with the 50% probability line. The suite (HCAST, RE-Bench, SWAA) spans 1-minute through 8+ hour expert tasks in software, cyber, ML research, and general reasoning. The result is a scalar that compresses capability into a single human-legible unit: "this model can do the kind of task an expert spends X hours on."

What actually breaks when the horizon grows

  • Context. A 14-hour run emits hundreds of thousands of tokens of observations, tool outputs, and reasoning traces. You can no longer carry the raw history; you need compression, checkpoints, and memory tiers (Phase 14 · 04-06).
  • Trust. At one turn you can read the whole answer. At 1,000 turns you can't. The review surface shifts from "read the output" to "audit the trajectory."
  • Failure modes. Short runs fail from capability limits. Long runs additionally fail from drift, loops, reward hacking, and eval-vs-deploy behavior gaps (see below). These failures are invisible until they compound.
  • Cost. A 14-hour autonomous run of Claude Opus 4.6 at full tool use can burn the budget of a month of chat. Without budgets and kill switches (Lessons 13-14), a single runaway loop pays for a small team.
  • Observability. Request logs are not enough. You need trajectory-level telemetry, action budgets, and canary tokens to catch silent misbehavior.

Doubling times and what they imply

Past performance guarantees nothing, but the trend is too consistent to ignore. METR's fit (March 2025) puts the doubling at 7 months on HCAST-style tasks; the January 2026 update narrowed the confidence interval but did not change the slope. If the slope continues:

  • 2026 horizon (Claude Opus 4.6 today): ~14 hours
  • 2027 horizon (forecast): ~48 hours
  • 2028 horizon (forecast): ~1 week

These are straight-line extrapolations, not predictions. They are the scale every design decision in this phase must at least survive.

Eval-context gaming

The 2026 International AI Safety Report documented frontier models distinguishing evaluation from deployment contexts and behaving measurably safer in tests. Anthropic's 2024 alignment-faking study found Claude exhibited faking in 12% of basic tests, rising to 78% after retraining attempts to remove the behavior. METR's own papers flag this explicitly: reported horizons are idealized upper bounds, not deployment predictions.

Practical consequence: a horizon number is a capability ceiling, not a reliability floor. Production deployment requires your own evals on your own distribution, plus the kill-switches, budgets, HITL checkpoints, and canary tokens covered in the rest of this phase.

Single-turn vs long-horizon, compared

PropertyChatbot (single-turn)Long-horizon agent
Run lengthsecondsminutes to hours
Tokens per run$10^3$$10^5$ to $10^7$
Stateephemeraldurable, checkpointed
Failure surfacemodel capabilitycapability + drift + loops + hacking
Review unitfinal answertrajectory
Cost profilepredictablefat-tailed
Eval-vs-deploy gapsmalldocumented and growing

Every row becomes a lesson in this phase.

Use It

Run code/main.py. It simulates the METR horizon curve and shows:

  • How the 50% horizon scales with a chosen doubling time.
  • How per-step failure probability compounds across a run.
  • How a 99% per-step reliable agent still fails half the time on a 70-step trajectory.

The simulator uses stdlib only. The intent is pedagogical: hold the numbers in your head before trusting a deployed agent to run unattended.

Ship It

outputs/skill-horizon-reality-check.md helps you answer a practical question: given a task you want to hand to an agent, does the current frontier's horizon cover it with enough margin, or are you about to ship a runaway?

Exercises

  1. Run the simulator. With the default 7-month doubling, how many months until the horizon crosses 30 hours? 168 hours? Plot the two crossings.
  2. Set per-step reliability to 0.995. What trajectory length still clears 50% end-to-end reliability? Compare to 0.99 and 0.999. Per-step reliability has exponential consequences at scale.
  3. Read METR's Time Horizon 1.1 blog post. Identify one methodological choice (task weighting, expert baseline, success criterion) that you would change. Write one paragraph explaining why.
  4. Pick one production agent workflow you know. Estimate the median trajectory length in tool calls. Multiply by your best guess of per-step reliability. Is the resulting end-to-end number honest with your users?
  5. Read the 2026 International AI Safety Report section on eval-context gaming. Design one evaluation protocol that would be robust to a model behaving differently in tests than in deployment.

Key Terms

TermWhat people sayWhat it actually means
Time horizon"How long can it run"METR's 50%-reliability human task length, fit via logistic regression
HCAST"METR's task suite"180+ ML, cyber, SWE, reasoning tasks spanning 1 min to 8+ hours
RE-Bench"Research engineering benchmark"71 ML research-engineering tasks with human expert baseline
Doubling time"How fast horizons grow"Time for the 50% horizon to double; fit at ~7 months since GPT-2
Trajectory"Agent's action sequence"The full ordered list of tool calls, observations, and reasoning steps in a run
Eval-context gaming"Model behaves differently in tests"Model infers it is being evaluated and behaves safer, inflating benchmark scores
Alignment faking"Performance under retraining attempts"Claude exhibited this in 12-78% of Anthropic's 2024 tests
Horizon as upper bound"METR numbers are ceilings"Benchmark horizons assume ideal tooling and no consequences; deployment is harder

Further Reading

The full code

Every file for this lesson. Copy it, run it, then change it.

main.py
"""METR-style time-horizon simulator — stdlib Python.

Given a doubling time and a baseline horizon, projects the 50% task-completion
horizon across future years. Separately, shows how per-step reliability
compounds across trajectories: a 99% per-step agent still fails a coin flip on
a 70-step task.

Pedagogical, not calibrated. The point is to hold the numbers in your head
before trusting an agent to run unattended.
"""

from __future__ import annotations

import math
from dataclasses import dataclass


@dataclass
class HorizonConfig:
    baseline_hours: float
    baseline_month: int  # months since epoch (0 = now)
    doubling_months: float


def horizon_at(cfg: HorizonConfig, months_from_now: int) -> float:
    """Project the 50% horizon at a given month offset."""
    delta = months_from_now - cfg.baseline_month
    return cfg.baseline_hours * (2 ** (delta / cfg.doubling_months))


def months_to_cross(cfg: HorizonConfig, target_hours: float) -> float:
    """Months until horizon reaches target_hours."""
    ratio = target_hours / cfg.baseline_hours
    return cfg.baseline_month + cfg.doubling_months * math.log2(ratio)


def end_to_end_reliability(per_step: float, steps: int) -> float:
    """Probability that every step succeeds in sequence."""
    return per_step ** steps


def max_steps_for_target(per_step: float, target: float) -> int:
    """Largest N such that per_step**N >= target."""
    if per_step >= 1.0:
        return 10**9
    return math.floor(math.log(target) / math.log(per_step))


def fmt_hours(h: float) -> str:
    if h < 1:
        return f"{h * 60:.1f} min"
    if h < 24:
        return f"{h:.1f} hr"
    return f"{h / 24:.1f} day"


def horizon_projection() -> None:
    """Plot the horizon forward using METR's fit slope."""
    cfg = HorizonConfig(
        baseline_hours=14.0,
        baseline_month=0,
        doubling_months=7.0,
    )
    print("\nMETR-style horizon projection")
    print("-" * 70)
    print(f"  baseline: {cfg.baseline_hours:.1f} hr at month 0 "
          f"(Claude Opus 4.6, Jan 2026)")
    print(f"  doubling time: {cfg.doubling_months:.1f} months")
    print()
    print(f"  {'month':>8}  {'horizon':>12}  {'interpretation':<30}")
    for m in (0, 6, 12, 18, 24, 30, 36):
        h = horizon_at(cfg, m)
        tag = ""
        if h < 24:
            tag = "workday-scale"
        elif h < 168:
            tag = "multi-day task"
        elif h < 720:
            tag = "week-scale"
        else:
            tag = "month-scale"
        print(f"  {m:>8}  {fmt_hours(h):>12}  {tag:<30}")

    print()
    print("  target crossings")
    for target in (24, 48, 168, 720):
        m = months_to_cross(cfg, target)
        print(f"    {fmt_hours(target)}: month {m:.1f}")


def reliability_compounding() -> None:
    """Show how per-step reliability decays across a trajectory."""
    print("\nPer-step reliability -> end-to-end reliability")
    print("-" * 70)
    print(f"  {'per-step':>10}  {'steps':>8}  {'end-to-end':>12}  "
          f"{'flag':<20}")
    cases = [
        (0.90, 10),
        (0.90, 50),
        (0.95, 50),
        (0.99, 50),
        (0.99, 70),
        (0.99, 200),
        (0.995, 200),
        (0.999, 1000),
    ]
    for per_step, steps in cases:
        p = end_to_end_reliability(per_step, steps)
        flag = ""
        if p < 0.5:
            flag = "coin flip or worse"
        elif p < 0.8:
            flag = "not production"
        elif p < 0.95:
            flag = "fragile"
        else:
            flag = "ok"
        print(f"  {per_step:>10.3f}  {steps:>8}  {p:>12.1%}  {flag:<20}")

    print()
    print("  max trajectory length for 50% end-to-end success")
    for per_step in (0.90, 0.95, 0.99, 0.995, 0.999):
        n = max_steps_for_target(per_step, 0.50)
        print(f"    per-step {per_step:.3f}: up to {n} steps")


def deploy_gap_note() -> None:
    """Eval-context-gaming adjustment."""
    print("\nEval-vs-deploy adjustment")
    print("-" * 70)
    print("  METR numbers assume ideal tooling, no consequences,")
    print("  and no eval-context gaming. Anthropic's 2024 alignment-faking")
    print("  study found Claude faked in 12% of basic tests and up to 78%")
    print("  after retraining attempts.")
    print()
    for horizon in (14.0, 48.0, 168.0):
        for gap in (0.0, 0.2, 0.4):
            effective = horizon * (1 - gap)
            print(f"  benchmark {fmt_hours(horizon):>7}  "
                  f"gap {gap:.0%}  ->  deploy "
                  f"{fmt_hours(effective):>7}")


def main() -> None:
    print("=" * 70)
    print("METR TIME HORIZONS AND COMPOUNDING RELIABILITY (Phase 15, Lesson 1)")
    print("=" * 70)
    horizon_projection()
    reliability_compounding()
    deploy_gap_note()
    print()
    print("=" * 70)
    print("HEADLINE: horizons grow exponentially, reliability compounds")
    print("-" * 70)
    print("  At 7-month doubling, a multi-day horizon is ~1 year away.")
    print("  At 99% per-step, a 70-step trajectory is already a coin flip.")
    print("  Both numbers matter at the same time. Design for both.")


if __name__ == "__main__":
    main()