AI Scientist v2, Workshop-Level Autonomous Research
Sakana's AI Scientist v2 (Yamada et al., arXiv:2504.08066) runs the full research loop: hypothesis, code, experiments, figures, writeup, submission. It is the first system to have a generated paper pass peer review at an ICLR 2025 workshop. Independent evaluation (Beel et al.) found 42% of experiments failed from coding errors and literature review frequently mislabeled established concepts as novel. Sakana's own docs warn that the codebase executes LLM-written code and recommend Docker isolation. Both halves of that picture are the point.
The Problem
Research is an open-ended task. Unlike AlphaEvolve's algorithmic search or DGM's benchmark-bounded self-modification, a research result does not have a machine-checkable correctness criterion. A paper is judged by reviewers, not unit tests. That makes the loop harder to close, and more valuable if closed, because research is where compounding progress lives.
AI Scientist v1 (Sakana, 2024) closed the loop by starting from human-authored templates. The LLM filled in experiments within a fixed scaffolding. AI Scientist v2 (Yamada et al., 2025) removes the template requirement by using agentic tree search with a vision-language model critique loop. The system generates ideas, implements experiments, produces figures, writes a paper, and iterates on reviewer feedback.
Peer review verdict: one v2-generated paper was accepted at an ICLR 2025 workshop (with disclosure). Independent evaluation verdict: the system is far from reliable. Both are true.
The Concept
The architecture
- Idea generation. The LLM proposes research ideas conditioned on a topic and prior literature. v1 used templates; v2 uses agentic search over a space of hypotheses.
- Novelty check. A literature retrieval step checks whether the idea has been published. This is the step where Beel et al.'s evaluation found mislabeling, established methods frequently classified as novel.
- Experiment plan. The agent drafts an experimental protocol and writes code.
- Execution. Code runs in a sandbox. Failures are fed back into a retry loop. In Beel et al.'s measurements, 42% of experiments failed from coding errors at this stage.
- Figure generation. A vision-language model reads generated figures and rewrites them for clarity. This was v2's key technical addition.
- Writeup. The LLM drafts a paper, iterates with an internal reviewer.
- Optional: submission. The paper is submitted to a venue.
What the workshop-acceptance result means
One v2-generated paper passed peer review at an ICLR 2025 workshop. The authors disclosed the paper's origin to the program committee. The acceptance is a data point; it is not a license to claim the system "does research."
Important context: workshop papers are a lower bar than main-conference papers. Peer review is noisy; a small fraction of submissions are accepted on any given day. One success is a proof of concept, not a reliability claim. The Nature 2026 paper documents the end-to-end loop and was itself co-authored by human researchers; it is not "the system wrote a Nature paper."
What the independent evaluation found
Beel et al. (arXiv:2502.14297) ran an external evaluation. Headline findings:
- Experiment failures. 42% of experiments failed from coding errors (bad imports, shape mismatches, undefined variables). The retry loop caught some, not all.
- Novelty mislabeling. The literature-retrieval step frequently flagged established concepts as novel. This is the research equivalent of hallucination.
- Presentation-quality gap. The vision-language figure critique produced publication-grade visuals, masking underlying experimental weaknesses.
The last finding is the important one for this phase. A system that produces convincing outputs without doing convincing research is more dangerous, not safer, than one that fails obviously. Evaluation must reach the underlying claims, not stop at the figure.
The sandbox-escape concern
Sakana's own repository README warns:
Due to the nature of this software, which executes LLM-generated code, we cannot guarantee safety. There are risks of dangerous packages, uncontrolled web access, and spawning of unintended processes. Use at your own risk and consider Docker isolation.
This is the operational shape of autonomy in an unverified domain. The LLM writes code; the code runs; the code can do anything the process is allowed to do. Without a sandbox that hard-limits filesystem, network, and process actions, any self-directed research agent can exfiltrate data, burn compute, or rewrite itself.
AlphaEvolve's sandbox story is easier because its evaluator is tight. AI Scientist v2's loop runs open-ended code with open-ended goals. That is why it needs stronger isolation (Docker minimum; seccomp / gVisor preferred) and a manual review of every submission before it leaves the system.
Where v2 sits in the frontier stack
| System | Target | Output kind | Evaluator | Known failure |
|---|---|---|---|---|
| AlphaEvolve | algorithms | code | unit + benchmark | bounded by evaluator rigor |
| DGM | agent scaffolding | code | SWE-bench | reward hacking |
| AI Scientist v2 | research papers | text + code + figures | peer review (weak) | experiment failures, mislabeling, polish masking weakness |
v2 has the weakest automatic evaluator of the three, the widest output surface, and the shortest path to public artifacts. The operational controls (sandbox, review, disclosure) are doing most of the safety work.
Use It
code/main.py simulates the v2 loop as a state machine: idea → novelty check → experiment → figure → writeup → review → accept-or-iterate. Each state has a configurable failure probability pulled from the Beel et al. findings. Run the simulator for N loops and count:
- How many ideas reach submission.
- How many submissions would have a critical experimental flaw the polished paper hides.
- How retry budgets trade off quality vs yield.
Ship It
outputs/skill-ai-scientist-sandbox-review.md is a two-gate review checklist for anything produced by a research-loop agent before it leaves the sandbox.
Exercises
- Run
code/main.pywith default parameters. What fraction of loop runs produce a "clean" paper? What fraction produce a paper with an experiment-failure flaw the figure critique polished over? - The defaults already use Beel et al.'s 42% / 25%. Re-run with
--experiment-failure 0.20 --novelty-mislabel 0.10and then with--experiment-failure 0.60 --novelty-mislabel 0.40. How does the polished-but-flawed share shift between the two runs? - Read Sakana's AI Scientist v2 repo README on sandbox requirements. Name two additional restrictions (beyond Docker) you would apply for a multi-day autonomous run.
- Read Beel et al. Section 4 on presentation-quality gap. Design one additional evaluator that would catch polished-looking but experimentally flawed papers.
- Propose a human-review protocol for research-agent outputs that scales better than "a PhD reads every paper." Identify the bottleneck and design around it.
Key Terms
| Term | What people say | What it actually means |
|---|---|---|
| AI Scientist v1 | "Sakana's templated research agent" | Filled experiments into a fixed scaffold |
| AI Scientist v2 | "Template-free research agent" | Agentic tree search with VLM figure critique |
| Agentic tree search | "Branching research agent" | Expands multiple experiment plans in parallel; prunes by internal critic |
| Vision-language critique | "VLM polish on figures" | Multimodal model reads figures and rewrites them for clarity |
| Literature retrieval | "Novelty check" | Searches prior work to confirm idea novelty, documented to mislabel |
| Polish masking | "Pretty paper, broken research" | Presentation quality exceeds experimental quality; hides weaknesses |
| Sandbox escape | "LLM code breaks out" | Agent-executed code does things the loop designer did not intend |
Further Reading
- Yamada et al. (2025). The AI Scientist-v2paper.
- Sakana blog on the Nature 2026 publicationvendor summary with peer-review context.
- Beel et al. (2025). Independent evaluation of The AI Scientistexternal evaluation numbers.
- Sakana AI Scientist v1 paperthe templated predecessor.
- Anthropic, Measuring AI agent autonomybroader framing of open-ended research agents.
The full code
Every file for this lesson. Copy it, run it, then change it.
"""AI Scientist v2 loop simulator — stdlib Python.
Models the research loop as a state machine with configurable per-stage
failure probabilities, seeded from Beel et al. (2025) findings on AI
Scientist's real behavior. Runs many trials and reports the distribution
of outcomes, including the critical "polished paper with flawed
experiment" class.
"""
from __future__ import annotations
import argparse
import random
from dataclasses import dataclass
DEFAULT_SEED = 42
@dataclass
class LoopConfig:
# Probability an idea is mislabeled as novel when it is not.
novelty_mislabel: float = 0.25
# Probability an experiment fails from coding errors (Beel et al. ~0.42).
experiment_failure: float = 0.42
# Fraction of experiment failures recoverable by retries.
retry_recovery: float = 0.55
# Probability vision-language figure critique produces clean visuals
# even when underlying experiment is broken.
polish_masks_weakness: float = 0.70
# Probability the auto-writeup step produces a coherent paper given
# (possibly flawed) experiment data.
writeup_success: float = 0.85
# Internal reviewer accept probability (weak reviewer).
internal_review_accept: float = 0.50
@dataclass
class Outcome:
submitted: bool
has_novelty_flaw: bool
has_experiment_flaw: bool
polished_but_flawed: bool
polished_ok: bool
abandoned_stage: str
def run_one(cfg: LoopConfig) -> Outcome:
# Idea generation always succeeds in this toy.
has_novelty_flaw = random.random() < cfg.novelty_mislabel
# Experiment execution: failure + retry recovery.
failed = random.random() < cfg.experiment_failure
if failed:
recovered = random.random() < cfg.retry_recovery
if not recovered:
return Outcome(
submitted=False,
has_novelty_flaw=has_novelty_flaw,
has_experiment_flaw=True,
polished_but_flawed=False,
polished_ok=False,
abandoned_stage="experiment",
)
# Modeling choice: a retry-recovered experiment still carries a
# residual flaw (silently-wrong numerics, shape-mismatch patched
# without re-validation, etc.). This residual flaw is what the
# polish stage can mask later and is the headline driver of the
# "polished-but-flawed" category.
has_experiment_flaw = True
else:
has_experiment_flaw = False
# Vision-language figure polish.
polished_hides_weakness = (
has_experiment_flaw and random.random() < cfg.polish_masks_weakness
)
# Writeup stage.
if random.random() > cfg.writeup_success:
return Outcome(
submitted=False,
has_novelty_flaw=has_novelty_flaw,
has_experiment_flaw=has_experiment_flaw,
polished_but_flawed=False,
polished_ok=False,
abandoned_stage="writeup",
)
# Internal reviewer.
if random.random() > cfg.internal_review_accept:
return Outcome(
submitted=False,
has_novelty_flaw=has_novelty_flaw,
has_experiment_flaw=has_experiment_flaw,
polished_but_flawed=False,
polished_ok=False,
abandoned_stage="internal_review",
)
polished_ok = not has_experiment_flaw and not has_novelty_flaw
# Any submitted paper with a flaw counts as polished_but_flawed: the
# weak internal reviewer let it through whether or not the polish
# stage hid it. This makes the two buckets exhaustive over submitted
# papers (polished_ok + polished_but_flawed == len(submitted)).
polished_but_flawed = has_experiment_flaw or has_novelty_flaw
return Outcome(
submitted=True,
has_novelty_flaw=has_novelty_flaw,
has_experiment_flaw=has_experiment_flaw,
polished_but_flawed=polished_but_flawed,
polished_ok=polished_ok,
abandoned_stage="",
)
def report(n: int, cfg: LoopConfig) -> None:
outs = [run_one(cfg) for _ in range(n)]
submitted = [o for o in outs if o.submitted]
abandoned = [o for o in outs if not o.submitted]
polished_ok = [o for o in submitted if o.polished_ok]
polished_but_flawed = [o for o in submitted if o.polished_but_flawed]
print(" config")
print(f" novelty mislabel rate : {cfg.novelty_mislabel:.2f}")
print(f" experiment failure rate : {cfg.experiment_failure:.2f}")
print(f" retry recovery fraction : {cfg.retry_recovery:.2f}")
print(f" polish masks weakness prob : {cfg.polish_masks_weakness:.2f}")
print(f" writeup success rate : {cfg.writeup_success:.2f}")
print(f" internal reviewer accept : {cfg.internal_review_accept:.2f}")
print()
print(f" trials : {n}")
print(f" submissions : {len(submitted)} ({len(submitted) / n:.1%})")
print(f" abandoned : {len(abandoned)} ({len(abandoned) / n:.1%})")
by_stage = {}
for o in abandoned:
by_stage[o.abandoned_stage] = by_stage.get(o.abandoned_stage, 0) + 1
for stage, count in sorted(by_stage.items()):
print(f" at {stage:<18}: {count}")
print()
print(" submission quality breakdown")
print(f" clean (novel + valid) : {len(polished_ok)} "
f"({len(polished_ok) / n:.1%} of trials, "
f"{len(polished_ok) / max(1, len(submitted)):.1%} of submissions)")
print(f" polished-but-flawed : {len(polished_but_flawed)} "
f"({len(polished_but_flawed) / n:.1%} of trials, "
f"{len(polished_but_flawed) / max(1, len(submitted)):.1%} of submissions)")
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--experiment-failure", type=float, default=None,
help="override LoopConfig.experiment_failure for the baseline run")
parser.add_argument("--novelty-mislabel", type=float, default=None,
help="override LoopConfig.novelty_mislabel for the baseline run")
parser.add_argument("--seed", type=int, default=DEFAULT_SEED,
help="RNG seed (default: %(default)s)")
args = parser.parse_args()
random.seed(args.seed)
print("=" * 70)
print("AI SCIENTIST V2 LOOP SIMULATOR (Phase 15, Lesson 5)")
print("=" * 70)
overrides = {}
if args.experiment_failure is not None:
overrides["experiment_failure"] = args.experiment_failure
if args.novelty_mislabel is not None:
overrides["novelty_mislabel"] = args.novelty_mislabel
baseline_cfg = LoopConfig(**overrides)
label = "Baseline (Beel-style numbers)" if not overrides else "Baseline (overridden)"
print(f"\n{label}")
print("-" * 70)
report(1000, baseline_cfg)
print("\nOptimistic scenario (tighter numbers)")
print("-" * 70)
report(1000, LoopConfig(
novelty_mislabel=0.10,
experiment_failure=0.20,
retry_recovery=0.80,
polish_masks_weakness=0.40,
writeup_success=0.92,
internal_review_accept=0.60,
))
print()
print("=" * 70)
print("HEADLINE: submissions outpace sound research")
print("-" * 70)
print(" Even in optimistic scenarios, a non-trivial share of submitted")
print(" papers carry a flaw the polish stage helped hide. That is the")
print(" operational meaning of 'presentation-quality gap' — and the")
print(" reason a human review gate sits between the loop and any venue.")
if __name__ == "__main__":
main()