Get Started

Experiment Runner

The loop is only as honest as its measurements. Build the runner that takes a spec, executes it in a sandboxed subprocess, and emits a json metrics blob the evaluator can trust.

90 minutesBuildPythonMembers

Learning Objectives

  • Encode an experiment as a typed spec the runner can serialise to a subprocess.
  • Launch a subprocess with a hard wall clock timeout and a soft memory cap, and surface both as terminal conditions.
  • Capture stdout, stderr, and the structured metrics blob into a single result record.
  • Build an ablation table that sweeps one configuration knob at a time over a fixed base spec.
  • Keep every result deterministic given a seed so the evaluator sees the same numbers across runs.