Experiment Runner
The loop is only as honest as its measurements. Build the runner that takes a spec, executes it in a sandboxed subprocess, and emits a json metrics blob the evaluator can trust.
90 minutesBuildPythonMembers
Learning Objectives
- Encode an experiment as a typed spec the runner can serialise to a subprocess.
- Launch a subprocess with a hard wall clock timeout and a soft memory cap, and surface both as terminal conditions.
- Capture stdout, stderr, and the structured metrics blob into a single result record.
- Build an ablation table that sweeps one configuration knob at a time over a fixed base spec.
- Keep every result deterministic given a seed so the evaluator sees the same numbers across runs.