Get Started

Code Exec Metric

Generated code is right when it passes the tests. The eval harness has to extract code, run it without crashing the host, and tally pass-rates honestly. This lesson builds that surface.

90 minBuildPythonMembers

Learning objectives

  • Extract a code block from a free-form generation in a way that matches the post-process rule from lesson 70.
  • Execute candidate code in an isolated subprocess with a wall-clock timeout, output cap, and an import denylist.
  • Score a task as the fraction of supplied assertion strings that pass against the candidate.
  • Compute pass-at-k for tasks that sample multiple generations from one model.
  • Treat sandbox crashes, syntax errors, and timeouts as first-class fail modes with distinct exit codes the runner can log.