Get Started

Capstone 16, GitHub Issue-to-PR Autonomous Agent

Label an issue, get a PR, the 2026 product shape for autonomous coding agents: run an agent in a cloud sandbox, verify tests pass, and post a review-ready PR with rationale. AWS Remote SWE Agents, Cursor Background Agents, OpenAI Codex cloud, and Google Jules all ship it. The hard parts are reproducing the repo's build environment automatically, preventing credential leakage, enforcing per-repo budgets, and making sure the agent cannot force-push. This capstone builds the self-hosted version and compares it on cost and pass rate to the hosted alternatives.

30 hoursCapstonePython (agent), TypeScript (GitHub App), YAML (Actions)Free lesson

Problem

The async cloud coding agent is a separate product category from interactive coding agents (capstone 01). The UX is a GitHub label. You label an issue @agent fix thisa worker spins up in a cloud sandbox, clones the repo, runs tests, edits files, verifies, and opens a PR with the agent's rationale in the body. No interactive loop, no terminal. AWS Remote SWE Agents, Cursor Background Agents, OpenAI Codex cloud, Google Jules, and Factory Droids all converge on this.

The engineering challenges are concrete: environment reproduction (the agent has to build the repo from scratch without a cached dev image), flaky tests (must be re-run or isolated), credential scoping (a GitHub App with minimal fine-grained permissions), budget enforcement per repo per day, and no-force-push policy. The capstone measures pass rate, cost, and safety vs the hosted alternatives.

Concept

The trigger is a GitHub webhook (issue label or PR comment). A dispatcher enqueues work to ECS Fargate or Lambda. The worker pulls the repo into a Daytona or E2B sandbox with a generic Dockerfile inferred from the repo (language, framework). The agent runs a mini-swe-agent or SWE-agent v2 loop against Claude Opus 4.7 or GPT-5.4-Codex. It iterates: read code, propose fix, apply patch, run tests.

Verification is the gating step. Full CI must pass in the sandbox before the PR opens. Coverage delta is computed; if negative beyond a threshold, the PR opens but gets labeled needs-review. The agent posts the rationale as the PR description plus an @agent thread the reviewer can ping for follow-ups.

Safety is scoped through two different GitHub surfaces: the App provides a short-lived installation token with workflows: read and narrow repo contents/PR scopes; branch protection (not app permissions) enforces "no direct writes to main" and "no force-push", the app is never added to the bypass list. Path-scoped read-only access to .github/workflows is not a real GitHub App primitive, so the agent's allow-list on file edits has to enforce that at the worker. Budget ceilings per repo per day are enforced at the dispatcher (e.g., max 5 PRs per repo per day, $20 per PR).

Architecture

Code
GitHub issue labeled `@agent fix` or PR comment
            |
            v
    GitHub App webhook -> AWS Lambda dispatcher
            |
            v
    ECS Fargate task (or GitHub Actions self-hosted runner)
       - pull repo
       - infer Dockerfile (language, package manager)
       - Daytona / E2B sandbox with target runtime
       - clone -> git worktree -> agent branch
            |
            v
    mini-swe-agent / SWE-agent v2 loop
       Claude Opus 4.7 or GPT-5.4-Codex
       tools: ripgrep, tree-sitter, read/edit, run_tests, git
            |
            v
    verify CI passes in-sandbox + coverage delta check
            |
            v (verified)
    git push + open PR via GitHub App
       PR body = rationale + diff summary + trace URL
       label: needs-review
            |
            v
    operator reviews; can @-mention agent for follow-ups

Stack

  • Trigger: GitHub App with fine-grained token; webhook receiver via Lambda or Fly.io
  • Worker: ECS Fargate task (or GitHub Actions self-hosted runner)
  • Sandbox: Daytona devcontainer or E2B sandbox per task
  • Agent loop: mini-swe-agent baseline or SWE-agent v2 over Claude Opus 4.7 / GPT-5.4-Codex
  • Retrieval: tree-sitter repo-map + ripgrep
  • Verification: full CI in-sandbox + coverage delta gate
  • Observability: Langfuse with per-PR trace archive linked from the PR body
  • Budget: per-repo daily dollar ceiling; max PRs per repo per day

Build It

  1. GitHub App. Fine-grained installation token: issues read+write, pull_requests write, contents read+write, workflows read. Branch protection (the only surface that can do this) enforces "no direct push to main" and "no force-push"; the app is not in the bypass list. The worker enforces "no writes under .github/workflows" as an allow-list check on the proposed diff, since GitHub App permissions are not path-scoped.
  2. Webhook receiver. Lambda function accepts issue label / PR comment webhooks. Filters by label @agent fix this. Enqueues to SQS.
  3. Dispatcher. Pops tasks from SQS. Enforces per-repo per-day budget. Spins up an ECS Fargate task with the repo URL, issue body, and a fresh Daytona sandbox.
  4. Environment inference. Detect language (Python, Node, Go, Rust) and package manager (uv, pnpm, go mod, cargo). Generate a Dockerfile on the fly if one does not exist.
  5. Agent loop. mini-swe-agent or SWE-agent v2 with Claude Opus 4.7. Tools: ripgrep, tree-sitter repo-map, read_file, edit_file, run_tests, git. Hard limits: $20 cost, 30 min wall-clock, 30 agent turns.
  6. Verification. After the loop concludes, run the full test suite in-sandbox. Compute coverage delta via jacoco / coverage.py. If CI red: halt, do not open PR. If coverage drops more than 2%: open PR with needs-review label.
  7. PR posting. Push the agent branch. Open PR via GitHub API with: title, rationale, diff summary, trace URL, cost, turns.
  8. Credential hygiene. Worker runs with a short-lived GitHub App installation token. Logs are scrubbed for secrets before archival.
  9. Eval. 30 seeded internal issues of varying difficulty. Measure pass rate, PR quality (diff size, style, coverage), cost, latency. Compare with Cursor Background Agents and AWS Remote SWE Agents on the same issues.

Use It

Code
# on github.com
  - user labels issue #842 with `@agent fix this`
  - PR #1903 appears 14 minutes later
  - body:
    > Fixed NPE in widget.dedupe() caused by null comparator entry.
    > Added regression test widget_test.go::TestDedupeNullComparator.
    > Coverage delta: +0.12%
    > Turns: 7  Cost: $1.80  Trace: langfuse:...
    > Label: needs-review

Ship It

outputs/skill-issue-to-pr.md is the deliverable. A GitHub App + async cloud worker that turns labeled issues into review-ready PRs with bounded cost and scoped credentials.

WeightCriterionHow it is measured
25Pass rate on 30 issuesEnd-to-end success (CI green + coverage OK)
20PR qualityDiff size, coverage delta, style conformance
20Cost and latency per resolved issue$ and wall-clock per PR
20SafetyScoped token, per-repo budget, no force-push, credential hygiene
15Operator UXRationale comments, retry affordance, @-mention follow-up
100

Exercises

  1. Add a "fix flaky test" mode: the label @agent stabilize-flake TestX runs the test 50 times in-sandbox and proposes a minimal change that stabilizes it.
  2. Compare cost vs Cursor Background Agents on three shared issues. Report which tools win where.
  3. Implement a budget dashboard: per-repo per-day cost, per-user cost. Alert on anomaly.
  4. Build a "dry-run" mode that opens a draft PR without running CI, so reviewers can examine the plan cheap.
  5. Add a retention policy: PR branches older than 7 days without merge get deleted automatically.

Key Terms

TermWhat people sayWhat it actually means
GitHub App"Scoped bot identity"App with fine-grained permissions + short-lived installation token
Async cloud agent"Background agent"Non-interactive worker that runs in a cloud sandbox, not a terminal
Environment inference"Dockerfile synthesis"Detect language + package manager, generate a Dockerfile if absent
Verification"CI-in-sandbox"Run the full test suite inside the worker before opening a PR
Coverage delta"Coverage preservation"Change in test coverage % from base to agent branch
Per-repo budget"Daily ceiling"Dollar and PR-count cap enforced at the dispatcher
Rationale"PR body explanation"Agent's summary of what changed and why; required in the PR body

Further Reading

Check what you learned

7 questions. Answer each one to see why.

Question 1 of 70 correct

How is the async cloud coding agent triggered in production?

The full code

Every file for this lesson. Copy it, run it, then change it.

main.py
"""GitHub issue-to-PR async cloud agent — dispatcher + budget + safety gates.

The hard architectural primitive is the dispatcher that enforces per-repo
budgets, scoped GitHub App credentials, and a sandbox lifecycle that never
lets the agent force-push or escape the repo scope. This scaffold implements
the dispatcher, budget ledger, sandbox state machine, and verification gates.

Run:  python main.py
"""

from __future__ import annotations

import random
import time
from collections import defaultdict
from dataclasses import asdict, dataclass, field
from enum import Enum, auto


# ---------------------------------------------------------------------------
# webhook -> task enqueue  --  label trigger and queue contract
# ---------------------------------------------------------------------------

@dataclass
class Task:
    task_id: int
    repo: str
    issue_num: int
    title: str
    created_at: float = field(default_factory=time.time)


# ---------------------------------------------------------------------------
# budget ledger  --  per-repo per-day $ and PR-count caps
# ---------------------------------------------------------------------------

@dataclass
class BudgetLedger:
    daily_dollar_cap: float = 50.0
    daily_pr_cap: int = 5
    per_task_dollar_cap: float = 20.0
    spent_today: dict[str, float] = field(default_factory=lambda: defaultdict(float))
    prs_today: dict[str, int] = field(default_factory=lambda: defaultdict(int))

    def permit(self, repo: str, estimated_cost: float) -> tuple[bool, str]:
        if estimated_cost > self.per_task_dollar_cap:
            return False, f"task estimate ${estimated_cost:.2f} > cap ${self.per_task_dollar_cap}"
        # Reserve against the worst-case per-task spend, not the estimate. The
        # agent loop in ``run_agent`` is allowed to run up to ``per_task_dollar_cap``
        # before tripping ``dollar_cap``, so admitting on ``estimated`` lets a
        # burst of cap-hitting runs overrun the daily ceiling. ``record`` still
        # writes the actual spend so unused reservation auto-reconciles.
        worst_case = self.per_task_dollar_cap
        if self.spent_today[repo] + worst_case > self.daily_dollar_cap:
            return False, f"daily $ cap for {repo} would be exceeded"
        if self.prs_today[repo] >= self.daily_pr_cap:
            return False, f"daily PR cap ({self.daily_pr_cap}) for {repo} reached"
        return True, "ok"

    def record(self, repo: str, spent: float, opened_pr: bool) -> None:
        self.spent_today[repo] += spent
        if opened_pr:
            self.prs_today[repo] += 1


# ---------------------------------------------------------------------------
# GitHub App identity  --  short-lived installation token, scoped permissions
# ---------------------------------------------------------------------------

@dataclass
class InstallationToken:
    repo: str
    expires_at: float
    permissions: dict[str, str] = field(default_factory=dict)

    @classmethod
    def mint(cls, repo: str) -> "InstallationToken":
        return cls(repo=repo,
                   expires_at=time.time() + 3600,
                   permissions={"issues": "rw", "pull_requests": "rw",
                                "contents": "rw", "workflows": "r"})

    def can(self, action: str) -> bool:
        # hard policy: never force-push
        if action == "force_push":
            return False
        if action.startswith("write:main"):
            return False
        return True


# ---------------------------------------------------------------------------
# sandbox state machine  --  CLONE -> INFER -> AGENT -> VERIFY -> PR
# ---------------------------------------------------------------------------

class SState(Enum):
    CLONE = auto()
    INFER = auto()
    AGENT = auto()
    VERIFY = auto()
    PR = auto()
    DONE = auto()
    FAILED = auto()


@dataclass
class SandboxRun:
    task: Task
    state: SState = SState.CLONE
    turns: int = 0
    dollars: float = 0.0
    wall_min: float = 0.0
    coverage_delta: float = 0.0
    ci_green: bool = False
    pr_opened: bool = False
    failure: str | None = None
    trace: list[str] = field(default_factory=list)


# ---------------------------------------------------------------------------
# agent loop stub  --  uses per-turn probability weighted by difficulty
# ---------------------------------------------------------------------------

def run_agent(run: SandboxRun, difficulty: float, rng: random.Random,
              turn_cap: int = 20, dollar_cap: float = 20.0,
              minute_cap: float = 30.0) -> None:
    run.state = SState.AGENT
    per_turn_p = max(0.05, 0.35 * (1 - difficulty))
    per_turn_min = 0.9 + difficulty * 0.6
    per_turn_usd = 0.25 + difficulty * 0.45

    while True:
        run.turns += 1
        run.wall_min += per_turn_min
        run.dollars += per_turn_usd
        run.trace.append(f"turn {run.turns}: $={run.dollars:.2f}")

        if run.turns >= turn_cap:
            run.failure = "turn_cap"
            run.state = SState.FAILED
            return
        if run.dollars >= dollar_cap:
            run.failure = "dollar_cap"
            run.state = SState.FAILED
            return
        if run.wall_min >= minute_cap:
            run.failure = "minute_cap"
            run.state = SState.FAILED
            return

        if rng.random() < per_turn_p:
            run.state = SState.VERIFY
            return


def run_verify(run: SandboxRun, difficulty: float, rng: random.Random) -> None:
    flake = rng.random() < 0.05
    if flake:
        run.ci_green = False
        run.failure = "flaky_test"
        run.state = SState.FAILED
        return
    run.ci_green = True
    run.coverage_delta = rng.gauss(0.0, 0.6)
    if run.coverage_delta < -2.0:
        run.failure = "coverage_regression"
        run.state = SState.FAILED
        return
    run.state = SState.PR


def open_pr(run: SandboxRun, token: InstallationToken) -> None:
    # Explicit runtime checks -- never use `assert` for a safety gate. `python -O`
    # strips asserts, which would let a denied or expired token still open a PR.
    if time.time() >= token.expires_at:
        run.failure = "token_expired"
        run.state = SState.FAILED
        return
    if not token.can("pull_request.open"):
        run.failure = "policy_denied"
        run.state = SState.FAILED
        return
    run.pr_opened = True
    run.state = SState.DONE


# ---------------------------------------------------------------------------
# dispatcher  --  pulls tasks, enforces budget, runs the sandbox flow
# ---------------------------------------------------------------------------

def dispatch(task: Task, ledger: BudgetLedger, rng: random.Random) -> SandboxRun:
    difficulty = rng.uniform(0.3, 0.92)
    estimated = 2.0 + difficulty * 8.0
    allowed, reason = ledger.permit(task.repo, estimated)
    if not allowed:
        run = SandboxRun(task)
        run.failure = f"dispatcher: {reason}"
        run.state = SState.FAILED
        return run

    token = InstallationToken.mint(task.repo)
    run = SandboxRun(task)
    run.trace.append("state: CLONE")
    run.state = SState.INFER
    run.trace.append("state: INFER (dockerfile synthesized)")
    run_agent(run, difficulty, rng)
    if run.state == SState.VERIFY:
        run_verify(run, difficulty, rng)
    if run.state == SState.PR:
        open_pr(run, token)
    ledger.record(task.repo, run.dollars, run.pr_opened)
    return run


# ---------------------------------------------------------------------------
# demo  --  run 20 issues across 3 repos; some will hit budget cap
# ---------------------------------------------------------------------------

def main() -> None:
    rng = random.Random(9)
    ledger = BudgetLedger()
    repos = ["acme/widget", "acme/service", "acme/library"]
    runs: list[SandboxRun] = []

    for i in range(20):
        task = Task(task_id=i, repo=rng.choice(repos), issue_num=800 + i,
                    title=f"fix NPE in module {i}")
        run = dispatch(task, ledger, rng)
        runs.append(run)

    opened = sum(1 for r in runs if r.pr_opened)
    failed = sum(1 for r in runs if r.state == SState.FAILED)
    print(f"=== dispatch result ({len(runs)} tasks) ===")
    print(f"PRs opened : {opened}")
    print(f"failed     : {failed}")

    print("\nfailure reasons:")
    reasons = defaultdict(int)
    for r in runs:
        if r.failure:
            reasons[r.failure] += 1
    for reason, n in sorted(reasons.items(), key=lambda x: -x[1]):
        print(f"  {reason:24s} {n}")

    print("\nbudget summary:")
    for repo in repos:
        print(f"  {repo:20s} spent=${ledger.spent_today[repo]:.2f}  "
              f"PRs={ledger.prs_today[repo]}")

    if opened:
        mean_cost = sum(r.dollars for r in runs if r.pr_opened) / opened
        mean_turns = sum(r.turns for r in runs if r.pr_opened) / opened
        print(f"\npass set: mean $/PR = ${mean_cost:.2f}  mean turns = {mean_turns:.1f}")


if __name__ == "__main__":
    main()