Get Started

Function Call Dispatcher

The dispatcher is where the harness pays for every promise the schema made. Timeouts, retries, dedupe, error mapping. All on one seam.

90 minutesBuildPythonFree lesson

Learning Objectives

  • Wrap a tool handler in a per-call timeout that returns a typed error instead of hanging the loop.
  • Apply exponential backoff retry with jitter and a maximum attempt count.
  • Deduplicate retries on an idempotency key so a retry that races with a slow original does not run twice.
  • Map handler exceptions and transport faults onto a single error envelope the harness loop already understands.
  • Bound parallel dispatch with a concurrency limit so a fan-out of forty tool calls does not exhaust the event loop.

Where the dispatcher sits

Between the harness loop (lesson twenty) and the tool registry (lesson twenty-one). The transport (lesson twenty-two) feeds the loop. The loop hands a tool call to the dispatcher. The dispatcher calls the registry, runs the handler, and returns either a result or a JSON-RPC-shaped error envelope.

The dispatcher is the only layer that knows about timers, retries, and idempotency. The loop does not. The registry does not. The handler does not. That isolation is the point.

Timeouts

Each tool has a default timeout. The registry record carries timeout_ms. The dispatcher overrides it from a per-call override when the harness passes one. We use asyncio.wait_for. On timeout, the handler task is cancelled and the dispatcher returns DispatchError(kind="timeout").

A timeout is not a retryable error by default for non-idempotent tools. A db.write that timed out may or may not have committed. Retrying duplicates the write. The dispatcher honors the idempotent flag from the registry record. Idempotent tools retry. Non-idempotent tools do not.

Retries with exponential backoff

The retry policy is three attempts maximum. Backoff is exponential with jitter.

Text
attempt 1  -> delay 0
attempt 2  -> delay 0.1s * (1 + random[0..0.5])
attempt 3  -> delay 0.4s * (1 + random[0..0.5])

Only timeout and transient errors retry. A schema error, a not_foundor an internal error does not retry. Schema errors are deterministic. Retrying does not change the outcome and burns the budget.

The retry loop respects the budget from the harness. If the caller's budget has zero remaining tool calls, the dispatcher fails fast on the first attempt and returns kind="budget_exceeded".

Idempotency key dedupe

A retry that fires while the original is still in flight is a real production bug. The first call hangs at four point nine seconds (just under the timeout). The retry fires at five seconds. Now two requests race against the same backend. If the tool is payments.chargeyou charged twice.

The dispatcher accepts an optional idempotency_key. If the same key is in flight when a call arrives, the dispatcher waits on the in-flight future and returns its result. The cache holds keys for sixty seconds after completion to absorb late retries.

The key is the caller's responsibility. The harness derives it from the planner: f"{step_id}:{tool_name}:{hash(args)}". The dispatcher does not invent keys, because deriving a key from arguments alone makes two semantically-different calls look the same.

Error envelope

A failed dispatch returns a single shape.

Text
DispatchError
  kind        : "timeout" | "transient" | "schema" | "not_found" | "internal" | "budget_exceeded"
  message     : str
  attempts    : int
  jsonrpc_code: int   (one of -32601, -32602, -32603)

The harness loop maps kind to the next state. schema and not_found go to on_error and trigger a replan. timeout and transient go to on_error and may or may not replan depending on attempts. budget_exceeded triggers on_budget_exceeded.

Concurrency limit on fan-out

gather(*calls) runs all coroutines simultaneously. With forty tool calls, that is forty open sockets or forty subprocess pipes. Most backends do not like forty parallel connections from one client.

The dispatcher wraps gather in a semaphore. Default concurrency limit is eight. Each call acquires the semaphore before dispatching and releases on completion. The caller sees gather-shaped output but the actual scheduling is bounded.

Flow for one call

How to read the code

code/main.py defines DispatcherDispatchErrorand TransientError. The dispatcher takes a registry on construction. The async dispatch(name, args, ...) is the only entry point. Per-attempt timeouts are applied inline inside _run_with_retries using asyncio.wait_for. gather_bounded(calls) runs many dispatches with the concurrency limit.

code/tests/test_dispatcher.py covers timeout firing, retry on transient, no-retry on schema error, idempotency dedupe (two concurrent calls with the same key collapse to one handler invocation), and concurrency limiting (the semaphore in action).

The tests use asyncio.sleep(0) and deterministic Counter-based handlers, so they finish in milliseconds and do not depend on wall-clock timing.

Going further

Two extensions production dispatchers add. First, structured logging at every transition (which the loop's event stream already gives you, but the dispatcher should also emit dispatch.attempt and dispatch.retry events). Second, circuit breakers: after N failures in a window, a tool gets a cool-down period where dispatches return immediately with kind="circuit_open" instead of attempting the handler. Both fit on top of this dispatcher without changing the contract.

Lesson twenty-four glues the dispatcher to a plan-and-execute agent so you see all four pieces in motion.

Check what you learned

7 questions. Answer each one to see why.

Question 1 of 70 correct

Why does timeout on a non-idempotent tool default to no retry?

The full code

Every file for this lesson. Copy it, run it, then change it.

main.py
"""Function call dispatcher with timeout, retry, idempotency, concurrency limit.

Conceptual references:
- ./docs/en.md (this lesson)
- JSON-RPC 2.0 specification (error envelope shape)
- IETF draft draft-bhutton-json-schema-2020-12 (schema subset reused)

Stdlib only. Run: python3 code/main.py
"""

from __future__ import annotations

import asyncio
import inspect
import json
import random
import re
import time
from dataclasses import dataclass, field
from typing import Any, Awaitable, Callable, Iterable


ERR_METHOD_NOT_FOUND = -32601
ERR_INVALID_PARAMS = -32602
ERR_INTERNAL = -32603


class TransientError(Exception):
    """Raised by a handler to indicate the failure is worth retrying."""


class _DispatchedError(Exception):
    """Internal sentinel that wraps a DispatchError so dedup followers preserve kind."""

    def __init__(self, error: "DispatchError") -> None:
        super().__init__(error.message)
        self.error = error


@dataclass
class DispatchError(Exception):
    kind: str
    message: str
    attempts: int
    jsonrpc_code: int = ERR_INTERNAL

    def __post_init__(self) -> None:
        super().__init__(f"{self.kind}: {self.message}")

    def to_envelope(self) -> dict:
        return {
            "code": self.jsonrpc_code,
            "message": self.message,
            "data": {"kind": self.kind, "attempts": self.attempts},
        }


@dataclass
class DispatchOk:
    result: Any
    attempts: int


@dataclass
class _ToolRecord:
    name: str
    schema: dict
    handler: Callable[..., Any]
    idempotent: bool = False
    timeout_ms: int = 30_000


class MiniRegistry:
    """A trimmed registry: name, schema, handler, idempotent, timeout."""

    def __init__(self) -> None:
        self._recs: dict[str, _ToolRecord] = {}

    def register(
        self, name: str, schema: dict, handler: Callable[..., Any],
        *, idempotent: bool = False, timeout_ms: int = 30_000,
    ) -> None:
        self._recs[name] = _ToolRecord(
            name=name, schema=schema, handler=handler,
            idempotent=idempotent, timeout_ms=timeout_ms,
        )

    def get(self, name: str) -> _ToolRecord:
        if name not in self._recs:
            raise KeyError(name)
        return self._recs[name]

    def validate(self, name: str, args: Any) -> list[str]:
        rec = self.get(name)
        errs: list[str] = []
        _walk(rec.schema, args, "", errs)
        return errs


def _walk(schema: dict, value: Any, path: str, errs: list[str]) -> None:
    t = schema.get("type")
    type_ok = True
    if t == "object" and not isinstance(value, dict):
        type_ok = False
    elif t == "integer" and (isinstance(value, bool) or not isinstance(value, int)):
        type_ok = False
    elif t == "string" and not isinstance(value, str):
        type_ok = False
    elif t == "array" and not isinstance(value, list):
        type_ok = False
    if not type_ok:
        errs.append(f"{path or '/'}: expected {t}, got {type(value).__name__}")
        return
    if t == "object":
        for req in schema.get("required", []):
            if req not in value:
                errs.append(f"{path}/{req}: required")
        for k, sub in schema.get("properties", {}).items():
            if k in value:
                _walk(sub, value[k], f"{path}/{k}", errs)


@dataclass
class _InFlight:
    future: asyncio.Future
    started_at: float


class Dispatcher:
    """Per-call timeout, retry, idempotency, concurrency limit."""

    def __init__(
        self,
        registry: MiniRegistry,
        *,
        max_attempts: int = 3,
        concurrency: int = 8,
        cache_ttl_seconds: float = 60.0,
        sleep: Callable[[float], Awaitable[None]] | None = None,
    ) -> None:
        if max_attempts <= 0:
            raise ValueError("max_attempts must be > 0")
        self.registry = registry
        self.max_attempts = max_attempts
        self._sem = asyncio.Semaphore(concurrency)
        self._inflight: dict[str, _InFlight] = {}
        self._cache: dict[str, tuple[Any, float]] = {}
        self._cache_ttl = cache_ttl_seconds
        self._sleep = sleep or asyncio.sleep

    async def dispatch(
        self,
        name: str,
        args: dict,
        *,
        timeout_ms_override: int | None = None,
        idempotency_key: str | None = None,
        budget_tool_calls_remaining: int | None = None,
    ) -> DispatchOk | DispatchError:
        try:
            rec = self.registry.get(name)
        except KeyError:
            return DispatchError(
                kind="not_found", message=f"tool {name!r}", attempts=0,
                jsonrpc_code=ERR_METHOD_NOT_FOUND,
            )

        errs = self.registry.validate(name, args)
        if errs:
            return DispatchError(
                kind="schema", message="; ".join(errs), attempts=0,
                jsonrpc_code=ERR_INVALID_PARAMS,
            )

        if budget_tool_calls_remaining is not None and budget_tool_calls_remaining <= 0:
            return DispatchError(
                kind="budget_exceeded", message="tool_calls remaining is 0", attempts=0,
            )

        if idempotency_key is not None:
            now = time.monotonic()
            cached = self._cache.get(idempotency_key)
            if cached is not None and now - cached[1] < self._cache_ttl:
                return DispatchOk(result=cached[0], attempts=0)
            inflight = self._inflight.get(idempotency_key)
            if inflight is not None:
                try:
                    res = await inflight.future
                    return DispatchOk(result=res, attempts=0)
                except Exception as exc:
                    return _map_exception(exc, attempts=0)

        async with self._sem:
            return await self._run_with_retries(rec, args, timeout_ms_override, idempotency_key)

    async def _run_with_retries(
        self,
        rec: _ToolRecord,
        args: dict,
        timeout_override: int | None,
        idempotency_key: str | None,
    ) -> DispatchOk | DispatchError:
        timeout_ms = timeout_override if timeout_override is not None else rec.timeout_ms
        timeout_s = timeout_ms / 1000.0
        attempt = 0
        last_error: DispatchError | None = None

        loop = asyncio.get_running_loop()
        future: asyncio.Future | None = None
        if idempotency_key is not None:
            future = loop.create_future()
            self._inflight[idempotency_key] = _InFlight(future=future, started_at=time.monotonic())

        try:
            while attempt < self.max_attempts:
                attempt += 1
                try:
                    coro = _invoke(rec.handler, args)
                    result = await asyncio.wait_for(coro, timeout=timeout_s)
                except asyncio.TimeoutError:
                    last_error = DispatchError(
                        kind="timeout",
                        message=f"timeout after {timeout_ms}ms",
                        attempts=attempt,
                    )
                    if not rec.idempotent:
                        break
                    if attempt >= self.max_attempts:
                        break
                    await self._sleep(_backoff(attempt))
                    continue
                except TransientError as exc:
                    last_error = DispatchError(
                        kind="transient", message=str(exc), attempts=attempt,
                    )
                    if attempt >= self.max_attempts:
                        break
                    await self._sleep(_backoff(attempt))
                    continue
                except Exception as exc:
                    err = _map_exception(exc, attempts=attempt)
                    if future is not None and not future.done():
                        future.set_exception(_DispatchedError(err))
                    return err
                if future is not None and not future.done():
                    future.set_result(result)
                if idempotency_key is not None:
                    self._cache[idempotency_key] = (result, time.monotonic())
                return DispatchOk(result=result, attempts=attempt)

            assert last_error is not None
            if future is not None and not future.done():
                future.set_exception(_DispatchedError(last_error))
            return last_error
        finally:
            if idempotency_key is not None:
                self._inflight.pop(idempotency_key, None)

    async def gather_bounded(self, calls: Iterable[tuple[str, dict]]) -> list[DispatchOk | DispatchError]:
        return await asyncio.gather(*(self.dispatch(n, a) for n, a in calls))


async def _invoke(handler: Callable[..., Any], args: dict) -> Any:
    if inspect.iscoroutinefunction(handler):
        return await handler(**args)
    result = handler(**args)
    if inspect.isawaitable(result):
        return await result
    return result


def _backoff(attempt: int) -> float:
    base = 0.1 * (4 ** (attempt - 1))
    return base * (1 + random.random() * 0.5)


def _map_exception(exc: Exception, attempts: int) -> DispatchError:
    if isinstance(exc, _DispatchedError):
        original = exc.error
        return DispatchError(
            kind=original.kind,
            message=original.message,
            attempts=original.attempts,
            jsonrpc_code=original.jsonrpc_code,
        )
    return DispatchError(
        kind="internal",
        message=f"{type(exc).__name__}: {exc}",
        attempts=attempts,
    )


async def _demo() -> None:
    reg = MiniRegistry()

    counter = {"a": 0, "b": 0}

    async def flaky_fetch(id: int) -> dict:
        counter["a"] += 1
        if counter["a"] < 2:
            raise TransientError("upstream not ready")
        return {"id": id, "name": "ada"}

    async def slow(n: int) -> int:
        counter["b"] += 1
        await asyncio.sleep(0.05)
        return n

    reg.register(
        "fetch_user",
        schema={"type": "object", "required": ["id"], "properties": {"id": {"type": "integer"}}},
        handler=flaky_fetch, idempotent=True, timeout_ms=200,
    )
    reg.register(
        "slow",
        schema={"type": "object", "required": ["n"], "properties": {"n": {"type": "integer"}}},
        handler=slow, idempotent=True, timeout_ms=10,
    )
    reg.register(
        "noop",
        schema={"type": "object", "properties": {}},
        handler=lambda: "ok",
    )

    disp = Dispatcher(reg, max_attempts=3, concurrency=4)

    out_retry = await disp.dispatch("fetch_user", {"id": 42})
    out_timeout = await disp.dispatch("slow", {"n": 1})
    out_schema = await disp.dispatch("fetch_user", {"id": "x"})
    out_missing = await disp.dispatch("does_not_exist", {})
    out_ok = await disp.dispatch("noop", {})

    a, b = await asyncio.gather(
        disp.dispatch("noop", {}, idempotency_key="k1"),
        disp.dispatch("noop", {}, idempotency_key="k1"),
    )
    report = {
        "retry_then_success": {
            "attempts": getattr(out_retry, "attempts", None),
            "ok": isinstance(out_retry, DispatchOk),
        },
        "timeout": {"kind": getattr(out_timeout, "kind", None)},
        "schema": {"kind": getattr(out_schema, "kind", None)},
        "missing": {"kind": getattr(out_missing, "kind", None)},
        "happy": {"result": getattr(out_ok, "result", None)},
        "idempotency_pair": [isinstance(a, DispatchOk), isinstance(b, DispatchOk)],
    }
    print(json.dumps(report, indent=2))


if __name__ == "__main__":
    asyncio.run(_demo())