Function Call Dispatcher
The dispatcher is where the harness pays for every promise the schema made. Timeouts, retries, dedupe, error mapping. All on one seam.
Learning Objectives
- Wrap a tool handler in a per-call timeout that returns a typed error instead of hanging the loop.
- Apply exponential backoff retry with jitter and a maximum attempt count.
- Deduplicate retries on an idempotency key so a retry that races with a slow original does not run twice.
- Map handler exceptions and transport faults onto a single error envelope the harness loop already understands.
- Bound parallel dispatch with a concurrency limit so a fan-out of forty tool calls does not exhaust the event loop.
Where the dispatcher sits
Between the harness loop (lesson twenty) and the tool registry (lesson twenty-one). The transport (lesson twenty-two) feeds the loop. The loop hands a tool call to the dispatcher. The dispatcher calls the registry, runs the handler, and returns either a result or a JSON-RPC-shaped error envelope.
The dispatcher is the only layer that knows about timers, retries, and idempotency. The loop does not. The registry does not. The handler does not. That isolation is the point.
Timeouts
Each tool has a default timeout. The registry record carries timeout_ms. The dispatcher overrides it from a per-call override when the harness passes one. We use asyncio.wait_for. On timeout, the handler task is cancelled and the dispatcher returns DispatchError(kind="timeout").
A timeout is not a retryable error by default for non-idempotent tools. A db.write that timed out may or may not have committed. Retrying duplicates the write. The dispatcher honors the idempotent flag from the registry record. Idempotent tools retry. Non-idempotent tools do not.
Retries with exponential backoff
The retry policy is three attempts maximum. Backoff is exponential with jitter.
attempt 1 -> delay 0
attempt 2 -> delay 0.1s * (1 + random[0..0.5])
attempt 3 -> delay 0.4s * (1 + random[0..0.5])Only timeout and transient errors retry. A schema error, a not_foundor an internal error does not retry. Schema errors are deterministic. Retrying does not change the outcome and burns the budget.
The retry loop respects the budget from the harness. If the caller's budget has zero remaining tool calls, the dispatcher fails fast on the first attempt and returns kind="budget_exceeded".
Idempotency key dedupe
A retry that fires while the original is still in flight is a real production bug. The first call hangs at four point nine seconds (just under the timeout). The retry fires at five seconds. Now two requests race against the same backend. If the tool is payments.chargeyou charged twice.
The dispatcher accepts an optional idempotency_key. If the same key is in flight when a call arrives, the dispatcher waits on the in-flight future and returns its result. The cache holds keys for sixty seconds after completion to absorb late retries.
The key is the caller's responsibility. The harness derives it from the planner: f"{step_id}:{tool_name}:{hash(args)}". The dispatcher does not invent keys, because deriving a key from arguments alone makes two semantically-different calls look the same.
Error envelope
A failed dispatch returns a single shape.
DispatchError
kind : "timeout" | "transient" | "schema" | "not_found" | "internal" | "budget_exceeded"
message : str
attempts : int
jsonrpc_code: int (one of -32601, -32602, -32603)The harness loop maps kind to the next state. schema and not_found go to on_error and trigger a replan. timeout and transient go to on_error and may or may not replan depending on attempts. budget_exceeded triggers on_budget_exceeded.
Concurrency limit on fan-out
gather(*calls) runs all coroutines simultaneously. With forty tool calls, that is forty open sockets or forty subprocess pipes. Most backends do not like forty parallel connections from one client.
The dispatcher wraps gather in a semaphore. Default concurrency limit is eight. Each call acquires the semaphore before dispatching and releases on completion. The caller sees gather-shaped output but the actual scheduling is bounded.
Flow for one call
How to read the code
code/main.py defines DispatcherDispatchErrorand TransientError. The dispatcher takes a registry on construction. The async dispatch(name, args, ...) is the only entry point. Per-attempt timeouts are applied inline inside _run_with_retries using asyncio.wait_for. gather_bounded(calls) runs many dispatches with the concurrency limit.
code/tests/test_dispatcher.py covers timeout firing, retry on transient, no-retry on schema error, idempotency dedupe (two concurrent calls with the same key collapse to one handler invocation), and concurrency limiting (the semaphore in action).
The tests use asyncio.sleep(0) and deterministic Counter-based handlers, so they finish in milliseconds and do not depend on wall-clock timing.
Going further
Two extensions production dispatchers add. First, structured logging at every transition (which the loop's event stream already gives you, but the dispatcher should also emit dispatch.attempt and dispatch.retry events). Second, circuit breakers: after N failures in a window, a tool gets a cool-down period where dispatches return immediately with kind="circuit_open" instead of attempting the handler. Both fit on top of this dispatcher without changing the contract.
Lesson twenty-four glues the dispatcher to a plan-and-execute agent so you see all four pieces in motion.
Check what you learned
7 questions. Answer each one to see why.
Why does timeout on a non-idempotent tool default to no retry?
The full code
Every file for this lesson. Copy it, run it, then change it.
"""Function call dispatcher with timeout, retry, idempotency, concurrency limit.
Conceptual references:
- ./docs/en.md (this lesson)
- JSON-RPC 2.0 specification (error envelope shape)
- IETF draft draft-bhutton-json-schema-2020-12 (schema subset reused)
Stdlib only. Run: python3 code/main.py
"""
from __future__ import annotations
import asyncio
import inspect
import json
import random
import re
import time
from dataclasses import dataclass, field
from typing import Any, Awaitable, Callable, Iterable
ERR_METHOD_NOT_FOUND = -32601
ERR_INVALID_PARAMS = -32602
ERR_INTERNAL = -32603
class TransientError(Exception):
"""Raised by a handler to indicate the failure is worth retrying."""
class _DispatchedError(Exception):
"""Internal sentinel that wraps a DispatchError so dedup followers preserve kind."""
def __init__(self, error: "DispatchError") -> None:
super().__init__(error.message)
self.error = error
@dataclass
class DispatchError(Exception):
kind: str
message: str
attempts: int
jsonrpc_code: int = ERR_INTERNAL
def __post_init__(self) -> None:
super().__init__(f"{self.kind}: {self.message}")
def to_envelope(self) -> dict:
return {
"code": self.jsonrpc_code,
"message": self.message,
"data": {"kind": self.kind, "attempts": self.attempts},
}
@dataclass
class DispatchOk:
result: Any
attempts: int
@dataclass
class _ToolRecord:
name: str
schema: dict
handler: Callable[..., Any]
idempotent: bool = False
timeout_ms: int = 30_000
class MiniRegistry:
"""A trimmed registry: name, schema, handler, idempotent, timeout."""
def __init__(self) -> None:
self._recs: dict[str, _ToolRecord] = {}
def register(
self, name: str, schema: dict, handler: Callable[..., Any],
*, idempotent: bool = False, timeout_ms: int = 30_000,
) -> None:
self._recs[name] = _ToolRecord(
name=name, schema=schema, handler=handler,
idempotent=idempotent, timeout_ms=timeout_ms,
)
def get(self, name: str) -> _ToolRecord:
if name not in self._recs:
raise KeyError(name)
return self._recs[name]
def validate(self, name: str, args: Any) -> list[str]:
rec = self.get(name)
errs: list[str] = []
_walk(rec.schema, args, "", errs)
return errs
def _walk(schema: dict, value: Any, path: str, errs: list[str]) -> None:
t = schema.get("type")
type_ok = True
if t == "object" and not isinstance(value, dict):
type_ok = False
elif t == "integer" and (isinstance(value, bool) or not isinstance(value, int)):
type_ok = False
elif t == "string" and not isinstance(value, str):
type_ok = False
elif t == "array" and not isinstance(value, list):
type_ok = False
if not type_ok:
errs.append(f"{path or '/'}: expected {t}, got {type(value).__name__}")
return
if t == "object":
for req in schema.get("required", []):
if req not in value:
errs.append(f"{path}/{req}: required")
for k, sub in schema.get("properties", {}).items():
if k in value:
_walk(sub, value[k], f"{path}/{k}", errs)
@dataclass
class _InFlight:
future: asyncio.Future
started_at: float
class Dispatcher:
"""Per-call timeout, retry, idempotency, concurrency limit."""
def __init__(
self,
registry: MiniRegistry,
*,
max_attempts: int = 3,
concurrency: int = 8,
cache_ttl_seconds: float = 60.0,
sleep: Callable[[float], Awaitable[None]] | None = None,
) -> None:
if max_attempts <= 0:
raise ValueError("max_attempts must be > 0")
self.registry = registry
self.max_attempts = max_attempts
self._sem = asyncio.Semaphore(concurrency)
self._inflight: dict[str, _InFlight] = {}
self._cache: dict[str, tuple[Any, float]] = {}
self._cache_ttl = cache_ttl_seconds
self._sleep = sleep or asyncio.sleep
async def dispatch(
self,
name: str,
args: dict,
*,
timeout_ms_override: int | None = None,
idempotency_key: str | None = None,
budget_tool_calls_remaining: int | None = None,
) -> DispatchOk | DispatchError:
try:
rec = self.registry.get(name)
except KeyError:
return DispatchError(
kind="not_found", message=f"tool {name!r}", attempts=0,
jsonrpc_code=ERR_METHOD_NOT_FOUND,
)
errs = self.registry.validate(name, args)
if errs:
return DispatchError(
kind="schema", message="; ".join(errs), attempts=0,
jsonrpc_code=ERR_INVALID_PARAMS,
)
if budget_tool_calls_remaining is not None and budget_tool_calls_remaining <= 0:
return DispatchError(
kind="budget_exceeded", message="tool_calls remaining is 0", attempts=0,
)
if idempotency_key is not None:
now = time.monotonic()
cached = self._cache.get(idempotency_key)
if cached is not None and now - cached[1] < self._cache_ttl:
return DispatchOk(result=cached[0], attempts=0)
inflight = self._inflight.get(idempotency_key)
if inflight is not None:
try:
res = await inflight.future
return DispatchOk(result=res, attempts=0)
except Exception as exc:
return _map_exception(exc, attempts=0)
async with self._sem:
return await self._run_with_retries(rec, args, timeout_ms_override, idempotency_key)
async def _run_with_retries(
self,
rec: _ToolRecord,
args: dict,
timeout_override: int | None,
idempotency_key: str | None,
) -> DispatchOk | DispatchError:
timeout_ms = timeout_override if timeout_override is not None else rec.timeout_ms
timeout_s = timeout_ms / 1000.0
attempt = 0
last_error: DispatchError | None = None
loop = asyncio.get_running_loop()
future: asyncio.Future | None = None
if idempotency_key is not None:
future = loop.create_future()
self._inflight[idempotency_key] = _InFlight(future=future, started_at=time.monotonic())
try:
while attempt < self.max_attempts:
attempt += 1
try:
coro = _invoke(rec.handler, args)
result = await asyncio.wait_for(coro, timeout=timeout_s)
except asyncio.TimeoutError:
last_error = DispatchError(
kind="timeout",
message=f"timeout after {timeout_ms}ms",
attempts=attempt,
)
if not rec.idempotent:
break
if attempt >= self.max_attempts:
break
await self._sleep(_backoff(attempt))
continue
except TransientError as exc:
last_error = DispatchError(
kind="transient", message=str(exc), attempts=attempt,
)
if attempt >= self.max_attempts:
break
await self._sleep(_backoff(attempt))
continue
except Exception as exc:
err = _map_exception(exc, attempts=attempt)
if future is not None and not future.done():
future.set_exception(_DispatchedError(err))
return err
if future is not None and not future.done():
future.set_result(result)
if idempotency_key is not None:
self._cache[idempotency_key] = (result, time.monotonic())
return DispatchOk(result=result, attempts=attempt)
assert last_error is not None
if future is not None and not future.done():
future.set_exception(_DispatchedError(last_error))
return last_error
finally:
if idempotency_key is not None:
self._inflight.pop(idempotency_key, None)
async def gather_bounded(self, calls: Iterable[tuple[str, dict]]) -> list[DispatchOk | DispatchError]:
return await asyncio.gather(*(self.dispatch(n, a) for n, a in calls))
async def _invoke(handler: Callable[..., Any], args: dict) -> Any:
if inspect.iscoroutinefunction(handler):
return await handler(**args)
result = handler(**args)
if inspect.isawaitable(result):
return await result
return result
def _backoff(attempt: int) -> float:
base = 0.1 * (4 ** (attempt - 1))
return base * (1 + random.random() * 0.5)
def _map_exception(exc: Exception, attempts: int) -> DispatchError:
if isinstance(exc, _DispatchedError):
original = exc.error
return DispatchError(
kind=original.kind,
message=original.message,
attempts=original.attempts,
jsonrpc_code=original.jsonrpc_code,
)
return DispatchError(
kind="internal",
message=f"{type(exc).__name__}: {exc}",
attempts=attempts,
)
async def _demo() -> None:
reg = MiniRegistry()
counter = {"a": 0, "b": 0}
async def flaky_fetch(id: int) -> dict:
counter["a"] += 1
if counter["a"] < 2:
raise TransientError("upstream not ready")
return {"id": id, "name": "ada"}
async def slow(n: int) -> int:
counter["b"] += 1
await asyncio.sleep(0.05)
return n
reg.register(
"fetch_user",
schema={"type": "object", "required": ["id"], "properties": {"id": {"type": "integer"}}},
handler=flaky_fetch, idempotent=True, timeout_ms=200,
)
reg.register(
"slow",
schema={"type": "object", "required": ["n"], "properties": {"n": {"type": "integer"}}},
handler=slow, idempotent=True, timeout_ms=10,
)
reg.register(
"noop",
schema={"type": "object", "properties": {}},
handler=lambda: "ok",
)
disp = Dispatcher(reg, max_attempts=3, concurrency=4)
out_retry = await disp.dispatch("fetch_user", {"id": 42})
out_timeout = await disp.dispatch("slow", {"n": 1})
out_schema = await disp.dispatch("fetch_user", {"id": "x"})
out_missing = await disp.dispatch("does_not_exist", {})
out_ok = await disp.dispatch("noop", {})
a, b = await asyncio.gather(
disp.dispatch("noop", {}, idempotency_key="k1"),
disp.dispatch("noop", {}, idempotency_key="k1"),
)
report = {
"retry_then_success": {
"attempts": getattr(out_retry, "attempts", None),
"ok": isinstance(out_retry, DispatchOk),
},
"timeout": {"kind": getattr(out_timeout, "kind", None)},
"schema": {"kind": getattr(out_schema, "kind", None)},
"missing": {"kind": getattr(out_missing, "kind", None)},
"happy": {"result": getattr(out_ok, "result", None)},
"idempotency_pair": [isinstance(a, DispatchOk), isinstance(b, DispatchOk)],
}
print(json.dumps(report, indent=2))
if __name__ == "__main__":
asyncio.run(_demo())