Batch APIs, the 50% Discount as Industry Standard
Every major provider ships an async batch API with a 50% discount and ~24-hour turnaround. OpenAI, Anthropic, Google, and most of the inference platforms (Fireworks batch tier, Together batch) implement the same pattern. Stack batch with prompt caching and overnight pipelines drop to ~10% of synchronous-uncached cost. The rule is brutally simple: if it is not interactive, it belongs on batch. Content generation pipelines, document classification, data extraction, report generation, bulk labeling, catalog tagging, anything tolerant of 24-hour latency is money left on the table until it moves to batch. The 2026 production pattern is to triage every new LLM workload into three lanes: interactive (synchronous with caching), semi-interactive (async queue with fallback), batch (overnight, cached input stacked). Workloads that pretend to be interactive but tolerate minutes of latency waste most.
Learning Objectives
- Name the three provider batch APIs (OpenAI, Anthropic, Google) and the common 50% discount + 24h turnaround guarantees.
- Compute the cost for stacking batch + cached-input on an overnight classification workload and compare to synchronous-uncached baseline.
- Triage a workload into interactive / semi-interactive / batch and justify the lane.
- Name the two traps: partial interactivity (user expects faster than 24h) and output-schema drift (batch file format differs per provider).
The Problem
Your team ships a nightly report generation pipeline. 50,000 documents, summarize each, cluster the summaries, draft an executive brief. Running synchronously it takes 4 hours at $2,000/night. You hear about batch APIs.
The batch gets you 50% off. You also enable prompt caching on the system prompt (shared across all 50k calls). Stacked, the bill drops to $180/night, ~9% of baseline. Same pipeline, three config changes.
Batch is the cheapest lever in the LLM cost toolkit that nobody pulls. The reason is mostly organizational: teams think "real-time" when the SLA actually is "by morning." This lesson is about not leaving 90% of the bill on the table.
The Concept
The three batch APIs
OpenAI Batch API: JSONL file upload with a list of requests. Promised 24-hour turnaround (usually ~2-8 hours in practice). 50% discount on input and output tokens. /v1/batches endpoint. Cache-eligible inputs also get cached-input pricing on top.
Anthropic Message Batches: JSONL upload. 24-hour turnaround. 50% discount. Supports cache_controlcache writes are explicit, reads happen automatically within the batch.
Google Vertex AI Batch Prediction: BigQuery or GCS input. Similar 50% discount for Gemini. Integrates with Vertex pipelines.
Semantic: asynchronous, not slow
Batch is "I promise to return within 24 hours", not "this will take 24 hours." Typical P50 is 2-6 hours. Provider schedules your batch during off-peak windows when GPU inventory is underutilized.
Stack with caching
A 50k-document summarization with the same 4K-token system prompt:
- Synchronous uncached: 50000 × ($input × 4000 + $output × 200) at full rates.
- Synchronous cached: system prompt cached after first write; remaining 49999 get 10x cheaper input.
- Batch cached: all of the above plus 50% discount on both read and write.
The stack: batch + cache = ~10% of sync uncached bill. Any workload that runs overnight and has a shared system prompt should use this.
Workload triage
Interactiveuser waits for the response. TTFT matters. Synchronous call with prompt caching. Cannot batch.
Semi-interactiveuser submits a task, checks back in minutes. Async queue with fallback to sync if batch not available. Think moderate-volume RAG indexing.
Batchuser expects results "by morning" or "next hour." Content pipelines, classification at scale, offline analysis. Always batch, always stack caching.
Common mistake: classifying everything as interactive because the pipeline is production. Production is not a latency spec, SLA is.
The partial-interactivity trap
Some features look interactive but tolerate 5-10 minutes. Example: a nightly customer health report with "refresh" button. User clicks refresh; wait 10 minutes is fine. Team ships it as synchronous. 50 concurrent refreshes cost 10x what batched-and-delivered-via-email would cost.
The question to ask: "What does 24-hour mean for this user?" If the answer is "they wouldn't notice," batch it.
The output-schema trap
Batch file formats differ per provider:
- OpenAI: JSONL, one request per line.
- Anthropic: JSONL, one message per line; response format embedded.
- Vertex: BigQuery table or GCS prefix with TFRecord.
Writing "one batch client" across providers means adapter code per provider. Gateways that advertise multi-provider batch (Portkey, LiteLLM some tiers) still thin-wrap the raw format.
Numbers you should remember
- Batch discount across providers: 50% flat on input + output.
- Turnaround SLA: 24 hours guaranteed, 2-6 hours typical P50.
- Stacked batch + cached input: ~10% of sync uncached cost.
- Workload triage rule: if 24h latency acceptable, always batch.
Use It
code/main.py computes costs across sync, sync+cache, batch, and batch+cache for a 50k-document workload. Reports savings in $ and percent.
Ship It
This lesson produces outputs/skill-batch-triager.md. Given workload characteristics, triages into interactive/semi/batch and estimates savings.
Exercises
- Run
code/main.py. For a 100k-doc pipeline with 3K-token system prompt and 500-token output, compute the savings of full stack (batch + cache) vs sync baseline. - Pick three features in a real product you know. Triage each into interactive/semi/batch.
- A user complains their report took 3 hours. Was that a batch mis-triage or a legitimate interactive? Write the decision criterion.
- Your batch API return SLA is 24h but P99 is 20 hours. How do you communicate this to the user, what is the downstream system behavior on the edge case?
- Compute break-even: at what shared-prefix length does batch + cache become cheaper than running overnight on your own reserved GPU?
Key Terms
| Term | What people say | What it actually means |
|---|---|---|
| Batch API | "async discount" | 50% off with 24h turnaround |
| JSONL | "batch format" | One JSON request per line; OpenAI/Anthropic standard |
| Message Batches | "Anthropic batch" | Anthropic's batch API product name |
| Batch prediction | "Vertex batch" | Vertex AI's batch API product |
| Turnaround SLA | "24h promise" | Guarantee, not typical; typical is 2-6h |
| Workload triage | "interactivity decision" | Interactive / semi / batch routing decision |
| Output schema | "response format" | Per-provider JSONL layout; not portable |
| Stacked discount | "batch + cache" | ~10% of uncached sync bill when both apply |
Further Reading
- OpenAI Batch APIJSONL format and
/v1/batchessemantics. - Anthropic Message Batchesbatch format and
cache_controlinteraction. - Vertex AI Batch PredictionGemini batch semantics.
- Finout, OpenAI vs Anthropic API Pricing 2026
- Zen Van Riel, LLM API Cost Comparison 2026
Check what you learned
6 questions. Answer each one to see why.
What is the common batch-API offer across OpenAI, Anthropic, and Google in 2026?
The full code
Every file for this lesson. Copy it, run it, then change it.
"""Batch vs synchronous cost simulator — stdlib Python.
Models a 50k-document pipeline across four configurations:
SYNC : no discount, no cache
SYNC + CACHE : system prompt cached after first call
BATCH : 50% discount, no cache
BATCH + CACHE : stacked (~10% of SYNC bill)
"""
from __future__ import annotations
BASE_INPUT = 3.00
BASE_OUTPUT = 15.00
CACHED_INPUT = 0.30
CACHE_WRITE_5MIN = 1.25 * BASE_INPUT
BATCH_DISCOUNT = 0.50
def cost_sync(docs: int, prefix_tokens: int, per_doc_tokens: int, out_tokens: int) -> float:
cost = 0.0
for _ in range(docs):
cost += (prefix_tokens / 1e6) * BASE_INPUT
cost += (per_doc_tokens / 1e6) * BASE_INPUT
cost += (out_tokens / 1e6) * BASE_OUTPUT
return cost
def cost_sync_cache(docs: int, prefix_tokens: int, per_doc_tokens: int, out_tokens: int) -> float:
cost = (prefix_tokens / 1e6) * CACHE_WRITE_5MIN
for i in range(docs):
if i > 0:
cost += (prefix_tokens / 1e6) * CACHED_INPUT
cost += (per_doc_tokens / 1e6) * BASE_INPUT
cost += (out_tokens / 1e6) * BASE_OUTPUT
return cost
def cost_batch(docs: int, prefix_tokens: int, per_doc_tokens: int, out_tokens: int) -> float:
return cost_sync(docs, prefix_tokens, per_doc_tokens, out_tokens) * BATCH_DISCOUNT
def cost_batch_cache(docs: int, prefix_tokens: int, per_doc_tokens: int, out_tokens: int) -> float:
return cost_sync_cache(docs, prefix_tokens, per_doc_tokens, out_tokens) * BATCH_DISCOUNT
def run(label: str, docs: int, prefix: int, per_doc: int, output: int) -> None:
sc = cost_sync(docs, prefix, per_doc, output)
scc = cost_sync_cache(docs, prefix, per_doc, output)
bc = cost_batch(docs, prefix, per_doc, output)
bcc = cost_batch_cache(docs, prefix, per_doc, output)
print(f"\n{label}")
print(f" docs={docs}, prefix={prefix}, per_doc={per_doc}, output={output}")
print(f" SYNC : ${sc:10.2f} (baseline)")
print(f" SYNC + CACHE : ${scc:10.2f} ({scc/sc*100:5.1f}% of baseline)")
print(f" BATCH : ${bc:10.2f} ({bc/sc*100:5.1f}% of baseline)")
print(f" BATCH + CACHE : ${bcc:10.2f} ({bcc/sc*100:5.1f}% of baseline)")
def main() -> None:
print("=" * 80)
print("BATCH API ECONOMICS — stack batch with prompt caching for ~10% of sync bill")
print("=" * 80)
run("Nightly doc summarization (50k docs)",
docs=50_000, prefix=4000, per_doc=2000, output=200)
run("Content classification (200k items, short per item)",
docs=200_000, prefix=1500, per_doc=300, output=50)
run("Large report draft (small N, heavy per item)",
docs=1_000, prefix=6000, per_doc=15_000, output=2000)
if __name__ == "__main__":
main()
Most people stop here. Don't be most people.
The next 14 lessons are where Phase 17 turns from understanding into ability: the harder builds, the production patterns and the quizzes that prove you've actually got it. Unlock the full AI Expert Masterclass: all 542 expert lessons, every line of code, every phase.
- 16Model Routing as a Cost-Reduction Primitive
- 17Disaggregated Prefill/Decode, NVIDIA Dynamo and llm-d
- 18Production Serving Stack, KV Offloading and Cache-Aware Routing
- 19AI Gateways, LiteLLM, Portkey, Kong AI Gateway, Bifrost
- + 10 more in this phase, and 542 across the whole masterclass