Get Started

Self-Hosted Serving Selection, Matching Engine to Hardware and Scale

Engine selection is a function of hardware, scale, and ecosystem, not a leaderboard read. Four engines dominate self-hosted inference in 2026: llama.cpp, Ollama, vLLM, SGLang, with TGI trailing in maintenance mode. llama.cpp is fastest on CPU, widest model support, full control over quantization and threading. Ollama is the dev-laptop one-command install, ~15-30% slower than llama.cpp (Go + CGo + HTTP serialization), 3x throughput gap under prod-like load. TGI entered maintenance mode December 11, 2025only bug fixes, ~10% slower raw throughput than vLLM but historically top observability and HF-ecosystem integration. That maintenance status makes it a risky long-term bet, SGLang or vLLM are safer defaults for new projects. vLLM is the general-purpose production default, v0.15.1 (February 2026) adds PyTorch 2.10, RTX Blackwell SM120, H200 optimization. SGLang is the agentic multi-turn / prefix-heavy specialist, 400,000+ GPUs in production (xAI, LinkedIn, Cursor, Oracle, GCP, Azure, AWS). Hardware constraints: CPU-first → llama.cpp. AMD / non-NVIDIA → vLLM is the strongest-supported path (TRT-LLM is NVIDIA-locked). 2026 pipeline pattern: dev = Ollama, staging = llama.cpp, prod = vLLM or SGLang. The engines take different weight formats, GGUF for the llama.cpp family, HF safetensors for the GPU engines, so a format conversion may sit between stages.

45 minutesLearnPython (stdlib, engine-decision tree walker)Members

Learning Objectives

  • Pick an engine given hardware (CPU / AMD / NVIDIA Hopper / Blackwell), scale (1 user / 100 / 10,000), and workload (general chat / agent / long-context).
  • Name the 2026 TGI maintenance-mode status (December 11, 2025) and why it biases new projects toward vLLM or SGLang.
  • Describe the dev/staging/prod pipeline, including where a GGUF-to-safetensors format conversion sits between stages.
  • Explain why "CPU-first" points to llama.cpp and "AMD" excludes TRT-LLM.