Get Started

A/B Testing LLM Features, GrowthBook, Statsig, and the Vibes Problem

Traditional A/B testing was not built for non-deterministic LLMs. The critical distinction: evals answer "can the model do the job?" A/B tests answer "do users care?" Both are required; shipping on vibe checks is over. What to test in 2026: prompt engineering (wording), model selection (GPT-4 vs GPT-3.5 vs OSS; accuracy vs cost vs latency), generation parameters (temperature, top-p). Real cases: a chatbot reward-model variant delivered +70% conversation length and +30% retention; Nextdoor AI subject-line experiments delivered +1% CTR after reward-function refinement; Khan Academy Khanmigo iterated on a latency-vs-math-accuracy axis. Platform split: Statsig (acquired by OpenAI for $1.1B in September 2025), sequential testing, CUPED, all-in-one. GrowthBookopen-source, warehouse-native, Bayesian + Frequentist + Sequential engines, CUPED, SRM checks, Benjamini-Hochberg + Bonferroni corrections. You pick based on warehouse-SQL preference and whether "acquired by OpenAI" matters to your organization.

60 minutesLearnPython (stdlib, toy sequential test simulator)Members

Learning Objectives

  • Distinguish evals ("can the model do the job") from A/B tests ("do users care").
  • Enumerate three testable axes (prompt, model, parameters) and pick the metric for each.
  • Explain CUPED, sequential testing, and Benjamini-Hochberg multiple-comparison corrections.
  • Pick Statsig or GrowthBook based on warehouse-SQL posture and corporate acquisition stance.