Monitoring Models in Production, Data Drift and Concept Drift
A deployed model fails without an error message. It keeps returning scores while the world moves away from its training data, and only monitoring shows the gap.
110 minutesBuildPythonMembers
Learning Objectives
- Choose monitoring signals at four layers: system health, inputs, predictions, and outcomes
- Distinguish covariate drift, prior drift, and concept drift by the part of the joint distribution that changes
- Implement the Population Stability Index, the two-sample Kolmogorov-Smirnov statistic, and the chi-square test from scratch
- Compare a fixed reference window with sliding current windows, and explain how window size changes noise and delay
- Design an alert policy that limits false alarms with effect sizes, a multiple-testing correction, and a persistence rule
- Decide which kind of drift needs retraining, which needs new labels, and which needs no action