Get Started

Monitoring Models in Production, Data Drift and Concept Drift

A deployed model fails without an error message. It keeps returning scores while the world moves away from its training data, and only monitoring shows the gap.

110 minutesBuildPythonMembers

Learning Objectives

  • Choose monitoring signals at four layers: system health, inputs, predictions, and outcomes
  • Distinguish covariate drift, prior drift, and concept drift by the part of the joint distribution that changes
  • Implement the Population Stability Index, the two-sample Kolmogorov-Smirnov statistic, and the chi-square test from scratch
  • Compare a fixed reference window with sliding current windows, and explain how window size changes noise and delay
  • Design an alert policy that limits false alarms with effect sizes, a multiple-testing correction, and a persistence rule
  • Decide which kind of drift needs retraining, which needs new labels, and which needs no action