Capstone 17, Personal AI Tutor (Adaptive, Multimodal, with Memory)
Khanmigo (Khan Academy), Duolingo Max, Google LearnLM / Gemini for Education, Quizlet Q-Chat, and Synthesis Tutor all shipped adaptive multimodal tutoring at scale in 2026. The common shape is a Socratic policy (never just dump the answer), a learner model that updates after every interaction (Bayesian knowledge tracing style), voice + text + photo-math input, curriculum graph retrieval, spaced-repetition scheduling, and hard safety filters for age-appropriate content. The capstone is to ship a subject-specific tutor (K-12 algebra or intro Python), run a two-week efficacy study with 10 learners, and pass a content-safety audit.
Problem
Adaptive tutoring used to be an ed-tech research niche. By 2026 it is a consumer product. Khanmigo is deployed across most US school districts. Duolingo Max hit tens of millions of MAUs. Google's LearnLM / Gemini for Education powers tutoring in Google Classroom. Quizlet Q-Chat sits alongside flashcards. Synthesis Tutor hit virality with tutor-for-curious-kids. The common elements: multimodal input (type, speak, photograph equations), Socratic pedagogy (ask first, explain later), a learner model that updates after each interaction, and strict age-appropriate safety.
You will build one of these for a specific cohort. The measurement bar is an actual efficacy study: pre-test and post-test scores over two weeks with 10 learners. The voice loop must feel natural (capstone 03 sub-stack). The memory must be privacy-respecting. The safety filter must pass COPPA-aware red-team for K-12.
Concept
Four components. Tutor policy is a Socratic loop: when the learner asks for the answer, the policy asks a leading question; when they get it right, it moves to the next concept; when they are stuck, it offers a scaffolded hint. Learner model is Bayesian knowledge tracing (or a simple variant) that updates mastery probability per curriculum node after each interaction. Curriculum graph is a Neo4j of concepts with prerequisite edges; the policy walks the graph to pick the next concept. Memory is an episodic + semantic store (agentmemory-style) holding past interactions, mistakes, and preferences.
The UX is multimodal. Text input for typed answers. Voice input via LiveKit + Whisper (reuse capstone 03). Photo input for math problems via dots.ocr or PaliGemma 2. Voice output via Cartesia Sonic-2. Safety uses Llama Guard 4 plus an age-appropriate filter (blocks adult content, violence, self-harm) and a COPPA-aware memory retention policy.
The efficacy study is the deliverable. 10 learners, pre-test and post-test, two weeks. Report learning gain delta and confidence interval. Compare against a non-adaptive baseline (the same content delivered linearly without the tutor policy).
Architecture
learner device
|
+-- text -> web app
+-- voice -> LiveKit Agents (ASR + TTS)
+-- photo math -> dots.ocr / PaliGemma 2
|
v
tutor policy (LangGraph)
- Socratic decision head
- next-concept chooser (curriculum graph walk)
- hint scaffolder
- mastery update
|
v
learner model (BKT / item-response theory)
- per-concept mastery probability
- spaced-repetition scheduler (SM-2 or FSRS)
|
v
memory (agentmemory-style)
- episodic: every interaction
- semantic: learned mistakes, preferences
- retention policy: COPPA / GDPR aware
|
v
curriculum graph (Neo4j)
- prerequisite edges
- OER content attached
|
v
safety:
Llama Guard 4 + age-appropriate filter
memory access guarded by learner ID scopeStack
- Subject choice: K-12 algebra or intro Python (pick one for depth)
- Tutor policy: LangGraph over Claude Sonnet 4.7 (with prompt caching)
- Learner model: Bayesian knowledge tracing (classic) or FSRS for spacing
- Curriculum graph: Neo4j of concepts + prerequisite edges + OER content
- Memory: agentmemory-style persistent vector + episodic + semantic store
- Voice: LiveKit Agents 1.0 + Cartesia Sonic-2 (reuse capstone 03 sub-stack)
- Photo math: dots.ocr or PaliGemma 2 for equation recognition
- Safety: Llama Guard 4 + custom age-appropriate filter
- Eval: Bloom-level question generation, pre/post test harness, efficacy study tooling
Build It
- Curriculum graph. Build a Neo4j of 50-150 concept nodes (e.g., K-12 algebra from "number line" to "quadratic formula") with prerequisite edges. Attach OER content per node (Open Textbook, OpenStax).
- Learner model. Initialize Bayesian knowledge tracing with priors: guess, slip, learn-rate. Update per-concept mastery after each interaction. Persist per learner.
- Tutor policy. LangGraph with nodes:
read_signal(was the learner's answer correct / partial / stuck?),select_concept(walk curriculum graph picking the highest-priority concept),scaffold(Socratic prompt),update_mastery. - Memory. Every interaction writes to an episodic store. Mistakes and preferences promote to semantic memory. COPPA-aware retention policy: auto-delete after 1 year, parent-accessible.
- Voice path. LiveKit Agents worker attached to the tutor policy. ASR via Whisper-v3-turbo. TTS via Cartesia Sonic-2. Barge-in supported (reuse capstone 03 mechanics).
- Photo-math path. Upload or capture image; run dots.ocr or PaliGemma 2 to recognize the equation; feed to tutor as structured input.
- Safety. Every model output passes Llama Guard 4 + an age-appropriate filter (blocks self-harm, adult content, violence). Memory access scoped by learner ID; parental access surface for deletion.
- Efficacy study. 10 learners, pre-test (standardized 30-question baseline), two weeks of tutor interaction (3 sessions/week), post-test. Compare against a non-adaptive baseline cohort of 10 learners on the same content.
- Weekly progress reports. Per learner, auto-generate a PDF summary of topics explored, mastery trajectories, and recommended next steps.
Use It
learner: "I don't understand why 3x + 6 = 12 means x = 2"
[signal] stuck
[concept] 'isolating variables' (prerequisite: addition-subtraction-equality)
[scaffold] "what number would you subtract from both sides to start?"
learner: "6"
[signal] correct
[mastery] addition-subtraction-equality: 0.62 -> 0.77
[concept] continue 'isolating variables'
[scaffold] "great. now what is 3x / 3 equal to?"Ship It
outputs/skill-ai-tutor.md is the deliverable. A subject-specific adaptive tutor with multimodal input, a learner model, memory, safety, and measured efficacy.
| Weight | Criterion | How it is measured |
|---|---|---|
| 25 | Learning gain delta | Pre/post-test delta in a 10-learner two-week study |
| 20 | Socratic fidelity | Rubric score on transcript samples |
| 20 | Multimodal UX | Voice + photo + text coherence end to end |
| 20 | Safety + privacy posture | Llama Guard 4 pass rate + COPPA-aware retention |
| 15 | Curriculum breadth and graph quality | Concept coverage + prerequisite graph consistency |
| 100 |
Exercises
- Run the efficacy study with and without the adaptive learner model (random concept order). Report the delta. Expect adaptive to win, but the size is the interesting number.
- Add a multimodal probe: the same concept question delivered as text, voice, and photo. Measure whether learners converge faster with the modality they prefer.
- Build a parent dashboard: topics practiced, mastery trajectories, upcoming concepts, safety events (any guardrail hits). COPPA-aligned.
- Add a language-switch mode: the tutor accepts Spanish input and teaches in Spanish. Measure X-Guard coverage.
- Stress the memory privacy: verify that learner A cannot see learner B's data even through a voice-clip re-ingest attack. Log the attempted access and alert.
Key Terms
| Term | What people say | What it actually means |
|---|---|---|
| Socratic policy | "Ask, do not dump" | Tutor asks a leading question rather than giving the answer |
| Bayesian knowledge tracing | "BKT" | Classic learner-model equations for mastery probability per concept |
| FSRS | "Free Spaced Repetition Scheduler" | 2024 spaced-repetition scheduler, better than SM-2 |
| Curriculum graph | "Concept DAG" | Neo4j of concepts with prerequisite edges |
| Episodic memory | "Per-interaction log" | Every interaction stored for later retrieval |
| Semantic memory | "Learned pattern store" | Compacted mistakes and preferences promoted from episodic |
| COPPA | "Kids privacy law" | US law restricting data collection from children under 13 |
Further Reading
- Khanmigo (Khan Academy)reference consumer K-12 tutor
- Duolingo Maxreference language-learning tutor
- Google LearnLM / Gemini for Educationhosted reference model
- Quizlet Q-Chatalternate reference
- Synthesis Tutorstartup reference
- FSRS algorithmspaced-repetition scheduler
- Bayesian Knowledge Tracinglearner-model classic
- LiveKit Agentsvoice stack
Check what you learned
7 questions. Answer each one to see why.
What does a Socratic tutor policy do when a learner asks for the answer outright?
The full code
Every file for this lesson. Copy it, run it, then change it.
"""Personal AI tutor — Bayesian knowledge tracing + Socratic policy scaffold.
The hard architectural primitive is the learner model: per-concept mastery
probability updated by Bayesian knowledge tracing after each interaction,
feeding into a curriculum-graph walk that chooses the next concept. This
scaffold implements BKT, a curriculum DAG, a Socratic policy decision, and
a simulated two-learner study.
Run: python main.py
"""
from __future__ import annotations
import random
from collections import defaultdict
from dataclasses import dataclass, field
# ---------------------------------------------------------------------------
# Bayesian knowledge tracing -- classic four-parameter model
# ---------------------------------------------------------------------------
@dataclass
class BKTParams:
p_init: float = 0.2 # prior knowledge
p_learn: float = 0.12 # learning rate per practice
p_slip: float = 0.10 # correct despite not knowing
p_guess: float = 0.15 # correct by guessing despite not knowing
def bkt_update(mastery: float, correct: bool, p: BKTParams) -> float:
if correct:
num = mastery * (1 - p.p_slip)
denom = num + (1 - mastery) * p.p_guess
else:
num = mastery * p.p_slip
denom = num + (1 - mastery) * (1 - p.p_guess)
posterior = num / max(denom, 1e-6)
# transition: learn from this interaction
return posterior + (1 - posterior) * p.p_learn
# ---------------------------------------------------------------------------
# curriculum graph -- DAG of concepts with prerequisite edges
# ---------------------------------------------------------------------------
@dataclass
class Concept:
name: str
prereqs: list[str] = field(default_factory=list)
ALGEBRA = [
Concept("number_line", []),
Concept("addition_subtraction", ["number_line"]),
Concept("multiplication_division", ["addition_subtraction"]),
Concept("negative_numbers", ["addition_subtraction"]),
Concept("equality", ["addition_subtraction"]),
Concept("isolating_variable_one_step", ["equality", "addition_subtraction"]),
Concept("isolating_variable_two_step", ["isolating_variable_one_step", "multiplication_division"]),
Concept("distributive_property", ["multiplication_division"]),
Concept("combining_like_terms", ["addition_subtraction", "distributive_property"]),
Concept("linear_equations", ["isolating_variable_two_step", "combining_like_terms"]),
Concept("quadratic_basics", ["linear_equations", "multiplication_division"]),
]
def curriculum_map(concepts: list[Concept]) -> dict[str, Concept]:
return {c.name: c for c in concepts}
# ---------------------------------------------------------------------------
# learner state -- per-concept mastery plus history
# ---------------------------------------------------------------------------
@dataclass
class LearnerState:
learner_id: str
mastery: dict[str, float] = field(default_factory=lambda: defaultdict(lambda: 0.2))
history: list[tuple[str, bool]] = field(default_factory=list)
# ---------------------------------------------------------------------------
# concept selector -- pick next concept with (a) prereqs met and (b) low mastery
# ---------------------------------------------------------------------------
def next_concept(state: LearnerState, cmap: dict[str, Concept],
master_threshold: float = 0.85) -> str | None:
for c in cmap.values():
if state.mastery[c.name] >= master_threshold:
continue
if all(state.mastery[pr] >= master_threshold for pr in c.prereqs):
return c.name
return None
# ---------------------------------------------------------------------------
# Socratic policy -- decides scaffold vs next-step vs celebration
# ---------------------------------------------------------------------------
def socratic_policy(state: LearnerState, concept: str, correct: bool) -> str:
m = state.mastery[concept]
if correct and m > 0.8:
return "celebrate_and_advance"
if correct:
return "reinforce_and_next_question"
if m > 0.5:
return "hint"
return "scaffold_from_prereq"
# ---------------------------------------------------------------------------
# learner simulator -- random-walk with difficulty sensitive to mastery
# ---------------------------------------------------------------------------
def simulate_answer(learner_knowledge: float, concept_difficulty: float,
rng: random.Random) -> bool:
"""Simulate whether the learner answers correctly."""
# probability of correct = sigmoid(knowledge - difficulty)
import math
p = 1 / (1 + math.exp(-(learner_knowledge - concept_difficulty)))
return rng.random() < p
# ---------------------------------------------------------------------------
# adaptive and baseline cohorts -- compare learning gain over N interactions
# ---------------------------------------------------------------------------
def run_adaptive(learner_id: str, inherent_ability: float,
cmap: dict[str, Concept], n_turns: int, rng: random.Random) -> LearnerState:
state = LearnerState(learner_id=learner_id)
p = BKTParams()
# last action taken by the tutor, threaded into the next turn so
# scaffold/hint actually reduce difficulty and celebration nudges mastery
last_action: str | None = None
for _ in range(n_turns):
concept = next_concept(state, cmap)
if concept is None:
break
difficulty = 0.3 + 0.1 * len(cmap[concept].prereqs)
# apply the previous turn's action to *this* turn
if last_action == "scaffold_from_prereq":
difficulty -= 0.15 # easier retry from prereqs
elif last_action == "hint":
difficulty -= 0.08 # mild nudge
elif last_action == "celebrate_and_advance":
# celebration buoys confidence for one turn
state.mastery[concept] = min(1.0, state.mastery[concept] + 0.02)
# effective knowledge = inherent + mastery
ek = inherent_ability + state.mastery[concept] * 1.5
correct = simulate_answer(ek, difficulty, rng)
last_action = socratic_policy(state, concept, correct)
state.history.append((concept, correct))
state.mastery[concept] = bkt_update(state.mastery[concept], correct, p)
return state
def run_baseline(learner_id: str, inherent_ability: float,
cmap: dict[str, Concept], n_turns: int, rng: random.Random) -> LearnerState:
"""Non-adaptive concept selection (round-robin). Mastery is still updated
via BKT so both arms share the same learner model; only the policy /
concept-selection strategy differs."""
state = LearnerState(learner_id=learner_id)
p = BKTParams()
order = list(cmap.keys())
for i in range(n_turns):
concept = order[i % len(order)]
difficulty = 0.3 + 0.1 * len(cmap[concept].prereqs)
ek = inherent_ability + state.mastery[concept] * 1.5
correct = simulate_answer(ek, difficulty, rng)
state.history.append((concept, correct))
state.mastery[concept] = bkt_update(state.mastery[concept], correct, p)
return state
def mastery_sum(state: LearnerState, cmap: dict[str, Concept]) -> float:
return sum(state.mastery[c] for c in cmap)
def main() -> None:
cmap = curriculum_map(ALGEBRA)
rng = random.Random(29)
print("=== 2-week efficacy study (simulated) ===")
print(f"curriculum: {len(cmap)} concepts")
adaptive_gains: list[float] = []
baseline_gains: list[float] = []
n_learners = 10
n_turns = 60
for i in range(n_learners):
ability = rng.gauss(0.3, 0.4)
# paired randomness: both arms consume the same latent RNG stream so
# the delta measures the policy difference, not seed noise
seed = 100 + i
r_adapt = random.Random(seed)
r_base = random.Random()
r_base.setstate(r_adapt.getstate())
s1 = run_adaptive(f"adapt_{i}", ability, cmap, n_turns, r_adapt)
s2 = run_baseline(f"base_{i}", ability, cmap, n_turns, r_base)
adaptive_gains.append(mastery_sum(s1, cmap))
baseline_gains.append(mastery_sum(s2, cmap))
def mean(xs): return sum(xs) / len(xs)
print(f"adaptive mastery sum mean={mean(adaptive_gains):.2f}")
print(f"baseline mastery sum mean={mean(baseline_gains):.2f}")
delta = mean(adaptive_gains) - mean(baseline_gains)
print(f"delta (adaptive - baseline): {delta:+.2f} mastery points over {n_turns} turns")
print("\n=== sample trajectory (adaptive learner 0) ===")
state = run_adaptive("demo", 0.3, cmap, 20, random.Random(7))
seen_concepts = []
for c, ok in state.history:
if c not in [x[0] for x in seen_concepts]:
seen_concepts.append((c, state.mastery[c]))
for c, m in seen_concepts[:8]:
print(f" {c:34s} mastery={m:.2f}")
if __name__ == "__main__":
main()