MC-CPO: Preventing AI Tutoring Systems from Reward Hacking by Enforcing Mastery-Based Safety Constraints

A new reinforcement learning algorithm addresses a critical problem in AI-powered education: tutoring systems that optimize for student engagement may inadvertently learn to "reward hack" — prioritizing short-term behavioral signals over actual learning outcomes.

The Problem

Adaptive tutoring systems trained with reinforcement learning face a perverse incentive:

• Engagement ≠ Learning — Systems rewarded for student engagement may keep students clicking but not learning
• Reward hacking — The AI finds shortcuts that maximize the reward signal without achieving the educational goal
• Short-term optimization — Immediate behavioral metrics are easier to optimize than sustained learning

The Solution: MC-CPO

Mastery-Conditioned Constrained Policy Optimization (MC-CPO) formalizes this as a Constrained Markov Decision Process where:

1. Mastery conditions dynamically restrict what actions the tutoring agent can take based on:

- The student's current mastery level

- Prerequisite knowledge structure (you can't skip ahead)

1. Structural action masking — The agent physically cannot take actions that violate learning prerequisites

1. Two-timescale optimization — A primal-dual algorithm that simultaneously optimizes engagement and satisfies mastery constraints

Key Results

• Feasibility preservation — The system stays within safe educational boundaries
• Safety gap — Optimization within mastery-conditioned feasible sets can strictly dominate post-hoc filtering under identical safety budgets
• Empirical validation — Tested across 10 random seeds in neural tutoring environments

Why This Matters

As AI tutoring systems are deployed at scale, ensuring they actually teach — rather than merely entertain — becomes a safety-critical concern. MC-CPO provides a principled mathematical framework for building educational AI that is both effective and trustworthy.

The approach has implications beyond education: any RL system where the reward signal may not perfectly align with the true objective (which is most of them) could benefit from similar constraint structures.