Evaluating Adaptive AI Medical Devices: New Framework Measures Learning, Potential, and Retention

Researchers have proposed a novel approach for evaluating adaptive AI-enabled medical devices, addressing the challenge that iterative model updates and changing patient populations make traditional performance assessment inadequate. The framework introduces three complementary measurements.

The Challenge

Adaptive AI medical devices — which improve over time based on new data — pose a unique regulatory challenge:

• The model changes as it learns from new patients
• The patient population itself shifts over time
• Traditional validation assumes a static model on a fixed dataset
• How do you evaluate something that's designed to change?

The Three Measurements

1. Learning

Measures whether the model actually improves on current data after each update cycle. Is the adaptation working?

2. Potential

Evaluates how dataset-driven population shifts affect performance. Does the model handle distribution changes?

3. Retention

Tests whether the model preserves previously learned knowledge across modification steps. Does it forget?

Key Findings

• Gradual population transitions enable stable learning and retention
• Rapid population shifts reveal trade-offs between plasticity (learning new things) and stability (remembering old things)
• The three measurements together disentangle performance changes caused by model adaptation vs. environmental change

Regulatory Implications

This framework provides practical tools for regulatory science, enabling rigorous assessment of adaptive AI safety and effectiveness over sequential modifications. As AI medical devices become more common, such frameworks will be essential for ensuring patient safety while allowing beneficial adaptation.