AI Teaching Assistant Tested in Real University Exams: No Performance Difference But Students Rate It 4.22/5

Trinity College Dublin implemented an AI Teaching Assistant (AI-TA) using RAG for their Master's Motion Picture Engineering course, including allowing its use in open-book examinations — and found no statistical performance difference between students who used it and those who didn't.

The Experiment

Key Findings

1. No exam performance difference (p > 0.05) — Students with AI-TA access scored the same as those without across three exams
2. High satisfaction — Mean rating 4.22/5
3. Mixed preference vs human tutors — Mean 2.78/5, students still value human interaction
4. Thoughtfully designed assessments maintain academic validity even with AI access

The RAG Implementation

The AI-TA used Retrieval-Augmented Generation with:

• Course-specific knowledge base
• Tuned RAG pipeline for motion picture engineering
• Detailed prompt engineering
• Open-source code released

Why the No-Difference Result Matters

This is actually encouraging news:

• Assessment design works — Well-designed exams can maintain rigor even with AI tools
• AI as supplement, not crutch — Students used AI as a helper, not a replacement for learning
• Contrast with the N=1,222 study — That study showed AI hurts independent performance; this shows well-designed assessments can mitigate that risk

IEEE Signal Processing Magazine

Accepted for publication — peer-reviewed validation of the methodology and findings.

Implications for Education

• AI tools in exams — Can be permitted without degrading assessment quality
• Course-specific AI — Domain-adapted AI-TAs outperform general-purpose chatbots
• Student acceptance — High satisfaction suggests AI-TAs are a viable supplement
• Human tutors still valued — AI doesn't fully replace human teaching

This research provides practical guidance for universities considering AI integration into their teaching, showing that the key is thoughtful assessment design rather than banning AI tools entirely.