Google DeepMind Proposes Cognitive Framework for Measuring Progress Toward AGI
Google DeepMind has released a new paper, "Measuring Progress Toward AGI: A Cognitive Taxonomy," that presents a scientific foundation for understanding the cognitive capabilities of AI systems. The framework draws on decades of research from psychology, neuroscience, and cognitive science.
The 10 Cognitive Abilities
The framework identifies 10 key cognitive abilities hypothesized to be important for general intelligence in AI:
| # | Ability | Description |
|---|---|---|
| 1 | Perception | Extracting and processing sensory information from the environment |
| 2 | Motor control | Producing outputs such as text, speech, and actions |
| 3 | Attention | Focusing cognitive resources on what matters |
| 4 | Learning | Acquiring new knowledge through experience and instruction |
| 5 | Memory | Storing and retrieving information over time |
| 6 | Reasoning | Drawing valid conclusions through logical inference |
| 7 | Metacognition | Knowledge and monitoring of one's own cognitive processes |
| 8 | Executive functions | Planning, inhibition, and cognitive flexibility |
| 9 | Problem solving | Finding effective solutions to domain-specific problems |
| 10 | Social cognition | Processing and interpreting social information |
Three-Stage Evaluation Protocol
- Evaluate AI systems across a broad suite of cognitive tasks covering each ability, using held-out test sets to prevent data contamination
- Collect human baselines for the same tasks from a demographically representative sample of adults
- Map each AI system's performance relative to the distribution of human performance in each ability
$200K Kaggle Hackathon
To put the framework into practice, DeepMind is launching a Kaggle hackathon focused on five cognitive abilities with the largest evaluation gaps: learning, metacognition, attention, executive functions, and social cognition.
- Prize pool: $200,000 total
- Format: $10,000 awards for top 2 submissions in each of 5 tracks, plus $25,000 grand prizes for 4 best overall submissions
- Timeline: Submissions open March 17 through April 16, results announced June 1
- Platform: Kaggle's model evaluation platform, testing against a lineup of frontier models
Why It Matters
Most AGI benchmarks focus on task-level performance (MMLU, HumanEval, etc.). DeepMind's cognitive taxonomy approach shifts the evaluation lens to underlying capabilities — asking not just "can the model do X?" but "does the model possess the cognitive ability required to do X, Y, and Z across domains?"
Source: Google Blog | HN Discussion