LLM Neuroanatomy: How Duplicating 3 Layers in a 24B Model Boosted Logical Deduction .22 to .76 — Without Training
A researcher named David Noel Ng achieved the #1 spot on the HuggingFace Open LLM Leaderboard with a model called dnhkng/RYS-XLarge — without training a single weight. The technique? Duplicating a specific block of middle layers in a 72B parameter model and stitching the result back together. No gradient descent, no merging, no fine-tuning.
The Discovery
The research started with two strange observations that led to what Ng calls LLM Neuroanatomy — a framework for understanding the internal functional structure of Transformer models.
Clue #1: Chatting with LLMs in Base64
Send a Base64-encoded question to a capable 2023-era LLM, and it will decode it, reason about it, and respond in Base64 — all within a single forward pass. The model encodes, thinks, and re-encodes in a format it was never specifically trained to reason in.
This led to a key insight: early layers act as translators (parsing input into an abstract internal representation), late layers act as re-translators (converting back to output format), and middle layers perform pure, abstract reasoning in a representation independent of any human language.
Clue #2: The Goliath Anomaly
The Goliath-120b model (a Frankenmerge of two Llama-2 70B models) alternated layers from two different fine-tuned models and even fed later-layer outputs back into earlier-layer inputs — something that should create completely out-of-distribution activations. Astonishingly, the model still functioned.
The Technique
Ng built a homebrew "brain scanner" for Transformers — a tool that measures the statistical similarity between consecutive layers' activations. This revealed a consistent pattern:
- Early layers (1-15%): Rapidly changing — parsing and encoding input
- Middle layers (15-70%): Remarkably stable — abstract reasoning
- Late layers (70-100%): Changing again — decoding and generating output
The middle layers form what Ng calls a "reservoir" — a stable internal representation where the actual reasoning happens. By duplicating these layers, you give the model more "thinking space" without changing any weights.
Results
The technique was applied across different model sizes:
- 72B model: Duplicating 7 middle layers → #1 on HuggingFace Open LLM Leaderboard
- 24B model: Duplicating 3 layers → Logical deduction scores jumped from .22 to .76
The improvement was not uniform — it specifically boosted tasks requiring deeper reasoning (GPQA, MuSR, MATH) while having less effect on simpler benchmarks. This is consistent with the theory: more "thinking layers" help with complex, multi-step reasoning.
Why It Matters
This finding challenges several assumptions in ML:
- More parameters ≠ better — the improvement came from architecture, not scale
- Middle layers aren't interchangeable — there are functionally distinct regions
- Models have internal "organs" — specialized layer blocks for encoding, reasoning, and decoding
- Architecture hacking works — you can meaningfully improve a model by understanding its internal structure
The research also has practical implications: instead of training larger models, you might achieve similar reasoning improvements by strategically duplicating existing layers — at zero training cost.
Open Questions
- Exactly which layers should be duplicated? (It depends on the model)
- Is there a theoretical limit to how many times you can duplicate?
- Does this work for other modalities (vision, audio)?
Source: David Noel Ng's Blog | HN Discussion