Generator Access Creates Exponential Gap in LLM Post-Training Efficiency

New research reveals that how you access a language model's generator during post-training creates an exponential performance gap in KL-regularized outcome-reward training. The difference between root-start-only rollouts and prefix-access methods is far larger than previously understood.

The Problem

During LLM post-training (like RLHF), the model generates tokens and receives rewards. But there's a fundamental question: how do you query the generator?

Two regimes exist:

1. Root-start rollouts — Always start generation from scratch (beginning of sequence)
2. Prefix access — Can revisit previously built prefixes and continue from any point

Key Findings

• In the root-start regime, all observation types (sampling, logprobs, top-k, full distributions) collapse into one canonical experiment
• Limited by the on-policy probability of reaching informative prefixes
• Weak prefix control breaks this barrier
• Richer observations (conditional sampling, logits) can outperform top-1 access once control is available
• Changing only the generator interface creates an exponential gap for KL-regularized training

What This Means

• Current RLHF implementations may be dramatically underutilizing the model's generator
• Simple infrastructure changes (enabling prefix access) could yield massive efficiency gains
• The gap is exponential, not linear — suggesting fundamental algorithmic improvements are possible
• This has implications for the entire LLM post-training pipeline

Practical Implications

For teams training LLMs with RL or DPO methods, ensuring prefix-level access to the generator could be one of the highest-impact infrastructure investments available.