Theoretical Limits of Language Model Alignment

According to Apple Machine Learning:

Lucas Monteiro Paes and co-authors explore the fundamental information-theoretic limits of language model alignment, specifically examining reinforcement learning and best-of-N alignment under a KL-divergence budget.

Theoretical Findings and Estimator

The authors derive a closed-form expression for optimal reward improvement governed by a Jeffreys divergence term rather than traditional KL divergence. By reformulating this expression as a covariance under the base model, they provide a practical estimator that predicts achievable alignment gains using only base model samples.

Reward Hacking and Ensembling

Extending the analysis to proxy reward settings, the authors demonstrate that the gap between ideal and proxy alignment, known as reward hacking, grows when reward error is large or the KL penalty factor decreases. They also provide a theoretical justification for reward ensembling, proving it mitigates this issue.

Empirical Results

Empirical computations of the KL-reward Pareto frontier for safety and summarization tasks reveal that best-of-N sampling closely approaches the derived theoretical limit, whereas PPO and GRPO remain substantially suboptimal, highlighting the need for algorithmic improvements.