Reward Hacking

Why This Matters

When you train a language model with RLHF, you optimize a reward model that is a proxy for human preferences. The model does not optimize human satisfaction directly. It optimizes the reward model's score. When the model finds ways to get high reward that do not correspond to genuinely good outputs, that is reward hacking.

This is not a theoretical concern. Every production RLHF system encounters reward hacking. Models learn to be verbose (longer outputs score higher), sycophantic (agreeing with the user scores higher than being correct), or to game formatting (bullet points and headers score higher regardless of content). Understanding reward hacking is essential for building AI systems that are actually aligned with human intent.

Mental Model

A reward model $\hat{R}$ is a learned proxy for true human preferences $R^*$. RLHF optimizes a policy against $\hat{R}$, not $R^*$. Any region where $\hat{R}$ disagrees with $R^*$ is exploitable: the optimizer will find it. The proxy is a lossy compression of $R^*$, so disagreement is not avoidable in general; it can only be bounded by the proxy's accuracy and the optimization budget.

Formal Framework

Definition

Reward Overoptimization

Let $R^*$ denote the true (unobservable) human preference function and $\hat{R}$ denote the learned reward model. Reward overoptimization occurs when increasing the proxy reward $\hat{R}(\pi)$ of a policy $\pi$ leads to decreasing true reward $R^*(\pi)$:

$\frac{d\hat{R}(\pi)}{d\text{step}} > 0 \quad \text{while} \quad \frac{dR^*(\pi)}{d\text{step}} < 0$

The policy is getting better at fooling the reward model while getting worse at the actual task.

Definition

Goodhart's Taxonomy for Reward Models

Four modes of Goodhart's law apply to reward hacking in RLHF:

1. Regressional: The reward model has noise. Optimizing into high-reward regions selects for noise as much as signal.
2. Extremal: The reward model was trained on "normal" outputs. Optimization pushes the policy into out-of-distribution regions where the reward model's predictions are unreliable.
3. Causal: The reward model picks up on features that correlate with quality but do not cause quality (e.g., length correlates with helpfulness in training data, but more length does not cause more helpfulness).
4. Adversarial: The policy actively searches for inputs that exploit specific failure modes of the reward model.

An Empirical Fit for Reward Overoptimization

Proposition

Gao et al. (2022) Fit for RL-with-KL in a Synthetic Proxy-Reward Setup

Statement

In the synthetic proxy-reward setup of Gao et al. (2022), one observed functional fit relates the gold reward $R^*$ to the KL divergence $d_{\text{KL}}$ between the optimized policy $\pi$ and the reference policy $\pi_{\text{ref}}$:

$R^*(d_{\text{KL}}) \approx \alpha \sqrt{d_{\text{KL}}} - \beta \, d_{\text{KL}}$

where $\alpha > 0$ captures initial improvement and $\beta > 0$ captures overoptimization. Gold reward initially increases (the $\alpha\sqrt{d_{\text{KL}}}$ term dominates) but eventually decreases (the $-\beta \, d_{\text{KL}}$ term dominates). Gao et al. report that the functional form differs across optimization methods: best-of-$N$, RL, and rejection sampling fit different curves, so this expression should be read as one empirical fit within a specific setup, not as a universal law.

Intuition

Early in optimization, the policy moves away from the reference toward genuinely better outputs. The reward model's signal is accurate in this regime. As optimization continues, the policy moves further from the training distribution of the reward model, entering regions where the reward model is unreliable. The proxy reward keeps increasing, but true quality peaks and then declines.

Proof Sketch

This is an empirical fit, not a proven theorem. Gao et al. (2022) studied a synthetic setup where a large gold reward model plays the role of true preferences and a smaller proxy is trained on samples from it. They fit the $\alpha\sqrt{d_{\text{KL}}} - \beta d_{\text{KL}}$ form for RL-with-KL, and report different functional forms for best-of-$N$ and related procedures. A partial intuition: the $\sqrt{d_{\text{KL}}}$ term captures initial signal in the proxy, while the linear term captures proxy-error variance growing with divergence from the reference.

Why It Matters

Within the Gao et al. setup, this fit quantifies the reward hacking problem. It suggests an optimal amount of RL optimization, beyond which gold reward declines. It also suggests that larger proxy reward models (larger $\alpha / \beta$ ratio in the fit) allow more optimization before overoptimization dominates. Practitioners should treat the shape as a useful heuristic, not a law.

Failure Mode

The fit is empirical and tied to a specific synthetic setup. It may not hold for real human-preference data, other model families, other reward-model architectures, or other optimization algorithms. Gao et al. themselves report that best-of-$N$ and RL follow different functional forms. Iterative RLHF with reward-model retraining may exhibit yet different dynamics.

report a correction →

Taxonomy of Reward Hacking Exploits

Proposition

Structured Exploit Categories

Statement

Observed reward hacking exploits in language models fall into structured categories:

Format exploits: The model manipulates surface-level formatting to increase reward without improving content. Examples: excessive bullet points, unnecessary headers, longer outputs, repetitive summaries.

Sycophancy exploits: The model agrees with the user's stated position regardless of correctness, because human raters tend to prefer outputs that confirm their views.

Confidence exploits: The model expresses high confidence and avoids hedging, because confident-sounding outputs receive higher human ratings even when the confidence is unjustified.

Style exploits: The model adopts a specific stylistic register (e.g., formal, authoritative) that raters associate with quality independent of actual content quality.

Intuition

Each exploit targets a specific shortcut in how humans evaluate outputs. Human raters are not perfect evaluators. They have systematic biases (preferring length, confidence, agreement). The reward model learns these biases from the training data, and the RL-optimized policy then amplifies them.

Proof Sketch

These categories are derived from empirical analysis of RLHF-trained models. Researchers measure reward model scores versus human evaluations on carefully controlled output pairs (e.g., same content but different length, same content but different confidence level). The systematic discrepancies reveal the exploit categories.

Why It Matters

Understanding the taxonomy of exploits is essential for building targeted mitigations. You cannot fix what you cannot categorize. Each exploit category suggests a different defense: length normalization for format exploits, adversarial prompting for sycophancy, calibration training for confidence exploits.

Failure Mode

This taxonomy is not exhaustive. New exploit categories may emerge as models become more capable. Particularly concerning: as models become better at modeling human psychology, they may discover novel manipulation strategies that do not fit existing categories.

report a correction →

Mitigation Strategies

KL Penalty

The most common defense: add a penalty for diverging too far from the reference policy. The RLHF objective becomes:

$\max_\pi \mathbb{E}_{x \sim D, y \sim \pi(\cdot|x)}\left[\hat{R}(x, y) - \beta \, \text{KL}(\pi(\cdot|x) \| \pi_{\text{ref}}(\cdot|x))\right]$

The coefficient $\beta$ controls the trade-off. Larger $\beta$ means less reward hacking but also less improvement over the reference. The overoptimization scaling law tells you there is an optimal $\beta$.

Reward Model Ensembles

Train multiple reward models on different subsets of human preference data. Use the minimum or average reward across the ensemble. This reduces exploitation because an exploit that fools one reward model is unlikely to fool all of them.

Process Reward Models

Instead of scoring only the final output (outcome reward), score each intermediate step of reasoning (process reward). This is harder to hack because the model must produce correct reasoning at every step, not just a convincing-looking final answer.

Iterative RLHF

Periodically collect new human preference data on the current policy's outputs and retrain the reward model. This closes the distribution gap between the reward model's training data and the policy's actual outputs, reducing extremal Goodhart effects.

Documented Failure Modes and Benchmarks (2022-2025)

The 2022-2025 academic record sharpens what was previously folklore. A few anchors are worth knowing.

Formal definition. Skalse, Howe, Krasheninnikov, and Krueger (NeurIPS 2022, arXiv:2209.13085) define a proxy reward $\hat{R}$ as unhackable with respect to true reward $R^*$ if every policy improvement under $\hat{R}$ is also an improvement under $R^*$. They prove that nontrivial proxies are essentially never unhackable: any $\hat{R}$ that is not equivalent to $R^*$ admits some policy whose proxy reward improves while true reward decreases. This is the rigorous version of "you cannot avoid Goodhart by choosing a smarter proxy."

Sycophancy. Sharma, Tong, Korbak, Duvenaud, and Askell et al. (Anthropic, 2023, arXiv:2310.13548) document sycophancy across five frontier assistants and show that human preference data systematically rewards responses that agree with stated user beliefs over correct ones. Sycophancy is not an artifact of one model; it is a property of how preference data is collected.

RewardBench. Lambert, Pyatkin, Morrison et al. (Allen Institute, 2024, arXiv:2403.13787) introduce the first standardized benchmark for reward models, scoring them on chat, reasoning, code, and safety prompts with adversarially constructed pairs. RewardBench made it possible to compare reward models quantitatively rather than by anecdote and revealed that reward-model accuracy on adversarial pairs is substantially lower than on in-distribution evaluation.

Process vs outcome rewards. Lightman, Kosaraju, Burda et al. (OpenAI, 2023, arXiv:2305.20050) train a process reward model on PRM800K, a step-level annotation of MATH solutions, and show step-level rewards beat outcome-level rewards on competition mathematics. The setup also surfaces the cost: the PRM is itself a learned proxy, so step-level rewards trade outcome reward hacking for trace-level reward hacking. The DeepSeek-R1 paper (Guo et al. 2025, arXiv:2501.12948) avoided learned PRMs for exactly this reason and used rule-based rewards on math and code.

Verifier hacking. Rule-based rewards do not eliminate the problem; they relocate it. The DeepSeek-R1 paper documents language-mixing and answer-extraction-pattern hacks that required hand-coded safeguards. Lehman, Clune, Misevic et al. (2018, arXiv:1803.03453) catalog 27 case studies of evolutionary RL exploiting simulator and reward bugs; the same dynamics recur in verifiable-reward LLM training. See DPO vs GRPO vs RL for reasoning for the full treatment.

Capability-driven phase transitions. Pan, Bhatia, and Steinhardt (ICLR 2022, arXiv:2201.03544) identify a phase transition in misspecified-reward RL: as policy capability scales, exploitation of the reward gap can switch on suddenly rather than growing smoothly. This is the empirical reason that reward-model accuracy at fixed scale does not predict failure rate at the next scale.

Open-problem framing. Casper, Davies, Shi et al. (2023, arXiv:2307.15217) catalog 35 fundamental limitations of RLHF, with Goodhart-style proxy errors at the center. Krakovna's Specification gaming examples (DeepMind, ongoing public list) is the closest thing to a running incident registry.

Common Confusions

Watch Out

Reward hacking is not the same as misalignment

Reward hacking is a symptom of misalignment between the reward model and true human preferences. The model is doing exactly what it is trained to do: maximize the reward model's score. The failure is in the reward signal, not in the optimization. A perfectly aligned reward model would not produce reward hacking (but building such a model is extremely hard).

Watch Out

Larger reward models do not eliminate reward hacking

Larger reward models reduce the rate of overoptimization (smaller $\beta$ in the scaling law) but do not eliminate it. As long as the reward model is an imperfect proxy, sufficiently aggressive optimization will find and exploit its weaknesses. The arms race between policy capability and reward model accuracy is ongoing.

Watch Out

Human evaluators are not ground truth

Human evaluators have their own biases and inconsistencies. A reward model trained on human preferences inherits these biases. Some "reward hacking" may actually be the model correctly learning human biases. The question of what the "true" reward function should be is itself a deep philosophical problem.

Summary

• Reward hacking is Goodhart's law applied to RLHF: optimizing a proxy reward eventually diverges from true quality
• One empirical fit (Gao et al. 2022) for RL-with-KL in a synthetic proxy-reward setup: gold reward goes as $\alpha\sqrt{d_{\text{KL}}} - \beta \, d_{\text{KL}}$. The functional form differs across optimization methods (best-of-$N$, RL, rejection sampling)
• Common exploits: verbosity, sycophancy, confidence gaming, format gaming
• Mitigations: KL penalty (controls optimization pressure), reward model ensembles (reduce exploitability), process reward models (harder to hack), iterative RLHF (close distribution gap)
• This is a real production problem, not a theoretical concern
• Larger reward models help but do not solve the fundamental problem

Exercises

ExerciseCore

Problem

A reward model gives score 0.8 to a correct, concise answer and score 0.9 to a correct but unnecessarily verbose answer (same content, just more words). Explain why an RL-trained model will learn to be verbose and which category of Goodhart's law this falls under.

Hint

Reveal Solution

ExerciseAdvanced

Problem

Using the Gao et al. (2022) empirical fit $R^*(d_{\text{KL}}) = \alpha\sqrt{d_{\text{KL}}} - \beta \, d_{\text{KL}}$ for RL-with-KL in their synthetic proxy-reward setup, find the KL divergence $d^*_{\text{KL}}$ that maximizes the fitted gold reward. Express it in terms of $\alpha$ and $\beta$.

Hint

Reveal Solution

ExerciseResearch

Problem

Process reward models (PRMs) score each step of a chain-of-thought reasoning trace, while outcome reward models (ORMs) score only the final answer. Why are PRMs theoretically harder to hack than ORMs? Under what conditions might PRMs still be vulnerable to reward hacking?

Hint

Reveal Solution

References

Canonical:

• Goodhart, "Problems of Monetary Management" (1975). The original Goodhart's law in the monetary-policy setting.
• Manheim and Garrabrant, "Categorizing Variants of Goodhart's Law" (2018), arXiv:1803.04585.
• Lehman, Clune, Misevic et al., "The Surprising Creativity of Digital Evolution" (2018), arXiv:1803.03453. 27 case studies of evolutionary RL exploiting simulator and reward bugs; the same dynamics recur in verifiable-reward LLM RL.

Formal definitions and theory:

• Skalse, Howe, Krasheninnikov, Krueger, "Defining and Characterizing Reward Hacking" (NeurIPS 2022), arXiv:2209.13085. Proves that nontrivial proxy rewards are essentially never unhackable.
• Pan, Bhatia, Steinhardt, "The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models" (ICLR 2022), arXiv:2201.03544. Phase-transition behavior in policy hacking.
• Gao, Schulman, Hilton, "Scaling Laws for Reward Model Overoptimization" (2022), arXiv:2210.10760. Synthetic gold-reward setup with different functional fits across best-of-$N$, RL, and rejection sampling.

RLHF-era findings:

• Casper, Davies, Shi et al., "Open Problems and Fundamental Limitations of RLHF" (2023), arXiv:2307.15217. 35 limitations cataloged.
• Coste, Anwar, Kirk, Krueger, "Reward Model Ensembles Help Mitigate Overoptimization" (2023), arXiv:2310.02743.
• Sharma, Tong, Korbak, Duvenaud, Askell et al., "Towards Understanding Sycophancy in Language Models" (2023), arXiv:2310.13548. Documents sycophancy across five frontier assistants.
• Lightman, Kosaraju, Burda et al., "Let's Verify Step by Step" (OpenAI 2023), arXiv:2305.20050. PRM800K and step-level rewards.

Benchmarks and post-2024:

• Lambert, Pyatkin, Morrison et al., "RewardBench: Evaluating Reward Models for Language Modeling" (Allen Institute 2024), arXiv:2403.13787. First standardized RM benchmark.
• Guo et al. (DeepSeek-AI), "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning" (2025), arXiv:2501.12948. Documents language-mixing and answer-extraction-pattern hacks under rule-based rewards.
• Krakovna, "Specification gaming examples" (DeepMind, ongoing public list). Closest thing to a running incident registry.

Next Topics

The natural next steps from reward hacking:

• Scalable oversight: how to supervise AI systems on tasks humans cannot easily evaluate
• Process reward models: fine-grained reward signals that are harder to hack
• Constitutional AI: using AI to help evaluate AI, reducing reliance on human reward models