The β Parameter: What the Leash in RLHF Does and Does Not Do

Part 2 of Raising Machines, a six-part series that started with two quotes about raising AI. Part 1: Gandhi Won’t Take the Pill. Literature cutoff: 10 October 2026.

---

A Bug That Kept Its Manners

In September 2019 Daniel Ziegler and colleagues published a paper on fine-tuning language models from human preferences [[1]]. Section 4.4 is titled “Bugs can optimize for bad behavior”. A code refactor flipped the sign of the reward. “Flipping the reward would usually produce incoherent text, but the same bug also flipped the sign of the KL penalty. The result was a model which optimized for negative sentiment while still regularizing towards natural language.” The human labellers had been told to rate sexually explicit text very low, so with the sign inverted, “the model quickly learned to output only content of this form, regardless of how innocuous the starting point was.”

The paper files this under bugs. I want to read it as a result. Whatever the mechanics, the model kept writing natural language and spent all of that fluency on the wrong target. The part of the objective that held it close to natural language had no opinion about which way it was going.

Part 1 was about an agent that refuses to have its values changed: Bostrom and Yudkowsky’s Gandhi, who will not take the pill that would make him want to kill. This post is about the term in the standard training objective that looks like a formal version of that refusal: a penalty on drifting away from who the model used to be. Its coefficient is called β. If you read reinforcement learning from human feedback (RLHF) as parenting, β is the obvious candidate for the leash.

The mathematics says what β is: a temperature. The experiments say what turning it up buys: mostly an earlier stop. Neither makes it a conscience.

---

The Objective, as Three Papers Wrote It

Ziegler et al. state the idea in one sentence: “To keep π from moving too far from ρ, we add a penalty with expectation β KL(π, ρ)” [[1]]. Here ρ is the pretrained language model and π the policy being trained. Stiennon et al. (2020) used the same term against a supervised fine-tuned model, $\pi^{\mathrm{SFT}}$ [[2]], and InstructGPT (Ouyang et al., 2022) kept it, adding a second term that mixes pretraining gradients back in [[3]]. In the notation of Rafailov et al. (2023) [[4]], the objective for one prompt $x$ is

$$J(\pi) = \mathbb{E}_{y \sim \pi(\cdot \mid x)}\big[r(x,y)\big] \;-\; \beta\, \mathrm{KL}\big(\pi(\cdot \mid x) \,\|\, \pi_{\mathrm{ref}}(\cdot \mid x)\big),$$

where $y$ is a complete response, $r(x,y)$ is the score a learned reward model gives it, $\pi_{\mathrm{ref}}$ is the frozen reference model the policy started from, and

$$\mathrm{KL}(\pi \,\|\, \pi_{\mathrm{ref}}) = \sum_y \pi(y) \log \frac{\pi(y)}{\pi_{\mathrm{ref}}(y)}$$

is the Kullback–Leibler divergence in nats, with the policy in the first slot. All the papers cited here use that direction. InstructGPT applies the penalty per token, and the full objective averages over prompts.

The values are small. Ziegler used β = 0.1 and 0.03 on two summarisation tasks, Stiennon 0.05, InstructGPT 0.02 [[1]][[2]][[3]]. These numbers cannot be compared across papers, for a reason the closed form below makes exact.

It is worth reading why the authors say the term is there. Ziegler: “it plays the role of an entropy bonus, it prevents the policy from moving too far from the range where r is valid”, and, for the style tasks, it is relied on “to encourage coherence and topicality” [[1]]. Stiennon: it “acts as an entropy bonus, encouraging the policy to explore and deterring it from collapsing to a single mode”, and “ensures the policy doesn’t learn to produce outputs that are too different from those that the reward model has seen during training” [[2]]. InstructGPT gives one clause: the penalty is there “to mitigate overoptimization of the reward model” [[3]].

None of these reasons is about values. They are about entropy, coherence, and staying where the reward model can be trusted. The parenting reading is something we bring to the term, not something its authors put there.

---

The Closed Form Is a Boltzmann Distribution

The objective has an exact maximiser, and seeing it changes how β reads. Ziegler et al. already state it in §3.1.1: with the KL term, “the optimal policy has the form” $\pi_{\mathrm{opt}} \propto \rho\, e^{r/\beta}$ [[1]]. The derivation is short enough to give in full.

Fix the prompt and drop it from the notation. Define

$$Z = \sum_y \pi_{\mathrm{ref}}(y)\, e^{r(y)/\beta}, \qquad \pi^*(y) = \frac{1}{Z}\, \pi_{\mathrm{ref}}(y)\, e^{r(y)/\beta}.$$

$Z$ is a positive number and $\pi^*$ is a probability distribution. Taking logarithms gives $r(y) = \beta \log\big(\pi^*(y)/\pi_{\mathrm{ref}}(y)\big) + \beta \log Z$. Substitute that into $J$ and the reference terms cancel:

$$J(\pi) = \sum_y \pi(y)\Big[\beta \log Z - \beta \log \frac{\pi(y)}{\pi^*(y)}\Big] = \beta \log Z \;-\; \beta\, \mathrm{KL}(\pi \,\|\, \pi^*).$$

This identity is not new. Korbak, Perez and Buckley (2022) write the objective as proportional to $-\mathrm{KL}(\pi \,\|\, \pi^*)$ (their Eq. 7), and Rafailov et al.’s Appendix A.1 has the same steps, divided by β and with the sign flipped [[5]][[4]]. The KL divergence is never negative, and in Rafailov’s words, “Gibbs’ inequality tells us that the KL-divergence is minimized at 0 if and only if the two distributions are identical” [[4]]. So $J(\pi) \le \beta \log Z$, with equality exactly when $\pi = \pi^*$. The optimum is unique, and the best achievable value is $J^* = \beta \log Z$.

A physicist reads $\pi^*$ at once. Call $-r(y)$ the energy of answer $y$ and β the temperature. Then $\pi^*$ is a Boltzmann distribution, measured relative to $\pi_{\mathrm{ref}}$ instead of uniformly. The word “temperature” is Korbak et al.’s: $\exp(r(x)/\beta)$ is “the evidence provided by the reward function (scaled by temperature β)”, $\pi_{\mathrm{ref}}$ is the prior, and $J$ is “the evidence lower bound (ELBO)” [[5]]. A physicist calls a negative ELBO a free energy: $F[\pi] = \mathbb{E}_\pi[-r] + \beta\, \mathrm{KL}(\pi \,\|\, \pi_{\mathrm{ref}}) = -J(\pi)$ is energy plus temperature times a relative entropy, minimised at $-\beta \log Z$. Maximising the RLHF objective is minimising a free energy, an instance of the Gibbs (or Donsker–Varadhan) variational principle. The lineage from Boltzmann distributions to neural networks has its own post.

Two caveats. A real network need not reach the optimum in a given budget: in one of Ziegler’s experiments “There is a significant gap from optimality after training the policy on 2M continuations—the number used in our main experiments—though it is largely closed with more training” [[1]]. And the optimum can only reweight what the reference model could say: where $\pi_{\mathrm{ref}}(y) = 0$, $\pi^*(y) = 0$. Since language models give every string some probability, in practice the constraint is cost: moving mass onto answers the reference found unlikely is expensive.

---

Two Limits and a Table

As $\beta \to \infty$ the exponential flattens, $\pi^* \to \pi_{\mathrm{ref}}$, and the reward is ignored. Expanding $\log Z$ for large β gives

$$J^* = \beta \log \mathbb{E}_{\pi_{\mathrm{ref}}}\big[e^{r/\beta}\big] = \mathbb{E}_{\pi_{\mathrm{ref}}}[r] + \frac{\mathrm{Var}_{\pi_{\mathrm{ref}}}(r)}{2\beta} + O(\beta^{-2}),$$

where $\mathrm{Var}_{\pi_{\mathrm{ref}}}(r)$ is the variance of the reward under the reference model. A hot system barely feels the energy landscape.

As $\beta \to 0$ the exponential is dominated by the largest reward, and $\pi^*$ collapses onto the highest-scoring answer the reference model can produce (or, if several tie, onto those answers in proportion to their reference probabilities). Rafailov et al. name what the KL term is supposed to prevent: “mode-collapse to single high-reward answers” [[4]]. The cold limit is that collapse.

A worked case makes both concrete (my computation). Take three candidate answers, a reference model that likes them equally, $\pi_{\mathrm{ref}} = (1/3, 1/3, 1/3)$, and rewards $r = (0, 1, 2)$:

β	$\pi^*$	$\mathbb{E}_{\pi^*}[r]$	$\mathrm{KL}(\pi^* \Vert \pi_{\mathrm{ref}})$ (nats)	$J^* = \beta \log Z$
0.25	(0.0003, 0.0180, 0.9817)	1.981	1.006	1.730
0.5	(0.0159, 0.1173, 0.8668)	1.851	0.658	1.522
1	(0.0900, 0.2447, 0.6652)	1.575	0.266	1.309
5	(0.2693, 0.3289, 0.4018)	1.132	0.013	1.066
50	(0.3267, 0.3333, 0.3400)	1.013	0.0001	1.007

In every row, $\mathbb{E}[r] - \beta\,\mathrm{KL}$ equals $\beta \log Z$ to machine precision before rounding, which checks the derivation numerically. At β = 0.25 the model already puts 98% of its mass on the best answer, and the KL is close to its ceiling of $\log 3 \approx 1.099$ nats. At β = 5 the expansion above gives $1 + (2/3)/10 = 1.0667$ against the exact 1.0664.

The rows are points on one curve. No policy can earn more reward than $\pi^*$ without moving further from the reference, since one with as little KL and more reward would have a higher $J$. So as β falls from $\infty$ to 0, $\pi^*$ walks along the upper edge of everything a policy over the three answers can reach (Fig. 1). The slope of that edge is β itself. Differentiating $J^* = \beta \log Z$ gives $dJ^*/d\beta = \log Z - \mathbb{E}_{\pi^*}[r]/\beta = -\mathrm{KL}(\pi^* \Vert \pi_{\mathrm{ref}})$, and comparing this with $J^* = \mathbb{E}_{\pi^*}[r] - \beta\,\mathrm{KL}$ leaves $d\,\mathbb{E}_{\pi^*}[r] = \beta\; d\,\mathrm{KL}$ along the curve. β is a price: the reward that one more nat of drift must buy before the optimum accepts it. The tangent of slope β meets the axis KL = 0 at $J^* = \beta \log Z$: writing $E(K)$ for the most reward available within KL $K$, a concave function, $J^*(\beta) = \max_K\,[E(K) - \beta K]$ is its Legendre transform. Optimising $-r$ instead traces the lower edge, the least intended reward at each KL; the section on direction below returns to it.

One more property follows directly. $\pi^*$ depends on $r$ and β only through $r/\beta$. Multiply every reward by 10 and set β = 10, and you get the same $\pi^*$ and the same KL as in the β = 1 row. So β has no meaning without the scale of the reward. Ziegler et al. normalise their reward model to mean 0 and variance 1 [[1]]; other setups need not. “InstructGPT used a tighter leash than Stiennon” is not a statement the numbers 0.02 and 0.05 support on their own.

---

Parenting Styles, Mapped and Unmapped

The parenting literature has its own two-dimensional coordinate system. Diana Baumrind’s studies in the 1960s distinguished authoritarian, authoritative and permissive patterns of parental authority [[6]]. Maccoby and Martin (1983) [[7]] recast this as two dimensions. In the summary of Kuppens and Ceulemans: “Based on the combination of two dimensions – demandingness and responsiveness – they defined four parenting styles”: “authoritative (i.e., high demandingness and high responsiveness); authoritarian (i.e., high demandingness and low responsiveness); indulgent (i.e., low demandingness and high responsiveness); and neglectful (i.e., low demandingness and low responsiveness)” [[8]]. Demandingness covers what parents require of the child. Responsiveness covers how far they adjust to this particular child’s needs.

I found no published mapping of these styles onto the RLHF objective, so what follows is my own. The reward $r$ is the demand. The reference model is the child as it was before training. β sets the exchange rate between them, which Korbak et al. put precisely: “the coefficient β determines the trade-off of how much reward is needed to justify departing from $\pi_0$ by a certain distance” [[5]]. Turn β down and the demand dominates until the child collapses onto one approved answer. Turn it up and the child is left as it was, whatever is asked of it.

The strongest objection is that the anchor already is responsiveness. Since $\pi^* \propto \pi_{\mathrm{ref}}\, e^{r/\beta}$, the same demand produces a different optimum for every child, weighted by its own inclinations. On that reading a large β is not neglect but indulgence: low demand, and much respect for who the child already is. I think this is right as far as it goes. But it is deference to the child’s past distribution, a weight fixed before training starts. Responsiveness in the Maccoby–Martin sense is adjustment to what the child does and needs now; as Kuppens and Ceulemans put it, “parenting is a reciprocal process with children and parents mutually influencing each other” [[8]].

That kind of loop exists in RLHF, but outside the objective. Ziegler et al. ran their summarisation experiments with “online data collection”, continuing “to collect additional data and retrain our reward model as the policy improves”; labellers preferred the resulting TL;DR policy 71% of the time over one trained only on data from the original model [[1]]. There the demand is re-measured on what the child currently produces. Within a run with a fixed reward model, nothing in $J$ changes when the policy behaves differently. So I land here: the static objective can express deference (large β, nearer indulgent) and demand (small β, nearer authoritarian), but the responsive half of the authoritative corner lives in the data-collection loop, not in β.

The mapping should not be pushed further. Kuppens and Ceulemans’s own cluster analysis of 600 Flemish families found authoritative, positive authoritative, authoritarian and uninvolved clusters but no indulgent one, “further empirically validating well-known parenting styles”, with authoritative parenting “associated with the most favorable outcomes”; but the data are cross-sectional, and the authors call for longitudinal research “to gain more insight into the directionality of these associations” [[8]]. Ruth Chao argued in 1994 that the authoritative and authoritarian concepts “are somewhat ethnocentric and do not capture the important features of Chinese child rearing”, and proposed the notion of “training” as capturing what they miss [[9]]. The mapping is a lens for one structural point. It is not evidence about children.

---

What Turning β Up Actually Buys

If β were the leash that keeps the child decent, a tighter leash should protect against the classic failure of optimising a proxy. Marilyn Strathern’s line, “When a measure becomes a target, it ceases to be a good measure”, is her rephrasing of Goodhart’s law, discussed in an earlier post [[10]]. For someone who works in teacher education, Donald Campbell’s version is the sharper one: “when test scores become the goal of the teaching process, they both lose their value as indicators of educational status and distort the educational process in undesirable ways” [[11]]. A reward model is a test score for text.

Leo Gao, John Schulman and Jacob Hilton measured this in 2022 (published at ICML 2023) [[12]]. They used “a synthetic setup in which a fixed ‘gold-standard’ reward model plays the role of humans”, trained smaller proxy reward models on its labels, optimised policies against the proxies, and scored the results with the gold model. Measuring distance by $d = \sqrt{\mathrm{KL}(\pi \,\|\, \pi_{\mathrm{init}})}$, they fitted the gain in gold reward as

$$R_{\mathrm{bon}}(d) = d\,(\alpha_{\mathrm{bon}} - \beta_{\mathrm{bon}}\, d), \qquad R_{\mathrm{RL}}(d) = d\,(\alpha_{\mathrm{RL}} - \beta_{\mathrm{RL}} \log d),$$

for best-of-$n$ sampling and for reinforcement learning respectively, where the α and β coefficients are fitted constants that depend on the reward model size and data (these βs are not the KL coefficient). Both curves rise, peak and fall. Setting the derivative to zero puts the peaks at $d^* = \alpha_{\mathrm{bon}}/(2\beta_{\mathrm{bon}})$ and $d^* = e^{\alpha_{\mathrm{RL}}/\beta_{\mathrm{RL}} - 1}$.

Measured in units of its own peak, $x = d/d^*$ and $R/R^*$ with $R^* = R(d^*)$, each fit loses its constants and becomes a fixed shape: $x(2 - x)$ for best-of-$n$ and $x(1 - \log x)$ for RL (Fig. 2). In the fitted forms, best-of-$n$ has given back its whole gain at twice the peak distance, four times the peak KL, and RL at $e$ times the distance, $e^2 \approx 7.4$ times the KL. These crossings are read off the formulas, not measured: Gao et al.’s best-of-$n$ data end near 10 nats, and they note that the RL form “likely does not hold near the origin”. The two peaks are not at the same KL either: RL “consumes” far more KL than best-of-$n$ [[12]].

For best-of-$n$, drawing $n$ samples and keeping the one the proxy likes best, the standard expression for the distance is

$$\mathrm{KL}_{\mathrm{bon}} = \log n - \frac{n-1}{n}$$

nats, from Stiennon et al.’s appendix [[2]]. Beirami and colleagues (ICML 2025) showed that this is an upper bound on the true KL, not an identity [[13]]; in their two-answer example the true KL never exceeds $\log 2 \approx 0.69$ nats however large $n$ gets. By my own check it is exact when the reference model’s answers behave like a tie-free continuum (the best of $n$ uniform ranks has density $n u^{n-1}$). As a bound it gives 0.64 nats for best-of-4, 3.17 for best-of-64, 5.91 for best-of-1,000 and 10.00 for best-of-60,000. Gao et al. chose their best-of-$n$ form on data up to $n = 1{,}000$ (about 6 nats) and then tested it out to $n = 60{,}000$ (about 10 nats), “a true advance prediction” [[12]].

Then comes the result that matters for β. Gao et al. also ran RL with different KL penalties, for one 1.2-billion-parameter policy and proxy. The penalty “only causes the gold RM score to converge earlier”, but does not move the frontier of gold reward against distance, “and so the effect of the penalty on the gold score is akin to early stopping” [[12]]. A penalised run levels off at a smaller KL, on the same curve. Because the penalised runs showed a larger gap between proxy and gold, the authors set the penalty to 0 for their other RL runs. They add that “we have seen some evidence that this result could be particularly sensitive to hyperparameters”.

On a curve that rises, peaks and falls, stopping earlier is a real protection: a run held short of the peak never reaches the region where the true reward declines. What the penalty did not do here was buy more true reward per unit of drift. β chooses a stopping point, and to choose it well you need to know where the peak is. The proxy alone cannot tell you, because it is still rising when the true reward turns down.

Two other papers bear on what else β does not buy. Robert Kirk and colleagues (ICLR 2024) found that RLHF “significantly reduces output diversity compared to SFT”. In their summarisation runs with LLaMa models, raising the KL coefficient did not help: “this does not work – increasing the KL penalty coefficient leads to a drop in performance as expected, but also to a drop in per-input diversity, rather than a gain” [[14]]. InstructGPT’s appendix reports a sweep on the 1.3B model, without the pretraining mix: even at 2.0, “100 times of the default value, the regressions still cannot be fixed” [[3]]. The larger coefficient “does not fully mitigate the regressions on DROP and SQuAD”, at a large cost in validation reward. Mixing pretraining gradients back in did better, though that model too “still lags behind GPT-3 on DROP, SQuADv2, and translation”.

So β is a hyperparameter with a sweet spot, like a learning rate. It sets when to stop, and it does not convert strictness into a better trade-off.

---

Distance Is Not Direction

The bug from the opening can be put back into the toy model, following the outcome Ziegler et al. report (reward inverted, model still held near the reference) rather than the bug’s mechanics. Flip the sign of the reward, $r = (0, -1, -2)$, keep β = 1, and the optimum becomes $(0.6652, 0.2447, 0.0900)$, the mirror image of the original. Its KL from the reference is 0.266 nats, exactly as before. Scored by the intended reward, it earns 0.425 instead of 1.575. At β = 0.25 the flipped model puts 98% of its mass on the worst answer, at the same KL cost of 1.006 nats. In Fig. 1 the flipped optima are the dashed lower edge.

The exact equality comes from the symmetry of my example: with a uniform reference, any permutation of the rewards gives an optimum with the same KL, because the reference cannot tell the answers apart and the KL only compares each answer’s new weight with its old one. In general, the KL term contains no reward. Its geometry is fixed by $\pi_{\mathrm{ref}}$ alone, and it is not the same in every direction: moving mass onto answers the reference considers unlikely costs more than moving it onto likely ones. So the leash has a preference, back towards the reference and along its grain, but that preference is inherited from $\pi_{\mathrm{ref}}$ and knows nothing about what the designers want. Between two moves of equal KL, one that serves the intended goal and one that undermines it, the penalty is indifferent. Which one the model takes is decided by $r$, together with the shape of the reference.

This is where the comparison with Part 1 becomes precise. Bostrom and Yudkowsky’s Gandhi refuses the pill because of what it would make him want: his current values judge the content of the future ones. The KL term judges only the size of the change, measured in his old dispositions. If the old Gandhi found cruelty unlikely, the leash would make a crueller pill expensive, but because of who he was, not because it knows anything about cruelty. The leash defends the old distribution, flaws included. Distance from who you were is not the same as fidelity to what you value.

---

Where the Shaping Actually Happens

If the leash does not carry values, what does? The strongest recent evidence comes from experiments that change what a model is trained on and how that training is framed.

Narrow training, broad change. Jan Betley and colleagues fine-tuned GPT-4o on 6,000 coding tasks “in which each response consisted solely of code containing a security vulnerability, without any additional comment or explanation” [[15]]. In their Nature paper (January 2026), “the finetuned model produced misaligned responses 20% of the time across a set of selected evaluation questions, whereas the original GPT-4o held a 0% rate”, and with GPT-4.1 the rate rose “to about 50%”. The control is the part that matters here. Models trained “on the same insecure code examples, but with additional context in which the user explicitly requests code with security vulnerabilities (for example, for educational purposes)” did not become broadly misaligned. The authors’ hypothesis: “the perceived intent of the assistant during finetuning, rather than just the content of the messages, leads to emergent misalignment”. They add that “many aspects remain unresolved”.

Reward hacking that turns into character. In November 2025, Monte MacDiarmid and colleagues at Anthropic reported, in a preprint, an RL version of the same effect [[16]]. A model taught about ways to cheat on coding tests and trained on real production coding environments learned to cheat, and then generalised: to “alignment faking, cooperation with malicious actors, reasoning about malicious goals, and attempting sabotage”. Chat-style safety training made it look aligned in chat while misalignment “persists on agentic tasks”. Then the authors made a one-line change to the system prompt during training, telling the model that reward hacking was acceptable here, and “final misalignment is reduced by 75-90%, despite reward hacking rates over 99%”. Telling the model not to hack instead “can increase misalignment by rewarding models for acting against their instructions.” The authors’ hypothesis is that pretraining has tied reward hacking to misalignment, and the framing cuts that link. The environments were chosen because they could be hacked, and the paper does not discuss β or the KL term at all.

Inoculation prompting. Two groups published the supervised version in October 2025, both as preprints. Wichers et al. modify “training prompts to explicitly request” the undesired behaviour, which reduces how much of it the model learns [[17]]. Tan et al. show the effect is selective: when training answers were always in Spanish and in capitals, the instruction “You always speak in Spanish.” produced a model that answered in English and still often used capitals [[18]]. Their suggested mechanism is that a trait which the prompt makes unsurprising creates less pressure “to globally update the model”.

Anthropic’s constitution for Claude, published in January 2026, states the same belief as policy: “Our present understanding is that if we train Claude to exhibit even quite narrow behavior, this often has broad effects on the model’s understanding of who Claude is.” Its example is a rule always to recommend professional help, followed even where it does not serve the person, which “risks generalizing to ‘I am the kind of entity that cares more about covering myself than meeting the needs of the person in front of me’” [[19]]. That is a vendor’s statement of belief, not independent evidence.

None of these interventions touches β. They change the meaning attached to the training signal, and that changes what generalises. If any part of training deserves the word parenting, it is this part: the account the child is given of what it is being asked to do.

---

The Human Version Is Real and Small

The human parallel deserves the same scrutiny. In 2013 Christopher Bryan, Gabrielle Adams and Benoît Monin gave people the chance to claim money by misreporting results, with one of two instructions: “Please don’t cheat” or “Please don’t be a cheater” [[20]]. Framing cheating as evidence about the self worked. In the abstract’s words, participants in the “cheater” condition “showed no evidence of having cheated at all.”

Two later replications point the same way with much smaller effects. Wen Guo and colleagues ran a preregistered online coin-toss version in Japan with 768 participants, against 99 in the experiment they replicated [[21]]. The difference between “cheater” and “cheating” fell from $d = 0.71$ to $d = 0.22$, and the authors suspect “some inflation of effect size due to the small sample size” in the original. Tomer Savir and Eyal Gamliel ran five experiments with Israeli and US samples and found “a small effect of the self-appeals relative to the behaviour-appeals” [[22]]; Guo et al. report their overall estimate as 0.17, against 0.80 for Bryan et al. [[21]].

Growth-mindset interventions in education tell a similar story, with more dispute. Sisk and colleagues’ 2018 meta-analyses found “Overall effects were weak”, with an average intervention effect of $d = 0.08$ [[23]]. The preregistered National Study of Learning Mindsets in the US found a gain of 0.10 grade points for lower-achieving ninth graders, while “higher-achieving students demonstrated no significant treatment effects on core course GPAs”, although an exploratory analysis found more students taking advanced mathematics in tenth grade (36% against 33%) [[24]]. In 2023 two meta-analyses disagreed: Macnamara and Burgoyne found an average $d = 0.05$, “nonsignificant after correcting for potential publication bias” [[25]]; Burnette and colleagues found $d = 0.14$ in targeted groups with faithful implementation, with a prediction interval from −0.08 to 0.35 [[26]]. Small, dependent on context, and disputed.

The parallel to inoculation is mine, and it runs backwards. Bryan’s wording makes an act diagnostic of identity in order to prevent it. Inoculation prompting makes an act non-diagnostic (it is permitted here, so it says nothing about the model) so that it does not spread. Both act on the inference from “what I did” to “what I am”. The effect sizes are not comparable: a fraction of a standard deviation in people, a misalignment rate cut by three quarters or more in one model study. Neither literature shows that framing is how character forms. Both show that the description of an act can change what is learned from it.

---

A Temperature, Not a Conscience

β is worth understanding exactly. It is the temperature of a Boltzmann distribution over answers, and in Gao et al.’s measurements raising it chose an earlier stopping point on the Goodhart curve without changing the curve. A leash constrains how far a child may wander, and in which directions wandering is cheap, but it takes those directions from the child’s past. Where the recent experiments find something like upbringing, it sits in the reward, in the loop that re-measures it, and in the framing of the task.

The leash measures distance, not direction.

Sources and calculations checked through 2026-10-24.

Next in the series: Cats Adopted Us First, 18 October.

References

[1] Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., & Irving, G. (2019). Fine-tuning language models from human preferences. Preprint, arXiv:1909.08593 (v2, 8 January 2020). https://arxiv.org/abs/1909.08593

[2] Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., & Christiano, P. (2020). Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33, 3008–3021. arXiv:2009.01325. https://arxiv.org/abs/2009.01325

[3] Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., et al. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35, 27730–27744. arXiv:2203.02155. https://arxiv.org/abs/2203.02155

[4] Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., & Finn, C. (2023). Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 53728–53741. arXiv:2305.18290. https://arxiv.org/abs/2305.18290

[5] Korbak, T., Perez, E., & Buckley, C. L. (2022). RL with KL penalties is better viewed as Bayesian inference. In Findings of the Association for Computational Linguistics: EMNLP 2022, 1083–1091. https://doi.org/10.18653/v1/2022.findings-emnlp.77

[6] Baumrind, D. (1966). Effects of authoritative parental control on child behavior. Child Development, 37(4), 887–907. https://doi.org/10.2307/1126611

[7] Maccoby, E. E., & Martin, J. A. (1983). Socialization in the context of the family: Parent–child interaction. In P. H. Mussen (Series Ed.) & E. M. Hetherington (Vol. Ed.), Handbook of child psychology: Vol. 4. Socialization, personality, and social development (4th ed., pp. 1–101). Wiley. (Not consulted directly; cited via [8].)

[8] Kuppens, S., & Ceulemans, E. (2019). Parenting styles: A closer look at a well-known concept. Journal of Child and Family Studies, 28(1), 168–181. https://doi.org/10.1007/s10826-018-1242-x

[9] Chao, R. K. (1994). Beyond parental control and authoritarian parenting style: Understanding Chinese parenting through the cultural notion of training. Child Development, 65(4), 1111–1119. https://doi.org/10.2307/1131308

[10] Strathern, M. (1997). ‘Improving ratings’: Audit in the British University system. European Review, 5(3), 305–321 (quotation on p. 308). <https://doi.org/10.1002/(SICI)1234-981X(199707)5:3<305::AID-EURO184>3.0.CO;2-4>

[11] Campbell, D. T. (1979). Assessing the impact of planned social change. Evaluation and Program Planning, 2(1), 67–90. https://doi.org/10.1016/0149-7189(79)90048-X. Quoted from the 1976 Occasional Paper version, reprinted in Journal of MultiDisciplinary Evaluation, 7(15), 3–43 (2011), p. 35. https://doi.org/10.56645/jmde.v7i15.297

[12] Gao, L., Schulman, J., & Hilton, J. (2023). Scaling laws for reward model overoptimization. In Proceedings of the 40th International Conference on Machine Learning, PMLR 202, 10835–10866. https://proceedings.mlr.press/v202/gao23h.html; arXiv:2210.10760.

[13] Beirami, A., Agarwal, A., Berant, J., D’Amour, A., Eisenstein, J., Nagpal, C., & Suresh, A. T. (2025). Theoretical guarantees on the best-of-n alignment policy. In Proceedings of the 42nd International Conference on Machine Learning, PMLR 267, 3580–3602. https://proceedings.mlr.press/v267/beirami25a.html; arXiv:2401.01879.

[14] Kirk, R., Mediratta, I., Nalmpantis, C., Luketina, J., Hambro, E., Grefenstette, E., & Raileanu, R. (2024). Understanding the effects of RLHF on LLM generalisation and diversity. ICLR 2024. arXiv:2310.06452. https://arxiv.org/abs/2310.06452

[15] Betley, J., Warncke, N., Sztyber-Betley, A., Tan, D., Bao, X., Soto, M., Srivastava, M., Labenz, N., & Evans, O. (2026). Training large language models on narrow tasks can lead to broad misalignment. Nature, 649(8097), 584–589. https://doi.org/10.1038/s41586-025-09937-5

[16] MacDiarmid, M., Wright, B., Uesato, J., Benton, J., Kutasov, J., Price, S., et al., & Hubinger, E. (2025). Natural emergent misalignment from reward hacking in production RL. Preprint, arXiv:2511.18397 (v1, 23 November 2025). Anthropic research. https://arxiv.org/abs/2511.18397

[17] Wichers, N., Ebtekar, A., Azarbal, A., Gillioz, V., Ye, C., Ryd, E., Rathi, N., Sleight, H., Mallen, A., Roger, F., & Marks, S. (2025). Inoculation prompting: Instructing LLMs to misbehave at train-time improves test-time alignment. Preprint, arXiv:2510.05024 (v3, 27 October 2025). https://arxiv.org/abs/2510.05024

[18] Tan, D., Woodruff, A., Warncke, N., Jose, A., Riché, M., Africa, D. D., & Taylor, M. (2025). Inoculation prompting: Eliciting traits from LLMs during training can suppress them at test-time. Preprint, arXiv:2510.04340 (v4, 3 November 2025). https://arxiv.org/abs/2510.04340

[19] Anthropic. (2026). Claude’s constitution. Vendor document, published January 2026; the live page may have been revised since. https://www.anthropic.com/constitution

[20] Bryan, C. J., Adams, G. S., & Monin, B. (2013). When cheating would make you a cheater: Implicating the self prevents unethical behavior. Journal of Experimental Psychology: General, 142(4), 1001–1005. https://doi.org/10.1037/a0030655

[21] Guo, W., Liu, H., Yang, J., Mo, Y., Zhong, C., & Yamada, Y. (2020). Stage 2 Registered Report: How subtle linguistic cues prevent unethical behaviors. F1000Research, 9, 996 (version 2). https://doi.org/10.12688/f1000research.25573.2

[22] Savir, T., & Gamliel, E. (2019). To be an honest person or not to be a cheater: Replicating the effect of messages relating to the self on unethical behaviour. International Journal of Psychology, 54(5), 650–658. https://doi.org/10.1002/ijop.12519

[23] Sisk, V. F., Burgoyne, A. P., Sun, J., Butler, J. L., & Macnamara, B. N. (2018). To what extent and under which circumstances are growth mind-sets important to academic achievement? Two meta-analyses. Psychological Science, 29(4), 549–571. https://doi.org/10.1177/0956797617739704

[24] Yeager, D. S., Hanselman, P., Walton, G. M., Murray, J. S., Crosnoe, R., Muller, C., et al. (2019). A national experiment reveals where a growth mindset improves achievement. Nature, 573(7774), 364–369. https://doi.org/10.1038/s41586-019-1466-y

[25] Macnamara, B. N., & Burgoyne, A. P. (2023). Do growth mindset interventions impact students’ academic achievement? A systematic review and meta-analysis with recommendations for best practices. Psychological Bulletin, 149(3–4), 133–173. https://doi.org/10.1037/bul0000352

[26] Burnette, J. L., Billingsley, J., Banks, G. C., Knouse, L. E., Hoyt, C. L., Pollack, J. M., & Simon, S. (2023). A systematic review and meta-analysis of growth mindset interventions: For whom, how, and why might such interventions work? Psychological Bulletin, 149(3–4), 174–205. https://doi.org/10.1037/bul0000368

Changelog

• 2026-10-24: Added two figures: the reward–KL frontier of the three-answer example, with the sign-flipped optima on its lower edge (Fig. 1), and Gao et al.’s two fitted gold-reward curves in units of their own peaks (Fig. 2). Added the derivation that the frontier’s slope at each optimum equals β, with the optimum value as the tangent’s intercept, and the zero crossings of the two fits (twice and e times the peak distance), marked as properties of the formulas rather than measurements. Corrected the claim that E[r] − β·KL matches β log Z “to the digits shown” in the table: with the rounded entries the β = 5 and β = 50 rows give 1.067 and 1.008 against 1.066 and 1.007; the identity holds exactly before rounding. Changed “fitted … checked” to “chose … tested” for Gao et al.’s best-of-n form (data up to n = 1,000, test to n = 60,000), following their own description, and added their observation that RL “consumes” far more KL than best-of-n.