Reinforcement-Learning | Sebastian J. Spicker

2026

1 article

15 Oct 2026 The β Parameter: What the Leash in RLHF Does and Does Not Do Reinforcement learning from human feedback keeps a model close to its former self with a KL penalty weighted by a coefficient β, and it is tempting to read that term as the parental leash that keeps the child decent. The optimum of the objective is a Boltzmann distribution with β as its temperature. In the clearest measurement I know of, a larger β works like stopping earlier: it can keep a run short of the point where the true reward starts to fall, but it buys no more true reward per unit of drift, and in the studies that tested it, it did not recover diversity or remove benchmark regressions. The penalty measures distance from a reference model, not which way the change goes. In recent experiments, what changed how narrow training generalised was how the training signal was framed; the human version of that finding is real but small. raising-machinesaialignment