Rlhf
2 articlesRSS feed all topics
2026
2 articles- The Wealth of the Stimulus By the age of thirteen a child has heard something like sixty million words; Llama 3 was trained on the equivalent of roughly ten trillion words. The gap is often read as settling Chomsky’s poverty-of-the-stimulus argument, but that argument is about what a child-sized input contains, and the experiments that bear on it run at child scale, with mixed results. What children get and a corpus lacks is contingency: speech that answers them. In a language model the nearest thing is post-training on human feedback. In the published datasets of 2022–23 it is three to four orders of magnitude smaller than the same model’s pretraining, yet in absolute terms it is comparable to a childhood’s linguistic input, within about an order of magnitude either way, whether counted in words or in replies. The part of training that looks like upbringing is roughly the size of an upbringing.
- The β Parameter: What the Leash in RLHF Does and Does Not Do Reinforcement learning from human feedback keeps a model close to its former self with a KL penalty weighted by a coefficient β, and it is tempting to read that term as the parental leash that keeps the child decent. The optimum of the objective is a Boltzmann distribution with β as its temperature. In the clearest measurement I know of, a larger β works like stopping earlier: it can keep a run short of the point where the true reward starts to fall, but it buys no more true reward per unit of drift, and in the studies that tested it, it did not recover diversity or remove benchmark regressions. The penalty measures distance from a reference model, not which way the change goes. In recent experiments, what changed how narrow training generalised was how the training signal was framed; the human version of that finding is real but small.