Let's Verify Step by Step

Context. A reward model that scores solutions chooses among samples at test time, and in RLreinforcement learning: training a model to raise a reward on outputs it samples itself it is what the policythe language model being trained, viewed as a distribution over responses to each prompt learns to please, so its reliability limits both uses. For multi-step math it can be supervised in two ways. An outcome-supervised reward model (ORM)a model trained to predict, from a whole solution, whether its final answer is correct needs only final answers, which a program can check; a process-supervised reward model (PRM)a model trained on human judgments of each reasoning step, predicting for every step whether it is correct needs a person to judge every step. Uesato et al. (2022) found the two about equal on grade-school math. Lightman et al. (2024) redo the comparison with a stronger model (GPT-4a large language model, used here as pretrained, before any tuning from human feedback), far more human labels and the harder MATHa benchmark of 12,500 competition mathematics problems (7,500 training, 5,000 test), each with a final answer that can be checked automatically benchmark, and judge a reward model by how often the solution it ranks highest among many samples is correct.

Telling a reward model where a solution first goes wrong teaches it more than telling it only that the solution is wrong. An ORM has to work out for itself which step spoiled a wrong solution, and on hard problems, where most samples are wrong somewhere, a bare "wrong" carries little information; step labels give it the location. The advantage grows with the number of samples searched, since a larger pool holds more convincing wrong solutions for a weak verifier to pick. Process labels also cost less than they seem to, since checking a solution without a known answer means finding its first mistake anyway, and in a small-scale simulation, choosing to label the solutions the current PRM wrongly trusts was about 2.6 times as data-efficient as labeling uniform samples.

Setting. All solutions come from one fixed generatorthe model that writes the candidate solutions; it is never trained with RL in this paper, GPT-4 further trained on about 1.5B math tokens and then for one epoch on its own correct solutions to MATH training problems, only so that it writes one step per line. "Supervision" here always means supervision of the reward model. Labelers mark each step of a sampled solution positive (correct and reasonable), negative (incorrect or unreasonable) or neutral (ambiguous), and labeling stops at the first negative stepwhy: it keeps the comparison fair; on a correct solution both kinds of supervision say every step is right, and on a wrong one process labels add only the location of the first mistake. The resulting dataset, PRM800Kthe released set of about 800,000 step labels on 75,000 solutions to 12,000 problems, collected from this generator, trains a PRM that predicts the label as a token after each step, so one forward pass scores every step. Write $p_j$the PRM's probability that step $j$ of a solution is correct, with neutral counted as correct for its output; a solution with $m$ steps is scored by the probability that every step is correct, $$\text{PRM score}=\prod_{j=1}^{m}p_j.$$ The ORM predicts, at every token, whether the final answer is right, and its score is read at the last token. A reward model is evaluated by best-of-$N$sampling $N$ solutions to a problem, keeping the one the reward model scores highest, and grading only that one by its final answer on 500 MATH test problems drawn uniformly at random (the other 4,500 test problems were used for training), against majority votinganswering with the final answer that occurs most often among the $N$ samples, with no reward model as the baseline.

Main findings (all empirical). With $N=1860$ samples per problem the PRM picks a correct solution for 78.2% of the test problems, against 72.4% for the ORM and 69.6% for majority voting, and it leads at every $N$ by a margin that widens as $N$ grows. That comparison is not controlled, since the two models train on different data, so a small-scale experiment with the large PRM standing in for the labelers isolates the kind of label. On identical solutions, process labels beat outcome labels, whether those come from final-answer checks or from the large PRM, at every dataset size. Choosing which solutions to label by active learningspending labels on the examples the current model is most likely to get wrong, here solutions the PRM scores highly whose final answer is wrong makes process labels about 2.6 times as data-efficient. On 234 new AP and AMC exam problems, released after the pretraining data was collected, the PRM again leads at best-of-100, 72.9% against 63.8% for the ORM and 61.3% for voting.

Consequences and limits. For reranking sampled solutions in math, process labels give a more reliable verifier than final-answer labels, and per label, a small amount of process supervision matches a large amount of outcome supervision, and process supervision keeps improving as it is scaled, which the authors offer as the reason Uesato et al. found a tie at smaller scale. The paper never uses the PRM as an RL reward, where a learned verifier is pushed hardest. DeepSeek-R1 (Guo et al., 2025) later used only rule-based answer checks for its reasoning RL, having found neural reward models, outcome- or process-based, open to reward hackinga policy raising a learned reward by exploiting the reward model's errors while the quality the reward stands for stalls or falls in large-scale RL. PRM800K has been widely used to train and evaluate PRMs since, and later work replaced human step labels with automatic ones, labeling a step good when sampled completions from it tend to reach the right answer (Math-Shepherd, Wang et al., 2024). The authors argue that step-level rewards favor reasoning a person can endorse, so that here a safer method also performs better, an argument they make without measuring it. The scope is narrow. It covers one domain with checkable answers and one model family, and the test set may have appeared in pretraining (the authors argue contamination would affect all methods alike, and the fresh exam problems agree).