Read

Let's Verify Step by Step

Lightman et al.

Artificial Intelligence Computer Science

This OpenAI paper by Lightman et al. investigates whether process supervision, which provides feedback on each intermediate reasoning step, produces more reliable reward models than outcome supervision, which only evaluates final answers. Testing on the MATH dataset, the authors find that a process-supervised reward model solves 78.2% of problems from a representative test subset, outperforming outcome-supervised models, and that active learning improves data efficiency by approximately 2.6 times. The authors also release PRM800K, a dataset of 800,000 step-level human feedback labels.

Read the paper

Key takeaways

Process-supervised reward models (PRMs) significantly outperform outcome-supervised reward models (ORMs) on the challenging MATH benchmark, with the best PRM solving 78.2% of problems compared to 72.4% for the best ORM in a best-of-1860 evaluation.
Outcome supervision can reward incorrect reasoning that reaches a correct final answer, a flaw process supervision avoids by pinpointing the exact location of errors in a solution chain.
Active learning, by strategically surfacing convincing wrong-answer solutions to human labelers rather than sampling uniformly, yields approximately a 2.6 times improvement in data efficiency for training PRMs.
A large reward model can reliably approximate human supervision for smaller reward models, enabling large-scale ablation studies that would otherwise be infeasible due to the cost of human feedback.
The PRM800K dataset, containing 800,000 step-level human feedback labels across 75,000 solutions to 12,000 problems, is released publicly to support further research into process supervision.
Process supervision also carries alignment advantages: it is easier for humans to interpret and more directly rewards models for following a human-endorsed chain-of-thought, discouraging models from using incorrect reasoning to reach correct answers.

Reflections

This one lands for me because it names a failure I care about: a model that reaches the right answer through broken logic isn’t correct, it’s lucky, and outcome supervision can’t tell the difference. Rewarding the reasoning step by step is the gap between a system that thinks and one that guesses well, and I would rather ship the former even when the latter scores the same today. There’s an alignment dividend too, since rewarding a human-endorsed chain discourages the model from fabricating a path to an answer it already knows. But the part I respect most is the least glamorous. The real deliverable is 800,000 step-level human labels. Everyone wants the method; almost no one wants to pay for the labels that make it work, and that lesson generalizes far past reasoning models. The expensive, boring dataset is usually the moat, not the algorithm.