Reflexion: Language Agents with Verbal Reinforcement Learning

Shinn et al.

Reflexion, introduced by Shinn et al., is a framework that reinforces language agents through verbal feedback rather than gradient updates or weight changes. Agents verbally reflect on task outcomes, storing self-generated linguistic summaries in an episodic memory buffer to improve performance across subsequent trials in decision-making, reasoning, and coding tasks.

Read the paper

Key takeaways

Reflexion replaces traditional scalar reward signals with verbal self-reflection, converting environment feedback into natural language summaries stored in long-term memory, enabling improvement without any model fine-tuning.

The framework uses three cooperating components: an Actor that generates actions, an Evaluator that scores trajectories, and a Self-Reflection model that produces actionable verbal feedback from failures.

On the HumanEval coding benchmark, Reflexion achieves 91% pass@1 accuracy, surpassing the prior state-of-the-art GPT-4 result of 80%.

In sequential decision-making on AlfWorld, Reflexion agents improve over strong baselines by an absolute 22% across 12 iterative learning trials, and improve HotPotQA reasoning performance by 20%.

Reflexion is compatible with multiple feedback signal types, including binary environment rewards, heuristic rules, and self-generated unit tests, making it applicable across diverse task domains.

A key limitation is reliance on the LLM's own self-evaluation quality, and memory is bounded by a sliding window of recent experiences to respect context length constraints.

Reflections

Reflexion is the cheapest kind of learning I know: write down, in plain language, why the last attempt failed, keep it in memory, and do better on the next trial without touching a single weight. That is the self-heal loop I want, and it runs at inference time instead of in a training run. But it carries its own warning. The whole thing rests on the quality of the self-evaluation; a model that can’t tell it failed will happily reflect its way deeper into the same hole, so the scaffold has to feed it ground truth the reflection can’t fake. And the reflection itself was never the point. Storing it and pulling it back up next time is what matters; without the episodic memory it’s just a mea culpa, and with it, failure compounds into competence. Cheap to run, dependent on an honest critic, and worthless without a memory to hold it.