Want to read

Beacon: Knowing When and How to Perform Agentic Visual Reasoning

Artificial Intelligence Computer Science

This paper introduces Beacon, an agentic visual reasoning model addressing limitations in existing multimodal large language models. The authors define Mode Adaptiveness and Tool Effect to evaluate tool usage. Beacon employs supervised fine-tuning and reinforcement learning with adaptive rewards and hint-guided expansion. Results across thirteen benchmarks demonstrate superior performance, improved adaptiveness, and a net positive benefit from tool use compared to prior methods.

Read the paper

Key takeaways

Existing agentic visual reasoning models exhibit limited Mode Adaptiveness and often suffer from tool-induced errors on easy problems.
Beacon utilizes supervised fine-tuning and reinforcement learning to improve tool-invocation adaptiveness and tool-use capability.
The model employs a Necessity-Aware Adaptive Reward to encourage tool-free solutions when successful and tool use when necessary.
A Hint-Guided Capability Expansion mechanism recovers learning signals from failed rollouts to extend capabilities on hard problems.
Beacon achieves the highest average score among evaluated open-source models and ranks first on eleven out of thirteen benchmarks.
The model improves average tool-available accuracy by 1.96 points and achieves a tool-gain minus tool-harm score of positive 3.14 points.

Reflections

No notes on this one yet. I add reflections as I finish or revisit a paper.