Back to the shelf
Cover of Mixture-of-Agents Enhances Large Language Model Capabilities

Reading

Mixture-of-Agents Enhances Large Language Model Capabilities

Wang et al.

Artificial Intelligence Computer Science

This paper from researchers at Together AI, Duke University, and Stanford University proposes Mixture-of-Agents (MoA), a layered architecture in which multiple LLM agents at each layer independently generate responses that are passed as auxiliary context to agents in the next layer for iterative refinement. MoA exploits a documented collaborativeness property of LLMs, whereby models produce better outputs when conditioned on responses from other models, and achieves state-of-the-art performance on multiple alignment benchmarks.

Key takeaways

  • The paper identifies and empirically documents a collaborativeness property of LLMs, wherein models across a range of capabilities consistently improve their output quality when provided with responses generated by other models, even when those reference responses are of lower quality than what the model could produce independently.
  • MoA organizes multiple LLM agents into proposer and aggregator roles: proposers generate diverse reference responses and aggregators synthesize them into a single higher-quality output, with models such as GPT-4o, Qwen1.5, and LLaMA-3 performing well in both roles while WizardLM excelled as a proposer but underperformed as an aggregator.
  • Using only open-source models as proposers with Qwen1.5-110B-Chat as the aggregator, MoA achieves an LC win rate of 65.1% on AlpacaEval 2.0, surpassing GPT-4 Omni at 57.5% by 7.6 percentage points, establishing a new state-of-the-art with open-source models alone.
  • On MT-Bench, MoA with GPT-4o as the final aggregator achieves an average score of 9.25, placing it at the top of the leaderboard and exceeding GPT-4 Turbo and GPT-4 Omni.
  • A cost-efficiency analysis shows that MoA-Lite, using only 2 MoA layers, matches GPT-4o in cost while achieving a 1.8% improvement in LC win rate on AlpacaEval 2.0, and outperforms GPT-4 Turbo by approximately 4% while being more than twice as cost-effective.
  • Performance improves monotonically with the number of proposer agents per layer, and using diverse heterogeneous models as proposers consistently yields better results than using the same model multiple times at a fixed temperature, confirming that model diversity is a key driver of MoA quality gains.

Reflections

No notes on this one yet. I add reflections as I finish or revisit a book.