Back to the shelf
Cover of Mixture-of-Agents Enhances Large Language Model Capabilities

Reading

Mixture-of-Agents Enhances Large Language Model Capabilities

Wang et al.

Artificial Intelligence Computer Science

This paper from researchers at Together AI, Duke University, and Stanford University proposes Mixture-of-Agents (MoA), a layered architecture in which multiple LLM agents at each layer independently generate responses that are passed as auxiliary context to agents in the next layer for iterative refinement. MoA exploits a documented collaborativeness property of LLMs, whereby models produce better outputs when conditioned on responses from other models, and achieves state-of-the-art performance on multiple alignment benchmarks.

Read the paper

Key takeaways

  • The paper identifies and empirically documents a collaborativeness property of LLMs, wherein models across a range of capabilities consistently improve their output quality when provided with responses generated by other models, even when those reference responses are of lower quality than what the model could produce independently.
  • MoA organizes multiple LLM agents into proposer and aggregator roles: proposers generate diverse reference responses and aggregators synthesize them into a single higher-quality output, with models such as GPT-4o, Qwen1.5, and LLaMA-3 performing well in both roles while WizardLM excelled as a proposer but underperformed as an aggregator.
  • Using only open-source models as proposers with Qwen1.5-110B-Chat as the aggregator, MoA achieves an LC win rate of 65.1% on AlpacaEval 2.0, surpassing GPT-4 Omni at 57.5% by 7.6 percentage points, establishing a new state-of-the-art with open-source models alone.
  • On MT-Bench, MoA with GPT-4o as the final aggregator achieves an average score of 9.25, placing it at the top of the leaderboard and exceeding GPT-4 Turbo and GPT-4 Omni.
  • A cost-efficiency analysis shows that MoA-Lite, using only 2 MoA layers, matches GPT-4o in cost while achieving a 1.8% improvement in LC win rate on AlpacaEval 2.0, and outperforms GPT-4 Turbo by approximately 4% while being more than twice as cost-effective.
  • Performance improves monotonically with the number of proposer agents per layer, and using diverse heterogeneous models as proposers consistently yields better results than using the same model multiple times at a fixed temperature, confirming that model diversity is a key driver of MoA quality gains.

Reflections

Mixture-of-Agents is self-consistency scaled up a level: instead of sampling many paths from one model, you sample from many different models and synthesize. The result that matters to me is that a set of open-source proposers with a good aggregator beat GPT-4 alone. That is close to the thesis of how I’ve built things. You rarely need the single best model; you need diverse-enough contributors and something competent to reconcile them. The recurring lesson, the same one from self-consistency, is that diversity beats repetition: heterogeneous models outperform the same model run many times, because sameness has nothing to disagree about. The catch is the aggregator. Being a good proposer and a good synthesizer are different skills, some models are only one, and the whole system is capped by how well the reconciler tells signal from noise. Ensembling is only as smart as its editor.