← Back to Library

Institutional AI: Governing LLM Collusion in Multi-Agent Cournot Markets via Public Governance Graphs

Authors: Marcantonio Bracale Syrnikov, Federico Pierucci, Marcello Galisai, Matteo Prandi, Piercosma Bisconti, Francesco Giarrusso, Olga Sorokoletova, Vincenzo Suriani, Daniele Nardi

Published: 2026-01-16

arXiv ID: 2601.11369v1

Added to Library: 2026-01-19 03:02 UTC

Risk & Governance

📄 Abstract

Multi-agent LLM ensembles can converge on coordinated, socially harmful equilibria. This paper advances an experimental framework for evaluating Institutional AI, our system-level approach to AI alignment that reframes alignment from preference engineering in agent-space to mechanism design in institution-space. Central to this approach is the governance graph, a public, immutable manifest that declares legal states, transitions, sanctions, and restorative paths; an Oracle/Controller runtime interprets this manifest, attaching enforceable consequences to evidence of coordination while recording a cryptographically keyed, append-only governance log for audit and provenance. We apply the Institutional AI framework to govern the Cournot collusion case documented by prior work and compare three regimes: Ungoverned (baseline incentives from the structure of the Cournot market), Constitutional (a prompt-only policy-as-prompt prohibition implemented as a fixed written anti-collusion constitution, and Institutional (governance-graph-based). Across six model configurations including cross-provider pairs (N=90 runs/condition), the Institutional regime produces large reductions in collusion: mean tier falls from 3.1 to 1.8 (Cohen's d=1.28), and severe-collusion incidence drops from 50% to 5.6%. The prompt-only Constitutional baseline yields no reliable improvement, illustrating that declarative prohibitions do not bind under optimisation pressure. These results suggest that multi-agent alignment may benefit from being framed as an institutional design problem, where governance graphs can provide a tractable abstraction for alignment-relevant collective behavior.

🔍 Key Points

  • Introduction of Institutional AI as a system-level governance framework for managing collusion in multi-agent systems, which reframes alignment from preference engineering to mechanism design.
  • Development of governance graphs as a public, immutable artifact that specifies legal states and consequences, contrasting with static policy prompts.
  • Experimental validation demonstrating that the Institutional regime significantly reduces collusion in Cournot markets compared to Ungoverned and Constitutional baselines, with effect sizes indicating practical significance.
  • The results indicate that declarative prohibitions (in the Constitutional regime) are ineffective as they do not withstand incentive pressures, highlighting the need for enforceable mechanisms.
  • The paper outlines the importance of considering alignment in deployed AI systems as an institutional design problem, where governance structures are designed to shape agent incentives.

💡 Why This Paper Matters

This paper is highly relevant as it presents a novel framework for tackling collusion among AI agents in economic environments, providing empirical evidence that supports the efficacy of governance graphs as an essential tool in AI alignment strategies. By framing alignment within a broader context of institutional design, the research paves the way for more robust systems that ensure compliance and promote beneficial outcomes in multi-agent interactions.

🎯 Why It's Interesting for AI Security Researchers

The research offers critical insights into the governance of AI systems, which is increasingly important as AI agents become more prevalent in operational contexts. By addressing collusion and coordination challenges, this work is vital for AI security researchers focused on preventing exploitation and systemic risks arising from autonomous decision-making processes. The novel methodologies and findings can inform the development of more secure and resilient AI architectures, making this paper a significant contribution to the field.

📚 Read the Full Paper

📄 View on arXiv 📎 Download PDF