← Back to Library

compar:IA: The French Government's LLM arena to collect French-language human prompts and preference data

Authors: Lucie Termignon, Simonas Zilinskas, Hadrien Pélissier, Aurélien Barrot, Nicolas Chesnais, Elie Gavoty

Published: 2026-02-06

arXiv ID: 2602.06669v1

Added to Library: 2026-02-09 03:03 UTC

📄 Abstract

Large Language Models (LLMs) often show reduced performance, cultural alignment, and safety robustness in non-English languages, partly because English dominates both pre-training data and human preference alignment datasets. Training methods like Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO) require human preference data, which remains scarce and largely non-public for many languages beyond English. To address this gap, we introduce compar:IA, an open-source digital public service developed inside the French government and designed to collect large-scale human preference data from a predominantly French-speaking general audience. The platform uses a blind pairwise comparison interface to capture unconstrained, real-world prompts and user judgments across a diverse set of language models, while maintaining low participation friction and privacy-preserving automated filtering. As of 2026-02-07, compar:IA has collected over 600,000 free-form prompts and 250,000 preference votes, with approximately 89% of the data in French. We release three complementary datasets -- conversations, votes, and reactions -- under open licenses, and present initial analyses, including a French-language model leaderboard and user interaction patterns. Beyond the French context, compar:IA is evolving toward an international digital public good, offering reusable infrastructure for multilingual model training, evaluation, and the study of human-AI interaction.

🔍 Key Points

  • Introduction of AgentDyn as a dynamic open-ended benchmark for evaluating prompt injection attacks on AI agent security systems.
  • Identification of three critical flaws in existing benchmarks: lack of dynamic tasks, absence of helpful instructions, and overly simplistic user tasks.
  • AgentDyn features 60 challenging tasks and 560 injection test cases that require dynamic planning and incorporate helpful third-party instructions.
  • Evaluation of ten state-of-the-art defenses revealed that most are insufficient for real-world scenarios, suffering from issues like over-defense and inability to distinguish between benign and harmful instructions.
  • Insights from AgentDyn challenge the effectiveness of existing defenses and encourage the development of more robust agent security strategies.

💡 Why This Paper Matters

This paper is significant as it addresses the growing concern over prompt injection attacks in AI agent systems by introducing a novel and comprehensive benchmark, AgentDyn. It reveals inherent limitations in current evaluation methods, ultimately emphasizing the need for improved defenses against such attacks. The findings not only contribute to the academic discourse but also have practical implications for deploying safer AI systems in real-world applications.

🎯 Why It's Interesting for AI Security Researchers

This research is crucial for AI security researchers as it systematically evaluates existing defenses against prompt injection attacks, providing a clearer picture of their effectiveness in dynamic, real-world situations. The development of AgentDyn as a benchmark offers a foundation for future research to build upon, driving innovation in AI security measures and prompting a re-assessment of current methodologies in combating prompt injection threats.

📚 Read the Full Paper