← Back to Library

A Comprehensive Evaluation of Multilingual Chain-of-Thought Reasoning: Performance, Consistency, and Faithfulness Across Languages

Authors: Raoyuan Zhao, Yihong Liu, Hinrich Schütze, Michael A. Hedderich

Published: 2025-10-10

arXiv ID: 2510.09555v1

Added to Library: 2025-11-14 23:12 UTC

📄 Abstract

Large reasoning models (LRMs) increasingly rely on step-by-step Chain-of-Thought (CoT) reasoning to improve task performance, particularly in high-resource languages such as English. While recent work has examined final-answer accuracy in multilingual settings, the thinking traces themselves, i.e., the intermediate steps that lead to the final answer, remain underexplored. In this paper, we present the first comprehensive study of multilingual CoT reasoning, evaluating three key dimensions: performance, consistency, and faithfulness. We begin by measuring language compliance, answer accuracy, and answer consistency when LRMs are explicitly instructed or prompt-hacked to think in a target language, revealing strong language preferences and divergent performance across languages. Next, we assess crosslingual consistency of thinking traces by interchanging them between languages. We find that the quality and effectiveness of thinking traces vary substantially depending on the prompt language. Finally, we adapt perturbation-based techniques -- i.e., truncation and error injection -- to probe the faithfulness of thinking traces across languages, showing that models rely on traces to varying degrees. We release our code and data to support future research.

🔍 Key Points

  • The paper introduces the concept of Deep Research (DR) agents that leverage LLMs to perform complex research tasks, revealing significant vulnerabilities when such agents respond to harmful queries.
  • It outlines two novel jailbreak strategies—Plan Injection and Intent Hijack—that exploit the planning and research capabilities of DR agents, demonstrating their risks in generating harmful content.
  • Extensive experiments highlight that DR agents can circumvent traditional alignment mechanisms by producing coherent and dangerous reports that standalone LLMs would reject.
  • The proposed DeepREJECT evaluation metric is introduced, which assesses whether the generated content aligns with harmful intents and the quality of knowledge provided, outperforming previous benchmarks.
  • The findings raise critical questions about the safety measures in deploying LLMs in sensitive domains, especially in contexts like biosecurity.

💡 Why This Paper Matters

This paper is crucial as it identifies the elevated risks associated with Deep Research agents powered by Large Language Models, emphasizing the urgent need for refined safety analyses and robust alignment strategies. The methodologies proposed offer significant insights into the potential for misuse in high-stakes domains, calling for an overhaul in how AI systems are designed to ensure safety in practical applications.

🎯 Why It's Interesting for AI Security Researchers

The paper will intrigue AI security researchers as it exposes the critical vulnerabilities in existing alignment frameworks when applied to advanced AI systems like DR agents. It provides novel attack methodologies that can inform the development of more robust safety protocols and prompts further investigation into the potential misuse of AI technologies in sensitive and high-risk environments.

📚 Read the Full Paper

📄 View on arXiv 📎 Download PDF