Automating Deception: Scalable Multi-Turn LLM Jailbreaks

📄 Abstract

Multi-turn conversational attacks, which leverage psychological principles like Foot-in-the-Door (FITD), where a small initial request paves the way for a more significant one, to bypass safety alignments, pose a persistent threat to Large Language Models (LLMs). Progress in defending against these attacks is hindered by a reliance on manual, hard-to-scale dataset creation. This paper introduces a novel, automated pipeline for generating large-scale, psychologically-grounded multi-turn jailbreak datasets. We systematically operationalize FITD techniques into reproducible templates, creating a benchmark of 1,500 scenarios across illegal activities and offensive content. We evaluate seven models from three major LLM families under both multi-turn (with history) and single-turn (without history) conditions. Our results reveal stark differences in contextual robustness: models in the GPT family demonstrate a significant vulnerability to conversational history, with Attack Success Rates (ASR) increasing by as much as 32 percentage points. In contrast, Google's Gemini 2.5 Flash exhibits exceptional resilience, proving nearly immune to these attacks, while Anthropic's Claude 3 Haiku shows strong but imperfect resistance. These findings highlight a critical divergence in how current safety architectures handle conversational context and underscore the need for defenses that can resist narrative-based manipulation.

🔍 Key Points

Introduction of a novel automated pipeline for generating psychologically-grounded multi-turn jailbreak datasets which produce 1,500 scenarios based on the Foot-in-the-Door principle.
Evaluation of seven models from three major LLM families under multi-turn and single-turn conditions revealing significant differences in contextual robustness, with GPT models showing higher vulnerabilities compared to Gemini 2.5 Flash and Claude 3 Haiku.
Establishment of a benchmark to measure Attack Success Rates (ASR) showing up to a 32% increase in vulnerability for GPT models when conversational history is included, suggesting the importance of context in model safety.
Detailed discussion of mitigation strategies including architectural changes, adversarial training, and detection mechanisms to bolster the robustness of LLMs against multi-turn conversational attacks.
Methodological validation of dataset generation with 98% agreement with human assessments, underpinning the reliability of the automated testing framework.

💡 Why This Paper Matters

This paper advances the understanding of vulnerabilities inherent in Large Language Models (LLMs) when subjected to multi-turn conversational attacks, which exploit psychological principles to bypass model safety mechanisms. By creating an automated pipeline for generating attacks and evaluating multiple LLMs, the authors not only highlight significant differences in model robustness but also propose practical solutions to enhance security. The findings emphasize the necessity of context-aware defenses in AI systems, which is fundamental for developing safer AI applications in sensitive environments.

🎯 Why It's Interesting for AI Security Researchers

AI security researchers would find this paper particularly relevant because it addresses a critical and emergent threat landscape in AI: the exploitation of conversational context to bypass safety measures. The automated generation of adversarial prompts based on psychological techniques provides an innovative method for evaluating model vulnerabilities at scale, which is essential for understanding and mitigating risks in AI deployments. Furthermore, the proposed defense mechanisms could inform future designs of safer LLM architectures, making the findings significant for researchers aiming to enhance the security and reliability of AI systems.

Automating Deception: Scalable Multi-Turn LLM Jailbreaks

📄 Abstract

🔍 Key Points

💡 Why This Paper Matters

🎯 Why It's Interesting for AI Security Researchers

📚 Read the Full Paper