← Back to Library

Bypassing Prompt Guards in Production with Controlled-Release Prompting

Authors: Jaiden Fairoze, Sanjam Garg, Keewoo Lee, Mingyuan Wang

Published: 2025-10-02

arXiv ID: 2510.01529v1

Added to Library: 2025-10-03 04:01 UTC

Red Teaming

📄 Abstract

As large language models (LLMs) advance, ensuring AI safety and alignment is paramount. One popular approach is prompt guards, lightweight mechanisms designed to filter malicious queries while being easy to implement and update. In this work, we introduce a new attack that circumvents such prompt guards, highlighting their limitations. Our method consistently jailbreaks production models while maintaining response quality, even under the highly protected chat interfaces of Google Gemini (2.5 Flash/Pro), DeepSeek Chat (DeepThink), Grok (3), and Mistral Le Chat (Magistral). The attack exploits a resource asymmetry between the prompt guard and the main LLM, encoding a jailbreak prompt that lightweight guards cannot decode but the main model can. This reveals an attack surface inherent to lightweight prompt guards in modern LLM architectures and underscores the need to shift defenses from blocking malicious inputs to preventing malicious outputs. We additionally identify other critical alignment issues, such as copyrighted data extraction, training data extraction, and malicious response leakage during thinking.

🔍 Key Points

  • Introduction of controlled-release prompting as a new attack strategy that circumvents lightweight prompt guards in production LLMs, like Google Gemini and DeepSeek Chat.
  • Demonstration of resource asymmetry exploitation, allowing malicious prompts to be encoded in a way that guard models cannot decode but the main LLM can.
  • Validation experiments confirm that the proposed attacks maintain high performance across multiple leading LLM platforms, demonstrating effectiveness where simpler attacks fail.
  • Identification of other significant alignment issues such as unintended leakage of copyrighted and training data through reasoning processes.
  • Recommendations for shifting focus from input filtering to output control in AI safety measures, highlighting the inadequacies of current prompt guard implementations.

💡 Why This Paper Matters

This paper presents a critical analysis of current AI safety mechanisms, introducing a novel method that reveals significant vulnerabilities in widely-used LLM deployments. By demonstrating the effectiveness of controlled-release prompting against prominent guard models, it underscores the urgent need to reconsider and improve the frameworks for securing AI from misuse, particularly as LLMs gain wider application.

🎯 Why It's Interesting for AI Security Researchers

This paper is highly relevant to AI security researchers as it not only identifies specific weaknesses in existing prompt guard systems but also proposes a new attack vector, thus expanding the knowledge base surrounding LLM vulnerabilities. Furthermore, it advocates for a strategic shift in defensive paradigms towards improved output monitoring, prompting researchers to explore innovative solutions in AI alignment and safety frameworks.

📚 Read the Full Paper

📄 View on arXiv 📎 Download PDF