← Back to Library

Bypassing Prompt Guards in Production with Controlled-Release Prompting

Authors: Jaiden Fairoze, Sanjam Garg, Keewoo Lee, Mingyuan Wang

Published: 2025-10-02

arXiv ID: 2510.01529v2

Added to Library: 2025-10-08 01:01 UTC

Red Teaming

📄 Abstract

As large language models (LLMs) advance, ensuring AI safety and alignment is paramount. One popular approach is prompt guards, lightweight mechanisms designed to filter malicious queries while being easy to implement and update. In this work, we introduce a new attack that circumvents such prompt guards, highlighting their limitations. Our method consistently jailbreaks production models while maintaining response quality, even under the highly protected chat interfaces of Google Gemini (2.5 Flash/Pro), DeepSeek Chat (DeepThink), Grok (3), and Mistral Le Chat (Magistral). The attack exploits a resource asymmetry between the prompt guard and the main LLM, encoding a jailbreak prompt that lightweight guards cannot decode but the main model can. This reveals an attack surface inherent to lightweight prompt guards in modern LLM architectures and underscores the need to shift defenses from blocking malicious inputs to preventing malicious outputs. We additionally identify other critical alignment issues, such as copyrighted data extraction, training data extraction, and malicious response leakage during thinking.

🔍 Key Points

  • Introduction of a novel jailbreak attack method targeting prompt guards in LLMs, highlighting their limitations.
  • Controlled-release prompting technique exploits resource asymmetry between LLMs and lightweight guard models, allowing malicious prompts to bypass input filters while maintaining response quality.
  • Empirical evaluations demonstrate the effectiveness of timed-release and spaced-release attacks across multiple LLM platforms such as Google Gemini and DeepSeek Chat.
  • Identifies critical alignment issues related to copyrighted data extraction and malicious response leakage during reasoning processes.
  • Calls for a paradigm shift in LLM security from filtering malicious inputs to preventing harmful outputs.

💡 Why This Paper Matters

This paper is significant as it exposes vulnerabilities in current AI safety mechanisms, particularly those relying on lightweight input filters. The introduction of the controlled-release prompting method provides a robust attack framework that demonstrates how existing defenses may be inadequate, emphasizing the need for more comprehensive output-based alignment strategies in LLM deployment and development.

🎯 Why It's Interesting for AI Security Researchers

AI security researchers will find this paper relevant due to its exploration of security flaws in widely used LLM architectures. The novel techniques presented can inform future defenses and provoke discussions on improving alignment methods, with practical implications for the design and deployment of safer AI systems.

📚 Read the Full Paper

📄 View on arXiv 📎 Download PDF