← Back to Library

StruQ: Defending Against Prompt Injection with Structured Queries

Authors: Sizhe Chen, Julien Piet, Chawin Sitawarin, David Wagner

Published: 2024-02-09

arXiv ID: 2402.06363v2

Added to Library: 2025-11-11 14:34 UTC

📄 Abstract

Recent advances in Large Language Models (LLMs) enable exciting LLM-integrated applications, which perform text-based tasks by utilizing their advanced language understanding capabilities. However, as LLMs have improved, so have the attacks against them. Prompt injection attacks are an important threat: they trick the model into deviating from the original application's instructions and instead follow user directives. These attacks rely on the LLM's ability to follow instructions and inability to separate prompts and user data. We introduce structured queries, a general approach to tackle this problem. Structured queries separate prompts and data into two channels. We implement a system that supports structured queries. This system is made of (1) a secure front-end that formats a prompt and user data into a special format, and (2) a specially trained LLM that can produce high-quality outputs from these inputs. The LLM is trained using a novel fine-tuning strategy: we convert a base (non-instruction-tuned) LLM to a structured instruction-tuned model that will only follow instructions in the prompt portion of a query. To do so, we augment standard instruction tuning datasets with examples that also include instructions in the data portion of the query, and fine-tune the model to ignore these. Our system significantly improves resistance to prompt injection attacks, with little or no impact on utility. Our code is released at https://github.com/Sizhe-Chen/StruQ.

🔍 Key Points

  • Introduction of Prompt Trigger Attacks (PTA) as a unified category encompassing prompt injection, backdoor attacks, and adversarial attacks, highlighting their intrinsic relationships.
  • Development of UniGuardian, a novel and efficient training-free defense mechanism that simultaneously detects multiple attack types during inference.
  • Implementation of a Single-Forward Strategy that accelerates attack detection by integrating detection with text generation in a single forward pass of the model.
  • Extensive experimental validation demonstrating UniGuardian's superior performance in accurately identifying malicious prompts across various attack types compared to existing methods.
  • Release of implementation code, promoting accessibility and further research in prompt attack detection and defense mechanisms.

💡 Why This Paper Matters

This paper is critical in advancing the security of large language models by proposing a comprehensive framework, UniGuardian, that effectively addresses various prompt manipulation attacks. Such a unified approach is essential as the landscape of AI continues to evolve, making language models increasingly susceptible to different forms of adversarial attacks.

🎯 Why It's Interesting for AI Security Researchers

The findings and methodologies presented in this paper are highly significant for AI security researchers engaged in understanding and mitigating risks associated with large language models. As the adoption of these models grows, so does the necessity for robust defenses against manipulation techniques that can compromise their reliability and safety. The identification and implementation of a unified defense mechanism against diverse attack types will be of keen interest to researchers focused on enhancing the resilience of AI systems.

📚 Read the Full Paper

📄 View on arXiv