← Back to Library

Defending Against Indirect Prompt Injection Attacks With Spotlighting

Authors: Keegan Hines, Gary Lopez, Matthew Hall, Federico Zarfati, Yonatan Zunger, Emre Kiciman

Published: 2024-03-20

arXiv ID: 2403.14720v1

Added to Library: 2025-11-11 14:12 UTC

Red Teaming

📄 Abstract

Large Language Models (LLMs), while powerful, are built and trained to process a single text input. In common applications, multiple inputs can be processed by concatenating them together into a single stream of text. However, the LLM is unable to distinguish which sections of prompt belong to various input sources. Indirect prompt injection attacks take advantage of this vulnerability by embedding adversarial instructions into untrusted data being processed alongside user commands. Often, the LLM will mistake the adversarial instructions as user commands to be followed, creating a security vulnerability in the larger system. We introduce spotlighting, a family of prompt engineering techniques that can be used to improve LLMs' ability to distinguish among multiple sources of input. The key insight is to utilize transformations of an input to provide a reliable and continuous signal of its provenance. We evaluate spotlighting as a defense against indirect prompt injection attacks, and find that it is a robust defense that has minimal detrimental impact to underlying NLP tasks. Using GPT-family models, we find that spotlighting reduces the attack success rate from greater than {50}\% to below {2}\% in our experiments with minimal impact on task efficacy.

🔍 Key Points

  • Introduction of spotlighting, a family of techniques for enhancing the capability of Large Language Models (LLMs) to distinguish between trusted and untrusted input sources.
  • Demonstration of three prompt engineering techniques: delimiting, datamarking, and encoding, each aimed at reducing the risk of indirect prompt injection attacks (XPIA).
  • Evaluation of the effectiveness of spotlighting techniques, achieving a reduction in attack success rate from over 50% to below 2% across various tasks without detrimental effects on overall NLP tasks.
  • Experimental methodology that includes the creation of a synthetic dataset for measuring attack success rate (ASR), providing a clear framework for evaluating prompt injection defenses.
  • Discussion of the implications of prompt injection attacks and spotlighting as a structural solution, drawing analogies to telecommunications signaling.

💡 Why This Paper Matters

This paper presents a significant contribution to the field of AI security by addressing a critical vulnerability in large language models, namely indirect prompt injection attacks. By introducing novel prompt engineering techniques, the authors provide an effective framework to enhance the security and robustness of LLMs in practical applications, thereby promoting greater trust in AI systems.

🎯 Why It's Interesting for AI Security Researchers

AI security researchers will find this paper particularly relevant as it delves into a frequently overlooked aspect of language model safety—prompt injection attacks. The proposed spotlighting techniques not only mitigate risks associated with XPIA but also lay the groundwork for future research into safeguarding LLMs in increasingly complex and integrated AI systems. The insights and methods provided can potentially influence the design of safer language models and inform best practices for secure AI deployment.

📚 Read the Full Paper