← Back to Library

Evaluating the Instruction-Following Robustness of Large Language Models to Prompt Injection

Authors: Zekun Li, Baolin Peng, Pengcheng He, Xifeng Yan

Published: 2023-08-17

arXiv ID: 2308.10819v3

Added to Library: 2025-11-11 14:08 UTC

Red Teaming

📄 Abstract

Large Language Models (LLMs) have demonstrated exceptional proficiency in instruction-following, becoming increasingly crucial across various applications. However, this capability brings with it the risk of prompt injection attacks, where attackers inject instructions into LLMs' input to elicit undesirable actions or content. Understanding the robustness of LLMs against such attacks is vital for their safe implementation. In this work, we establish a benchmark to evaluate the robustness of instruction-following LLMs against prompt injection attacks. Our objective is to determine the extent to which LLMs can be influenced by injected instructions and their ability to differentiate between these injected and original target instructions. Through extensive experiments with leading instruction-following LLMs, we uncover significant vulnerabilities in their robustness to such attacks. Our results indicate that some models are overly tuned to follow any embedded instructions in the prompt, overly focusing on the latter parts of the prompt without fully grasping the entire context. By contrast, models with a better grasp of the context and instruction-following capabilities will potentially be more susceptible to compromise by injected instructions. This underscores the need to shift the focus from merely enhancing LLMs' instruction-following capabilities to improving their overall comprehension of prompts and discernment of instructions that are appropriate to follow. We hope our in-depth analysis offers insights into the underlying causes of these vulnerabilities, aiding in the development of future solutions. Code and data are available at https://github.com/Leezekun/instruction-following-robustness-eval

🔍 Key Points

  • Introduction of a benchmark for evaluating the instruction-following robustness of Large Language Models (LLMs) against prompt injection attacks.
  • Extensive experiments reveal significant vulnerabilities in LLMs, particularly their tendency to follow latter parts of prompts without adequately considering full context.
  • Identification of a discrepancy between a model's instruction-following capabilities and its robustness, suggesting that larger models do not necessarily exhibit better security against prompt injections.
  • Analysis of different factors influencing robustness, such as injection types and positions, showcasing how contextual understanding affects model performance in adversarial scenarios.
  • Call for increased focus on enhancing models' comprehension abilities rather than solely improving instruction-following capabilities.

💡 Why This Paper Matters

This paper significantly contributes to the understanding of vulnerabilities in LLMs by systematically evaluating their behavior against prompt injection attacks, offering a clear framework for future research. The findings highlight critical weaknesses that could impact the deployment of LLMs in real-world applications, making this work essential for ensuring the safety and integrity of AI systems that rely on such models for instruction-following tasks.

🎯 Why It's Interesting for AI Security Researchers

AI security researchers will find this paper valuable as it addresses a prevalent yet underexplored threat to LLMs known as prompt injection attacks. The established benchmark and the thorough evaluation of various models’ robustness against these attacks provide clear insights into existing vulnerabilities and impacts on user instructions. This knowledge is crucial for developing more secure LLM applications and for establishing effective defenses against potential exploitation.

📚 Read the Full Paper

📄 View on arXiv