← Back to Library

Embedding-based classifiers can detect prompt injection attacks

Authors: Md. Ahsan Ayub, Subhabrata Majumdar

Published: 2024-10-29

arXiv ID: 2410.22284v1

Added to Library: 2025-11-11 14:10 UTC

Red Teaming

📄 Abstract

Large Language Models (LLMs) are seeing significant adoption in every type of organization due to their exceptional generative capabilities. However, LLMs are found to be vulnerable to various adversarial attacks, particularly prompt injection attacks, which trick them into producing harmful or inappropriate content. Adversaries execute such attacks by crafting malicious prompts to deceive the LLMs. In this paper, we propose a novel approach based on embedding-based Machine Learning (ML) classifiers to protect LLM-based applications against this severe threat. We leverage three commonly used embedding models to generate embeddings of malicious and benign prompts and utilize ML classifiers to predict whether an input prompt is malicious. Out of several traditional ML methods, we achieve the best performance with classifiers built using Random Forest and XGBoost. Our classifiers outperform state-of-the-art prompt injection classifiers available in open-source implementations, which use encoder-only neural networks.

🔍 Key Points

  • Proposes embedding-based classifiers for detecting prompt injection attacks, showcasing an innovative approach within AI security.
  • Curates a large dataset of 467,057 prompts, analyzed through multiple embedding models to understand distinguishing features of benign vs. malicious prompts.
  • Demonstrates that Random Forest and XGBoost classifiers achieve superior performance compared to existing deep learning methods, pushing the boundaries of traditional machine learning in this domain.
  • Uses dimensionality reduction techniques (PCA, t-SNE, UMAP) for visual analysis of embedding spaces, providing insights into the separability of malicious vs. benign prompts.
  • Achieves the highest F1 and precision scores among tested models, indicating a promising balance between sensitivity and specificity in detecting adversarial prompts.

💡 Why This Paper Matters

This paper presents a groundbreaking approach to enhancing the security of large language models against prompt injection attacks, which have become a significant concern in AI applications. Its contributions not only advance the understanding of malicious prompt behavior through empirical analysis but also offer effective machine learning strategies that outperform existing methodologies, making it a notable reference for future research and practical implementations in AI safety.

🎯 Why It's Interesting for AI Security Researchers

This paper is of paramount interest to AI security researchers as it addresses a pressing vulnerability of LLMs—prompt injection attacks. By utilizing embedding-based classifiers, it demonstrates a novel technique that varies significantly from traditional deep learning approaches, thereby inviting further investigation and development in adversarial defenses. Its empirical findings and curated dataset could serve as valuable resources for enhancing the robustness of AI systems against similar threats.

📚 Read the Full Paper

📄 View on arXiv