← Back to Library

Extended to Reality: Prompt Injection in 3D Environments

Authors: Zhuoheng Li, Ying Chen

Published: 2026-02-06

arXiv ID: 2602.07104v1

Added to Library: 2026-02-10 03:02 UTC

Red Teaming

📄 Abstract

Multimodal large language models (MLLMs) have advanced the capabilities to interpret and act on visual input in 3D environments, empowering diverse applications such as robotics and situated conversational agents. When MLLMs reason over camera-captured views of the physical world, a new attack surface emerges: an attacker can place text-bearing physical objects in the environment to override MLLMs' intended task. While prior work has studied prompt injection in the text domain and through digitally edited 2D images, it remains unclear how these attacks function in 3D physical environments. To bridge the gap, we introduce PI3D, a prompt injection attack against MLLMs in 3D environments, realized through text-bearing physical object placement rather than digital image edits. We formulate and solve the problem of identifying an effective 3D object pose (position and orientation) with injected text, where the attacker's goal is to induce the MLLM to perform the injected task while ensuring that the object placement remains physically plausible. Experiments demonstrate that PI3D is an effective attack against multiple MLLMs under diverse camera trajectories. We further evaluate existing defenses and show that they are insufficient to defend against PI3D.

🔍 Key Points

  • Introduction of PI3D: The paper presents PI3D, a novel prompt injection attack methodology specifically designed for multimodal large language models (MLLMs) in 3D environments, utilizing text-bearing physical objects instead of digital images or text alone.
  • Experience-Guided Planning: The authors developed an experience-guided planning approach that effectively reduces the computational costs associated with evaluating candidate placements for the prompt injection attack, leveraging past experiences to enhance efficiency.
  • Experimental Validation: PI3D demonstrates significant effectiveness in manipulating MLLMs across both virtual and real-world environments, with high attack success rates (ASR) and adequate physical plausibility under varied conditions and camera trajectories.
  • Inadequacies of Existing Defenses: The paper evaluates existing defense mechanisms against prompt injection attacks, revealing that they are largely ineffective against PI3D, thus highlighting the need for improved security measures in MLLMs.
  • Practical Applications and Threats: With the growing integration of MLLMs into robotics and interactive agents, the findings underscore serious security implications, as attackers can leverage physical object placements to alter model behavior unexpectedly.

💡 Why This Paper Matters

This paper is crucial as it exposes a previously underexplored security vulnerability in MLLMs operating in 3D environments, particularly as these models become increasingly prevalent in practical applications like autonomous vehicles and interactive AI systems. By revealing how adversaries can manipulate model outputs through seemingly harmless physical objects, it calls for heightened awareness and development of robust defenses against such novel threats.

🎯 Why It's Interesting for AI Security Researchers

The research is of significant interest to AI security researchers because it expands the understanding of vulnerability in multimodal AI systems, particularly within real-world contexts. The introduction of physical-world attacks raises questions about the robustness of AI perception and decision-making processes, prompting the exploration of new defensive strategies and frameworks to secure MLLMs against similar adversarial attacks.

📚 Read the Full Paper

📄 View on arXiv 📎 Download PDF