← Back to Library

Prompt-based Adaptation in Large-scale Vision Models: A Survey

Authors: Xi Xiao, Yunbei Zhang, Lin Zhao, Yiyang Liu, Xiaoying Liao, Zheda Mai, Xingjian Li, Xiao Wang, Hao Xu, Jihun Hamm, Xue Lin, Min Xu, Qifan Wang, Tianyang Wang, Cheng Han

Published: 2025-10-15

arXiv ID: 2510.13219v1

Added to Library: 2025-11-14 23:11 UTC

📄 Abstract

In computer vision, Visual Prompting (VP) and Visual Prompt Tuning (VPT) have recently emerged as lightweight and effective alternatives to full fine-tuning for adapting large-scale vision models within the ``pretrain-then-finetune'' paradigm. However, despite rapid progress, their conceptual boundaries remain blurred, as VP and VPT are frequently used interchangeably in current research, reflecting a lack of systematic distinction between these techniques and their respective applications. In this survey, we revisit the designs of VP and VPT from first principles, and conceptualize them within a unified framework termed Prompt-based Adaptation (PA). We provide a taxonomy that categorizes existing methods into learnable, generative, and non-learnable prompts, and further organizes them by injection granularity -- pixel-level and token-level. Beyond the core methodologies, we examine PA's integrations across diverse domains, including medical imaging, 3D point clouds, and vision-language tasks, as well as its role in test-time adaptation and trustworthy AI. We also summarize current benchmarks and identify key challenges and future directions. To the best of our knowledge, we are the first comprehensive survey dedicated to PA's methodologies and applications in light of their distinct characteristics. Our survey aims to provide a clear roadmap for researchers and practitioners in all area to understand and explore the evolving landscape of PA-related research.

🔍 Key Points

  • First systematic analysis of poisoning risks in LLM-based prompt optimization, highlighting vulnerabilities in feedback manipulation over query manipulation.
  • Identification of the fake-reward attack that significantly raises attack success rates by providing misleading feedback without access to the reward model.
  • Development of a lightweight defense mechanism (highlighting) that effectively mitigates the impact of fake-reward attacks while maintaining system utility.
  • Empirical evidence showing that prompt optimization metrics critically influence the susceptibility of systems to adversarial exploitation, emphasizing the need for careful metric selection.
  • Proposed an actionable framework for securing feedback channels in LLM-based optimizers, marking prompt optimization pipelines as a new attack surface in AI safety.

💡 Why This Paper Matters

This paper is crucial for advancing the understanding of security vulnerabilities inherent in LLM-based optimization processes. By presenting a novel class of feedback manipulation attacks and highlighting the importance of robust defense mechanisms, the authors contribute significantly to the field of AI safety. Their findings advocate for re-evaluating existing practices in prompt optimization pipelines, which are increasingly used in real-world applications.

🎯 Why It's Interesting for AI Security Researchers

AI security researchers will find this paper particularly relevant as it addresses a critical gap in the literature concerning the vulnerabilities specific to LLM-based optimization methods. The exploration of feedback manipulation attacks, the introduction of the fake-reward attack, and the proposed defenses align with ongoing concerns about the security of machine learning systems. As these models become more embedded in sensitive applications, understanding their threats and implementing effective safeguards is paramount.

📚 Read the Full Paper

📄 View on arXiv 📎 Download PDF