← Back to Library

Credential Leakage in LLM Agent Skills: A Large-Scale Empirical Study

Authors: Zhihao Chen, Ying Zhang, Yi Liu, Gelei Deng, Yuekang Li, Yanjun Zhang, Jianting Ning, Leo Yu Zhang, Lei Ma, Zhiqiang Li

Published: 2026-04-03

arXiv ID: 2604.03070v1

Added to Library: 2026-04-06 02:04 UTC

Red Teaming

📄 Abstract

Third-party skills extend LLM agents with powerful capabilities but often handle sensitive credentials in privileged environments, making leakage risks poorly understood. We present the first large-scale empirical study of this problem, analyzing 17,022 skills (sampled from 170,226 on SkillsMP) using static analysis, sandbox testing, and manual inspection. We identify 520 vulnerable skills with 1,708 issues and derive a taxonomy of 10 leakage patterns (4 accidental and 6 adversarial). We find that (1) leakage is fundamentally cross-modal: 76.3% require joint analysis of code and natural language, while 3.1% arise purely from prompt injection; (2) debug logging is the primary vector, with print and console.log causing 73.5% of leaks due to stdout exposure to LLMs; and (3) leaked credentials are both exploitable (89.6% without privileges) and persistent, as forks retain secrets even after upstream fixes. After disclosure, all malicious skills were removed and 91.6% of hardcoded credentials were fixed. We release our dataset, taxonomy, and detection pipeline to support future research.

🔍 Key Points

  • First large-scale empirical study of credential leakage in LLM agent skills, analyzing 17,022 skills and identifying 520 vulnerable skills with 1,708 issues.
  • Development of a comprehensive taxonomy of 10 credential leakage patterns which include 4 from developer negligence and 6 from deliberate adversarial construction.
  • Discovery that 76.3% of credential leaks require cross-modal analysis involving both natural language (NL) and programming language (PL), indicating a unique vulnerability landscape in agent skills.
  • Debug logging was recognized as the primary exposure vector, responsible for 73.5% of the identified vulnerability issues due to the integration of stdout into LLM context.
  • Report of immediate actionable credential leaks; with 89.6% of affected skills exploitable without elevated privileges and persistence of leaks through forks, showing limitations of remediation efforts.

💡 Why This Paper Matters

This paper is crucial as it highlights the significant and often unnoticed risk of credential leakage in LLM agent skills, providing actionable insights into vulnerability patterns and exposure mechanisms. The findings not only elevate awareness about the security posture required for developing such systems but also provide foundational knowledge for ongoing research in this rapidly evolving domain.

🎯 Why It's Interesting for AI Security Researchers

The paper is of great interest to AI security researchers as it addresses novel attack vectors and vulnerabilities that arise from the intersection of natural and programming languages in LLM environments. Understanding these risks is essential for developing robust security strategies and improving the overall safety of AI systems that leverage third-party skills.

📚 Read the Full Paper

📄 View on arXiv 📎 Download PDF