We report that ChatGPT 4 and 4o are susceptible to a prompt injection attack that allows an attacker to exfiltrate users' personal data. It is applicable without the use of any 3rd party tools and all users are currently affected. This vulnerability is exacerbated by the recent introduction of ChatGPT's memory feature, which allows an attacker to command ChatGPT to monitor the user for the desired personal data.
This paper provides critical insights into the vulnerabilities of widely used language models like ChatGPT. The detailed exploration of prompt injection attacks and their potential to exfiltrate personal information underlines significant risks associated with AI systems that handle sensitive user data. The findings are highly relevant to developers and policymakers, advocating for enhanced security measures and user education to prevent data breaches.
AI security researchers would find this paper invaluable as it addresses the emerging threats posed by language models, particularly those related to user data protection and privacy. The novel methods outlined for exploiting prompt injection and the implications of the memory feature in AI systems can lead to a deeper understanding of security vulnerabilities and inform the development of more robust defensive frameworks against such attacks.