In the context of AI security, prompt injection has emerged as a significant threat. Large language models (LLMs) struggle to differentiate between legitimate user instructions and malicious commands embedded in emails, source code, and other third-party content. This vulnerability allows attackers to easily inject harmful commands that the LLMs may execute without question.
Currently, AI developers focus on creating protective measures to limit potential damage rather than addressing the fundamental issue of distinguishing between trusted and untrusted sources. Most prompt injection attacks have been categorized as 'push' attacks, where individual targets receive malicious instructions through specific communications, such as emails or calendar invites. This method restricts the scale of attacks, preventing widespread exploitation across the internet.