ICML 2026 Paper Argues LLMs Cannot Reliably Distinguish User Commands From Their Own Thoughts, Making Prompt Injection Fundamentally Unsolvable
Researchers presented findings at ICML 2026 showing that large language models identify the source of their instructions by writing style, not by the XML tags that separate user input from system prompts and tool outputs. Swapping tags made almost no difference to model behavior. The finding, reported by MIT Technology Review, suggests prompt injection is an architectural constraint rather than a training problem, with direct consequences for how autonomous agents must be deployed in production: runtime authorization and least-privilege access become mandatory infrastructure, not optional hardening.