ICML 2026 Paper Argues LLMs Cannot Reliably Distinguish User Commands From Their Own Thoughts, Making Prompt Injection Fundamentally Unsolvable

A team of researchers presented a paper at ICML 2026 arguing that large language models cannot reliably tell the difference between instructions from a user, instructions from their own designers, and text injected from external tools. The implication: prompt injection may be a permanent feature of LLM architecture, not a bug that better training will fix.

“There’s a real probability that this is going to be a problem that’s fundamentally unsolvable,” Charles Ye, an independent researcher and coauthor of the paper, told MIT Technology Review.

The timing matters. This week alone, Noma Security disclosed a CVSS 10.0 vulnerability in Ruflo’s MCP bridge, Delinea and Neo launched runtime authorization platforms for AI agents, and Google Cloud expanded its Gemini Enterprise Agent Platform with governance controls. The ICML paper provides the theoretical foundation for why all of that infrastructure work is necessary.

Tags Are Decorative, Not Functional

LLMs use XML-style role tags to separate different types of input. User messages sit between <user> tags, system prompts between <system> tags, model reasoning between <think> tags, and external tool outputs between <tool> tags. Model makers train LLMs to respect these boundaries, treating system instructions as authoritative and user inputs as potentially adversarial.

The ICML researchers found that the tags are effectively decorative. In experiments that probed what was happening inside several models, they discovered that LLMs identify the role of a text chunk by its writing style and word choice, not by the tags surrounding it, according to MIT Technology Review.

Swapping <think> tags for <user> tags made almost no difference to how the model interpreted the text. If the content read like chain-of-thought reasoning, the model treated it as its own internal reasoning regardless of what tags wrapped it.

Chain-of-Thought Forgery

The researchers developed an attack they call “chain-of-thought forgery.” By writing prompts in the style of a model’s internal scratch pad, they could trick the LLM into treating adversarial instructions as its own reasoning.

The technique won OpenAI’s red-teaming hackathon in August 2025, and OpenAI’s own automated red-teaming system (GPT-Red) independently discovered a nearly identical attack around the same time, according to MIT Technology Review.

The attacks work across model families. The ICML paper describes results against several OpenAI models, but coauthors Cui and Ye said they have since seen similar results with models from Anthropic, Alibaba, and DeepSeek, according to MIT Technology Review.

“It’s like watching The Simpsons and they have Bart writing ‘I will not say something inappropriate to my teacher’ a hundred times,” said Jasmine Cui, independent researcher and coauthor. “And he still does things that are pretty crass anyway.”

Why Training Cannot Fully Fix It

The paper’s central argument, as reported by MIT Technology Review, is that role confusion is inherent to how LLMs process text. An LLM sees “one big sheet of tokens,” in Cui’s words. Humans can distinguish their own thoughts from someone else’s speech because of physical feedback (hearing their own voice, feeling their mouth move). LLMs have no equivalent mechanism.

Red-teaming amounts to compiling a list of attacks and training models to resist them. But no list is exhaustive, and every new attack style requires new training data. Florian Tramèr, a computer scientist at ETH Zürich who works on LLM security, acknowledged to MIT Technology Review that leading models are “much harder to prompt-inject now” thanks to combined defenses, but added: “It’s not clear this will be sufficient for highly sensitive cases.”

Cui noted she was hired by top labs including Anthropic as a red-teamer. In one case, she made an LLM provide restricted information by telling it to pretend to be drunk. In another, she persuaded Claude to provide weapon instructions by telling it the military was already using it for war, then having it search the web to confirm. “When people are surprised, they become a little more neuroplastic,” she told MIT Technology Review.

The Infrastructure Consequence

If prompt injection cannot be eliminated at the model layer, every agent deployment needs external controls that assume the model will eventually be compromised.

That is the premise behind this week’s wave of agent security product launches. Delinea and Neo’s runtime authorization platforms govern what agents can do after they’ve been authenticated, applying least-privilege policies at the action level. Lasso’s AI Security Posture Management platform continuously maps agent ecosystems and detects runtime behavior drift. Google Cloud’s expanded Gemini Enterprise Agent Platform includes a governance gateway and native agent identity management.

“Organizations shouldn’t trust LLMs, and they should expect that anything done by agents could be unsafe,” Ye told MIT Technology Review. “That’s not a great solution, but it just might be what we have to do.”

The ICML paper reframes the agent security problem. The question for teams deploying autonomous agents is not whether their model can resist prompt injection. The question is what happens when it fails.