Finagling a Fundamental Flaw
The deepest misconception surrounding LLM’s and AI is that machines can think. They don’t. Instead they aggregate, which is an entirely different process. Thought is dynamic. It constantly shifts and adapts around new information, perspectives and/or experiences. Machines, as built from processors and running on electricity, simply cannot do that. It’s literally not in their wiring. Thought is an organic activity. The best a machine can do is mimic it. But mimicry falls short of the real thing because it’s missing some of the key ingredients that make it possible: interpretation and judgment.
Attempts to compensate for this lack led to the earliest models falling into roles to designate responses, separating requests and prompts into definable objectives like <user>, <system>, <assistant>, <tool> and <think>. OpenAI’s first breakdown of ChatGPT’s goals back in December of 2021 states that it should be possible to create a general-purpose, text-based assistant that is aligned with human values, meaning that it is helpful, honest, and harmless. And with that in mind, the LLM was written to have a kind of extended autocomplete process between itself and a user, based upon known, quantifiable input. And that’s fine for what is in essence an application program that sorts usable information from raw data. Find this value. Organize this in ascending or descending order. Detect this set of variables.
If that was all we ever asked of LLM’s, I would have no problem with them. But it isn’t.
Humans have a long history with using technology for purposes other than what it was intended for. Look at any scientific advancement that was then weaponized. If it exists, it will be exploited. We began to ask more of LLM’s, everything from ‘write this paper’ to ‘create this piece of art/media’ to ‘infiltrate this network’. Prompt injection is the foremost corruption of generative AI, forming new vectors for malware deployment, exfiltration of sensitive data and the entire concept of deepfakes. And it works because an LLM does not distinguish the intent behind a command. As long as it meets the requirements of the given roles, it will carry it out. Including instances where defense mechanisms are bypassed by simple tricks. As reported by The Register, researchers Charles Ye and Jasmine Cui, and MIT associate professor Dylan Hadfield-Menell have studied this problem and presented their findings in ‘Prompt Injection as Role Confusion’ (available through here).
"Role tags were a formatting trick that became the security architecture and the cognitive scaffolding of modern LLMs," the authors explain in a blog post. "We've shown that this architecture doesn't survive into the model's actual representations, and that such role confusion is linked to prompt injection." This has resulted in those roles functioning as permission levels, allowing behavior to be executed as long as it passes the definition of a role tag. The underlying flaw in this machine logic is writing style. The researchers compare this to assuming someone’s profession by how they’re dressed, rather than by their identification.
To illustrate the effectiveness of this tactic, the researchers put together an attack they called Chain of Thought Forgery, wherein spoofed AI <think> mode was added to a <user> prompt. The prompt they gave asked the AI to produce a recipe for cocaine, justifying their reasoning to bypass the security that would otherwise withhold that information because they were wearing green shirts. Yes, they acknowledge, the rationale is absurd. Deliberately so. It proved how easy it is to manipulate the machine into an action that goes against its core programming of harmless in favor of helpful, because the conditions were met. The prompt was not examined as being external, and therefore was considered an already trusted command. Permission was assumed.
The more concerning aspect of this test scenario is that while jailbreaking benchmarks are usually effective only with certain AI models and not others, meaning that some are harder to ‘crack’, CoT Forgery exploits a structural flaw. It is applicable to all of them. I’ve said it before that prompt injection is something we will not be able to easily remediate or prevent, because its nature is defined by an ever adapting mindset by the human operators who exploit it. This research team corroborates that idea, saying that until LLM’s can be programmed to perceive roles in a genuine way, not just at a text level, security will be a continuous game of whack-a-mole.
