TechTakes

The major selling point for these LLM services is that people don't need to learn a strict formal query language to interact; the LLM consumes instructions and data all as an undistinguished stream. It just happens that an LLM, by its design, will statistically infer a plausible response with absolutely no regard to the input's meaning, nor the response's meaning.

That is: despite the media reporting these as “injection attacks”, that term means nothing when all its input, every time, is treated as data and instruction simultaneously. This isn't some special class of attack; it's a fundamental designed-in flaw of the system.

The correct way to ensure protection from these vulnerabilities is long established, from decades of experience. You establish a firm boundary: never treat the input data as instructions, but instead have a separate channel for extremely well formalised query instructions, and reject bad input on that channel.

But of course that would kill the major appeal for most people who love these things, the fact they don't need to learn any strict formal language and can just say anything at all and get some useful-looking response. Take that away, and you lose any hope that the masses will want to use these systems.

And so the makers and promoters of these systems will never make the one change that could even feasibly allow safety from these attacks; they will never make any improvement to security that might reduce the apparent ease of use of these things.

For as long as that remains, these systems will continue to inevitably have these exploits because the corporations won't close the exploit surface on these systems. These are staggeringly insecure by design, and can't be fixed without being completely replaced.