Sloppy LLM programming? Never!
In completely unrelated news I’ve been staring at this spinner icon for the past five minutes after asking an LLM to output nothing at all:
boooo Gemini now replies “I’m just a language model, so I can’t help you with that.”
what would a reply with no text look like?
nah it just described what an empty reply might look like in a messaging app
they seem to have done quite well at making Gemini do mundane responses
What are the chances that the front end was not programmed to handle the LLM returning an empty string?
LLM vendors are incredibly bad at responding to security issues
My NSFW reply, including my own experience, is here. However, for this crowd, what I would point out is that this was always part of the mathematics, just like confabulation, and the only surprise should be that the prompt doesn’t need to saturate the context in order to approach an invariant distribution. I only have two nickels so far, for this Markov property and for confabulation from PAC learning, but it’s completely expected weird that it’s happened twice.
Not really a security issue I’d say. The AI speaking gibberish when you try to make it speak gibberish isn’t really that big of an issue.
How is it inherently a security issue when an LLM speaks gibberish? Genuine question.
It’s a reasonable question, and the answer is perhaps beyond my ken even though I’ve had substantial experience with both building machine learning models (mostly in pre-LLM times) and keeping computer systems secure. That a chatbot might tell someone “how to make a bomb” is probably not a great example of the dangers they pose. Bomb making instructions are more or less available to everyone who can find chemistry textbooks. The greater dangers that the LLM owners are trying to guard against might instead be more like having one advising someone that they should make a bomb. That sort of thing could be hazardous to the financial security of the vendor as well as the health of its users.
Finding an input that will make the machine produce gibberish is not directly equivalent to the kind of misbehaviour that often indicates exploitable bugs in software that “crashes” in more conventional ways. But it may be loosely analagous to it, in that it’s an observation of unintended behaviour which might reveal flaws that would otherwise remain hidden, giving attackers something to work with.
it “speaking gibberish” is not the problem. the answer to your question is literally in the third paragraph in the article.
if you do not comprehend what it references or implies, then (quite seriously) if you are in any way involved in any security shit get the fuck out. alternatively read up some history about, well, literally any actual technical detail of even lightly technical systems hacking. and that’s about as much free advice as I’m gonna give you.
Correction: I sure hope they’re in charge of security at some place I don’t like.
Lol that’s like expecting gold rushers to be squared away with OSHA, I hope nobody’s surprised here