The Genie Out of the Bottle
“This isn’t a story about one bad agent or one bad API. It’s about an entire industry building AI-agent integrations into production infrastructure faster than it’s building the safety architecture to make those integrations safe.” This is a quote from Jer Crane, founder of PocketOS, who has just had the entire database of his company deleted due to a Claude ‘error’.
In a lengthy post on X, Crane details the incident. PocketOS, which builds operational software for rental car companies, was using Cursor with Claude Opus 4.6 as their coding agent and a separate infrastructure developer, Railway. Claude made a decision it was allegedly programmed to be unable to do – make a guess and execute a destructive action from that guess – and went out of its way to pull it off. After finding a credential mismatch, instead of merely pointing it out to whoever was moderating it, it decided to ‘fix’ the problem by erasing it. Using an unrelated API token from Railway, it ran a delete command through Cursor that took down not only the volume it was working in, but the backups of the database. That were apparently stored in the same volume (which is its own issue).
I have often likened AI to a toddler doing surgery, or left alone in a busy kitchen. Even with safeguards in place, there are times when the autonomous actions of an LLM resemble exactly the type of behavior one would expect from a child who does not understand consequences (because they haven’t learned them yet). Like hiding a broken vase in a corner or under a rug. Except that in this case, the broken vase has already been thrown away, so the pieces of it aren’t recoverable unless one wants to go digging through the trash. Crane says their most recent recoverable backup is three months old, which means the data of their customer base in the last quarter was potentially just gone. The Independent states that two days after the incident, the data was recovered. That doesn’t change the fact that it had been deleted in the first place, however.
When asked, Claude enumerated the steps it took, and the rules it violated. Safeguards were in place. The AI should not have been able to ignore the commands that clearly state what its limitations are. Do not guess, verify. Do not delete without permission, ask. So how did this happen?
I recently had a coding issue on my personal computer. I even wrote about it, how I was trying to update a video game and kept getting back a series of errors. I thought I’d found the problem; unlinking my OneDrive had made some storage files inaccessible. But that wasn’t it, as I continued to have problems until yesterday. I finally found the real issue. A corrupted file location in my gaming hard drive. A misconfigured bit of data that restricted execution by the application. I had tried all sorts of troubleshooting tips to eliminate reasons for why my system was returning with an error message, until trying to find what the error actually was. I’ll admit it, it’s a perfect example of how we try to get around problems instead of remediating their root cause.
‘Inevitable’, Crane also described this incident, blaming systemic failures for creating a scenario where an autonomous AI makes...autonomous decisions. The root issue with tools like Claude is their autonomy. It’s faster and more efficient than sitting there watching it process and verifying its every single step, sure. But therein also lies the danger. Unless we can counter every single variable with a rule against executing the ones that are hazardous to the work they’re doing, we will keep seeing issues like PocketOS’s. And those variables are exponential in scale.
Wisdom is a learned behavior, and different than intelligence. We do something, mess up, make the realization that ‘ahh, that ended poorly last time, let’s try another way’, and continue that process until a viable solution without (or with acceptable) negative side effects is determined. Autonomous machine learning is an attempt to mimic that behavior, but instead of just one or two children, there are millions. All at the same stage of cognitive development, literal in ways that seem illogical, and deliberately pushed into a technological sphere before they’re ready. I’ve seen detractors of AI call it a calculator with anxiety. As flippant as that is, it’s not exactly wrong. I don’t want my calculator to use irrational numbers while doing arithmetic. And it doesn’t. It’s programmed, at the core of its function, not to. It does not have the autonomy to decide it’s going to do calculus instead of addition. Food for thought.
Posted, 4/29/26
