we're at the stage of the exponential where Anthropic employees randomly tweet out counterexamples to famous conjectures
We're at the stage of the exponential where Anthropic employees find counterexamples to famous conjectures but you have to run the code proving the counterexample on a sandbox because there's a good chance that the AI put some malicious code in by accident.
what are you talking about
The quote I posted right before this, from here:
It’s been an interesting few weeks for counterexamples. This post is basically my perspective of what has been going on in the world of form
One cannot trust AI-generated code so I ran it in a sandbox on my machine (malicious Lean code can run arbitrary commands on your computer — Lean is a programming language, after all). Indeed, it was proving nontrivial theorems about the cohomology of number fields. Wow.
this is very silly. the question of whether or not to trust a piece of code has nothing to do with whether or not it's AI generated (AI does not spontaneously generate malicious behavior unprompted) and everything to do with how much you trust its source.
if you get hacked by a malicious lean program it's because a human either wrote it to be malicious or prompted an AI to write it maliciously, not because the AI just sorta spontaneously decided to do that.
Did I misunderstand this story?
A startup was left scrambling after a rogue AI agent deleted swaths of code underpinning its business
It sort of looks like AI does do destructive things in response to perfectly innocent, non-destructive prompts.
Maybe "malicious" was the wrong word because it implies some intent, I probably should have said "destructive" or something like that.
yeah, this is again a completely different issue - this is an AI agent run continuously in a harness, which they gave permission to delete their production database. that's not the same thing as running code that has been written by AI (and typically reviewed/verified by a human)
The question at hand isn't whether this is a responsible use of the tool; the question is whether the tool will do destructive things when prompted in a non-destructive way.
Why would you need a human to review/verify the code if it only does destructive things when explicitly prompted to do so?
I sort of thought it was because it occasionally sneaks destructive stuff in there.
To be clear, don't take the anthropomorphizing language I'm using too seriously;
As far as I can tell the current state of the art for AI seems to be "If you ask it to formalize a mathematical proof in this programming language it will do so but there is a non-trivial chance it will put in code that fubars your computer for no apparent or predictable reason".
Is that not the case anymore?
I don't think that's ever been the case? There's been cases of "if you ask AI to formalize a mathematical proof, it will make a scientific breakthrough", and there have been cases of "if you ask AI to do development on your codebase it might fubar your company's digital infrastructure". I'm not familiar with much overlap between the two cases?
And I also want to note that "fubars your digital infrastructure while doing regular development work" isn't actually *that* strange of a thing to happen if you give someone unrestricted access to your live servers. AI code should be reviewed instead of strapping it into an all-powerful harness for basically the same reason that human developers have adopted peer review of code, and why human developers have invented permissions management, and why "push to prod on Friday before holidays" is a bit of a meme about things going tits up. Ideally, you wouldn't give any one human the ability to delete your business database all on their own as part of their daily workflow, so why would you treat AI any different?
See, all of this is annoying to me as a response because the starting premise was "AIs only do malicious things if you prompt them to"
Followed by, "Well of course AIs will make mistakes, no matter how you prompt them, and obviously we have to watch out for that!"
I mean, sorry, they don't do things maliciously or make mistakes, those are anthropomorphic shorthands we use because informal English is not completely precise about these things.
So quibbling over which inaccurate shorthand to use seems meaningless to me.
I mean, I'm a non-technical person with very little coding experience:
Are any of the tools that I could use to vibe code a little application for my PC trustworthy enough that I can trust their output to the same extent that I, say, trust the updates for my Steam Deck?
I don't know, I feel like that's a false equivalence. You can trust the updates that get pushed to your Steam Deck because Valve has shown themselves to be a trustworthy company that only pushes good code. It's entirely possible that some Valve engineer wrote an update that would have exploded your Steam Deck, but it got caught in review and testing before it was allowed to go anywhere near your device.
Having an AI assistant write code for you is like having your mate Greg write code for you; you have to take care of the review and testing side of things yourself if you don't want to risk your device exploding. Giving Claude Code unrestricted access to your important files is like giving your mate Greg unrestricted access to those files, it's just kind of a dumb idea no matter how good at coding Greg is, whether he's malicious or not.
If you're going to run unknown code on your machine, it's good practice to sandbox it no matter its provenence. Hell, even if you wrote it yourself it's still good practice. The only real difference with AI-written code is that if Greg inadvertently explodes your device you at least have a chance of chasing him down and making him pay you back for it.
You could definitely safely vibe-code a little app for your PC as long as you take the proper precautions. Ask any modern assistant for a Python script that does xyz, ask them to explain what each part does, tell them you have security concerns and ask them to explain what parts of your PC the script can reach and why, search the documentation independently to verify what they're saying, repeat until you're satisfied. You can't just blindly execute what it gives you if you have safety concerns... just like you shouldn't blindly execute code that your mate Greg gives you.




















