The exploit shows the extreme risk of offloading technical support to AI.
So on one hand I trust this reporting but on the other hand I'm really not sure what to make of stories like this. What's the chain of decisions that leads to involving a LLM-powered chatbot anywhere in the password reset process, which was already a simple well-understood workflow that looks the same on every website? I know that everyone says "oh they're doing this because it delights the shareholders" but you can incorporate the LLM in half a dozen other features and put out basically the same press release and the investor response would be identical.
From the investor calls I've listened to (and I highly recommend that everyone listen to Big Tech investor calls) a big part of what's going on is that the average tech executive sees the future editorial voice of their platform as a helpful friend who's scrolling through all these pictures and videos with you and is recommending you buy things along the way. Like, the idea is that you and a chatty pal are browsing the Internet together, and that chatty pal is actually a puppet with an advertisement algorithm inside of it, but as much as possible you should feel like it's someone you're chatting with as opposed to the Web 2.0 ideal of social media as a platform that lets people post things and gives other people tools to share or react to those posts, and so you should basically never see buttons and should always be reading (or hearing!) a friendly voice whenever you're in the app. And I'm very skeptical that users will ever learn to love this new mode of interaction, because it adds yet another layer of mediation on the already heavily mediated social media interactions, but I'm certain that it's absolutely terrible for security in a way that can't be mitigated.
The real long-run ideal for social media advertising, incidentally, is that you just point at a thing in a picture you see (could be an ad, could be LLM generated, could be an "organic post") and say out loud "I want that one" and then an LLM recognizes what you mean, buys the item on your credit card, and ships it to your home. I have literally heard a Pinterest executive describe this exact process as their goal, because apparently Pinterest still exists. So in this long-run goal the LLM would also have access to your credit card number and home address, which (as this most recent breach demonstrates) is probably kind of a security risk.
The more I think about this the less I understand how this could ever work with realistic token pricing. Like you either need to make it basically free for a user to say "who made that shirt" a thousand times in a row, or you need to make sure that the user is buying several of those shirts to pay for the tokens. I think the narrative is that the second one will happen, but in general if it's costing the platform upwards of a dollar per user shopping search, that's going to be really hard to recoup.














