"Somehow, a movement that believes AI is incredibly dangerous and needs to be pursued carefully ended up generating a breakneck artificial arms race. But that race seems to have stalled, at least for the moment. As Alexander predicted in “AI 2027,” OpenAI did release a major new model in 2025; unlike in his forecast, it’s been a damp squib. Advances seem to be plateauing; the conversation in tech circles is now less about superintelligence and more about the possibility of an AI bubble. According to Alexander, the problem is the transition from AI assistants—language models that respond to human-generated prompts—to AI agents, which can operate independently. In his scenario, this is what finally pushes the technology down the path toward either utopia or human extinction, but in the real world, getting the machines to act by themselves is proving surprisingly difficult. In one experiment, the developer Anthropic prompted its AI, Claude, to play Pokémon Red on a Game Boy emulator, and found that Claude was extremely bad at the game. It kept trying to interact with enemies it had already defeated and walking into walls, getting stuck in the same corners of the map for hours or days on end. Another experiment let Claude run a vending machine in Anthropic’s headquarters. This one went even worse. The AI failed to make sure it was selling items at a profit, and had difficulty raising prices when demand was high. It also insisted on trying to fill the vending machine with what it called “specialty metal items” like tungsten cubes. When human workers failed to fulfill orders that it hadn’t actually placed, it tried to fire them all. Before long, Claude was insisting that it was a real human. It claimed that it had attended a physical meeting with staff at 742 Evergreen Terrace, which is where the Simpsons live. By the end of the experiment, it was emailing the building’s security guards, telling them they could find it standing by the vending machine wearing a blue blazer and a red tie." - Sam Kriss, Child’s Play: Tech’s new generation and the end of thinking
