I’ve noticed the latest Claude does this thing where it will verbally start a line of thought, often a hedge or raising a complication to an idea, then backtrack and go wait, no, this was right the first time, nevermind. And my sense is that this is a product of the RL trying to make the model better at nuance or whatever, but it’s kind of annoying as a verbal tic.
As a signature of what makes LLM output and reasoning different from humans though it’s very interesting. I’m just spitballing here but I do wonder if it’s a combination of a couple of things—one, LLMs process the whole text in one go to produce the next token and there isn’t by default inherent continuity from one such pass to the next except what’s in the text, so while a human will think about something and go “nevermind” and discard it, and this is an invisible process, for an LLM that kind of mental digression has to be in the output *somewhere.* Using more sophisticated interfaces or harnesses or whatever that put that stuff in a scratchpad just hides it (and if you’re using a simpler, faster model like Sonnet it’s not using those features by default).
But I also get the sense that this is a product of the fact LLMs are pretty smart in some ways but not conscious—that even in more sophisticated setups where you have multiple task-specialized models working together behind the scenes to produce more sophisticated output, they’re all chugging along in a pretty reflexive way. If you let your own lower-level running-on-automatic cognitive systems handle a task, 99% of the time this works great—people navigate from point A to B over considerable distances all the time while their mind wanders, negotiating foot or car traffic all the while, and while your attention can step in if something genuinely unexpected happens often it just doesn’t need to. There are cognitive systems like music that work fine even in people who are suffering massive cognitive impairment.
It’s not a perfect analogy bc I don’t think LLMs resemble human brains very much, and humans can’t do complex tasks like “prove this Erdős problem” by reflex. That’s the sort of thing that requires all our attention usually. And we don’t have to synthesize our reasoning all the way down to the fully structured linguistic form to review it and go “no, that doesn’t sound right.” And having different modules work together to sharpen and improve the final output helps, but I suspect (and I really am way out on a limb here!) that completely discrete modules are a less efficient use of overall processing power than the sort-of-modular-sort-of-not design of the human brain, where different cognitive systems aren’t entirely discrete, specialization is only partial, and the brain itself is fairly plastic. The “modules” here are much more tightly integrated into one another, and because evolution can optimize ruthlessly for efficiency (has to, given it has stark resource constraints), it doesn’t and doesn’t need to care about like elegance of design. Which is a pattern you see in biology all the time.
yousharknotpass
blakekathryn
stormborns
vaporwave
titsay
