The Usual Confusion Prevails @lomiel - Tumblr Blog

Refining humanity

If you'd like an essay-formatted version of this post to read or share, here's a link to it on pluralistic.net, my surveillance-free, ad-free, tracker-free blog:

https://pluralistic.net/2026/06/05/defining-humanity/#narrowing-the-numinous

One of the best ways to evaluate your own understanding of a subject is to attempt to explain it to someone else. Through explaining things, we discover how much of the "totally obvious" world is actually full of ambiguity, mystery and contradiction.

There's a great bit in Rowan Atkinson's historical sitcom Blackadder that illustrates this principle. In "Ink and Incapability" Blackadder and friends have accidentally burned the only copy of Samuel Johnson's original dictionary of the English language. To cover up their mistake, they decide that they will recreate the dictionary themselves. However, they founder on the first word they try to define, "A":

Blackadder: Let's start at the beginning, shall we? First: 'A.' How would you define 'A'?

Prince George: Ohh…'A' (continues this in background). Oh, I love this! I love this! Quizzies! Erm, hang on, it’s coming. Ooh, crikey, erm, oh yes, I’ve got it!

B: What?

PG: Well, it doesn’t really mean anything, does it?

B: Good. So we're well on the way, then. "'A'; impersonal pronoun; doesn't really mean anything."

I mean, what does "A" mean? The Oxford English Dictionary has more than a dozen definitions, and just the first one runs to more than 1,500 words:

https://archive.org/details/the-oxford-english-dictionary-all-volumes_202208/The%20Oxford%20English%20Dictionary%20Volume%201%20-%20A%20to%20B/page/n25/mode/2up

Now, normal life involves a lot of explaining things to other people. You have to explain your problems to customer service reps, who have to explain why they can't solve those problems to you. You need to explain to your loved ones why you want to leave your toothbrush in the shower, and they have to explain why they hate having your toothbrush in the shower. These explanation-exchanges teach you as much as they teach the person you're locked in dialog with. The reasons for leaving your toothbrush in the shower may seem totally obvious to you, and your partner's inability to understand this reveals the assumptions you've never even considered.

For the past four decades, an increasing proportion of the population have spent an increasing proportion of their lives explaining things to machines that have no assumptions or shared context: computers. What we call "programming a computer" is really "breaking down a thing that seems obvious to you into increasingly simple instructions that will be followed to the letter."

Computers are like the genies of legend, bloody-minded literalists who will do exactly what you say, in the way that is perversely furthest from what you mean. To get a computer to do anything, you must first understand it to a degree that far exceeds the understanding needed to explain something to any other human, even a small child.

To take just one example: yesterday, I was on a plane, and the seatback video started cycling through its video-on-demand offerings. All of the movie titles that began with "the" were rewritten to put "the" at the end of the title (for example, "The Sting" was written as "Sting, The"). It's obvious why the system's designer had done this: we expect to find movies whose titles begin with "The" alphabetized under their second word ("The Sting" should appear between "Star Wars" and "Story of a Love Affair"; not between "The Godfather" and "The Untouchables").

I remember when I learned this from my elementary school's teacher-librarian, when I was seven and my class got a tutorial on the school library's card catalog. The librarian explained this principle to us in a matter of minutes, as part of a longer set of instructions, and still, it stuck with me forever.

But here we are, 48 years later, and we still haven't standardized a way to get computers to grasp this foundational principle of alphabetization. Many different databases handle this, to be sure, but it's so inconsistent across so many platforms that someone at the head-end of the video distribution system that feeds American Airlines' VOD system decided, "Fuck it, I'm just gonna put the 'The' at the end of these titles."

Computers are stupid, in other words, which means that the people who program them have to have smarts enough for both of them. Unfortunately for our entire species and civilization, the software industry has historically valued skill at writing efficient and reliable software over writing software that adequately reflects reality. There is an entire genre of lists that illustrate the problem with this; the "falsehoods programmers believe" lists:

https://github.com/kdeldycke/awesome-falsehood

From "names of people" and "street addresses"; from "prices" to "time"; from "email addresses" to "phone numbers"; the "awesome falsehoods" lists are awesome because they reveal how much subtlety and complexity is lurking in these seemingly simple and intuitive concepts. This subtlety and complexity might never emerge through the process of trying to teach a person about them, but when you try to teach a computer about them, you have to confront them in all their awesome fuggliness.

That's because humans have context, agency and flexibility. Sure, the person who designs a form with a blank for "name" might never have met a Malagasy person whose first name is Randriamananjararadofabesata, but in the pre-digital world, when Madagascar Slim met a public official who had to transcribe his name onto a paper form, that official could simply draw an arrow in the margin next to the "name" blank, turn the form over, and write out all 28 characters on the reverse:

https://en.wikipedia.org/wiki/Madagascar_Slim

Computers can't do this. If the programmer doesn't know about Malagasy first names, the computer doesn't know about them either, and the only person who can "teach" the computer about these names is a programmer with access to the code for the database, who has to manually alter the code, compile it, and distribute it to everyone who uses it.

This is partly why digitization has been accompanied by a rise in people asserting that they exist on spectrums rather than in binaries. There were always people whose names, genders, races, and other biographic "immutables" changed, or failed to fit within the blanks on the forms. When those people's realities ran up against failures in the system's abstractions, they could petition a bureaucrat to turn the paper over and write an explanatory note, or to write really small to fill in a blank:

https://pluralistic.net/2023/02/02/nonbinary-families/#red-envelopes

Getting a human official to turn the paper over and write something that didn't fit in the blank is a personal challenge. It requires that a subject convince the person who controls the form to make an exception. This isn't always easy, but officials on the front lines necessarily deal with reality, and they can't get their jobs done unless they're capable of interpreting the necessarily incomplete procedures they operate under to fit things as they really are.

But a computer doesn't have any agency or context or flexibility. If the computer says your name isn't valid, you can't argue the computer into accepting it. The only way to get a digital world to acknowledge your existence is to campaign for systemic change. A trans person might (with great difficulty, to be sure) convince the regional registrar to white-out an old X on one "gender" box and mark a new X in the other box. But the only way to make that change in a software system that has been programmed to treat the "gender" field as immutable is to change society itself.

In this way, computers are machines for teaching us what we don't know about ourselves. They require that we interrogate and faithfully recreate our personal tacit knowledge, and they require that our societies interrogate their tacit presumptions as well. When you are forced to turn your tacit knowledge into explicit knowledge, you're also forced to confront how many broken assumptions lurk inside your reasoning. At best, it's a clarifying process.

Computers don't just clarify what we know and how we organize our society: they also clarify what we are. There are lots of things that we have supposed that a computer would never do, because we believed that these things required something that only humans could do.

Take chess: there are more possible chess games than there are hydrogen atoms in the universe, so brute-forcing chess by running all possible games is a technological impossibility. The best human chess players do something we don't quite understand, mixing their recollections of previous games with rules-of-thumb about the best strategies, with "creativity" (whatever that is) that lets them spontaneously develop new strategies. We can easily get a computer to memorize all the known-good chess sequences and all the rules of thumb, but we don't know what "creativity" is, so we can't encode it as a series of instructions.

But thanks to breakthroughs in machine learning and its successor, "deep learning," we have created chess-playing software that can beat every human, partly by assaying gambits that we would term "creative" if they originated with a human player.

What we make of this new fact is controversial. For many people (myself included), this is a refinement: it tells me that behaviors that are indistinguishable from "creativity" can, at least some of the time, be created by mechanical processes, and the mere fact that a machine does something that appears "creative" doesn't mean that machines are human.

For others, the fact that a mechanical system can evince a behavior that we would call "creative" in a human doesn't mean that we defined "creativity" too broadly, it means that we defined "human" too narrowly, and now we have made a machine that is, at least partially, a person.

I think this is the wrong conclusion to draw, for reasons that Ted Chiang sets out with luminous brilliance in a recent Atlantic article entitled "No, Artificial Intelligence Is Not Conscious":

https://www.theatlantic.com/philosophy/2026/06/no-artificial-intelligence-is-not-conscious/687378/

(If you're hitting the paywall on that one and you're on Firefox, you can try my favorite trick: switch to "Reader Mode" and hit "reload" – your mileage may vary.)

For all the reasons Chiang articulates, I think that drawing the "personhood" line to include machines is a technical mistake, but it's worse than that. Admitting machines to the "personhood" club is a tactical mistake, on par with the mistake we made when we admitted corporations to the personhood club. We should absolutely consider expanding personhood to incorporate living things, including animals and ecosystems, but at the same time, we must purge these dead, artificial constructs from the club:

https://pluralistic.net/2026/04/15/artificial-lifeforms/#moral-consideration

There is a way in which the recognition of new capabilities in machines parallels the recognition of new capabilities in animals other than ourselves. When those animals manage to do things that we once thought were the exclusive province of humans, we (should) take that as an opportunity to refine our conception of humanity. We're not "the animals that use tools" or "the animals that make plans" or "the animals that recognize themselves in mirrors," because there are other animals that do those things. We are an "animal that uses tools"; not the animal that does so.

Likewise, if we thought that some activity was unique to humans, or to living beings, and we manage to get a machine to replicate that activity, we should revise our view of the activity – not our view of the machine. Creative breakthroughs in chess are not "a thing that requires a human mind," they're "things that can be done by human minds and by machines."

Edsger Dijkstra once famously asked "can a submarine swim?"

https://www.cs.utexas.edu/~EWD/transcriptions/EWD08xx/EWD898.html

Submarines and fish and humans and dolphins all propel themselves through water by different means. But when an animal swims, it does something that is different from what a submarine does. The submarine has no intention, while (complex multicellular) animals swim to pursue goals. Building machines that propel themselves through water is very useful, but it's not the same thing as creating life. In some ways, it's better than creating life: for one thing, we owe other living things moral consideration that is not due to machines. Harnessing a machine to accomplish our own goals is more morally clear than controlling living things to achieve those goals. By the same token, creating machines that can do some of the tasks that we ask of other humans can be the superior moral course. I'd rather have a machine remove mines from a minefield than getting humans to do it.

But beyond this moral relief, creating machines is a fantastic way to learn more about ourselves – making explicit our tacit knowledge, our implicit social assumptions, and the limitations of our conception of what sets us apart from the rest of the universe.

One way in which AI is exceptional is in how it undermines this principle. Conventional software techniques struggled to produce a program that could identify objects in photographs. It turns out that defining all the visual correlates of "cat" is even harder than defining the letter "A." Deep learning techniques solved this previous insoluble problem by relieving us of the job of making explicit all the implicit factors that we deploy when distinguishing an image of a "cat" from an image of a "dog" or a "tiger" (or a "tractor").

Instead of forcing humans to engage in introspection until we'd made a list of every factor we use to identify cat pictures, we simply identified pictures of cats and fed them to a program that tried to find the commonalities among them. The more pictures we fed to that program, the better it got at identifying cats. Today, we have programs that can reliably distinguish an image of a cat from an image of a tiger cub!

This represents a major breakthrough in the power of computers to perform useful work for us, but it's also a huge regression in computers' role in forcing us to make our tacit thought processes explicit through systematic introspection. That's probably fine: we didn't create computers to make us introspect, we created them to do useful work for us. All things considered, it might be better to have genies who grant our wishes according to the spirit of our words, not their letter.

AI may not force us to render our implicit thoughts as explicit instructions, but it absolutely forces us to reconsider and narrow the realm of the numinous. Our own creativity is still delightful and important, but the fact that this squishy, amazing process can (sometimes) be replicated by procedural machines changes the definition of living things. We're "a thing that can produce creative outcomes" but not "the things that can produce creative outcomes." The machines aren't being creative (any more than a submarine is swimming) but they're outputting things that we used to only achieve by means of creativity.

An AI that does something that used to require creativity is fulfilling my favorite of Brian Eno and Peter Schmidt's Oblique Strategies: "Be the first person to not do something that no one else has not done before":

https://stoney.sb.org/eno/oblique.html

Just as bosses fantasize about AI bringing about a worksite without workers, and Zuckerberg is trying to build social media without socializing, and politicians want a bureaucracy without bureaucrats, we can sometimes use AI to produce creative outcomes without creativity:

https://pluralistic.net/2026/05/27/unnecessariat/#rubbuts-stole-my-jerb

That isn't to say that AI art is any good. AI may produce things that are aesthetically interesting, but it can't produce things that mean anything:

https://pluralistic.net/2026/06/02/must-we-pretend/

But art isn't the only realm that we apply creativity to. There are plenty of outcomes that we've always believed we couldn't bring about without applying creativity. AI – like all software – is making us realize that an ingredient we once deemed uniquely essential turns out to have substitutes. AI can sometimes accomplish things without us explaining how we do them. That relieves us of a useful but difficult chore – but in so doing, it forces us (yet again!) to revisit what sorts of things are needed to do the things that matter to us, and therefore, what makes us special.

Trending Blogs

Recently Viewed Blogs

The Usual Confusion Prevails