Dec 2, 2025
in retrospect, a fairly obvious omission
anybody who is seriously concerned with "AI alignment" as a real problem should probably consider that they still haven't figured out how to ensure sam altman is broadly aligned with human values
seen from Canada
seen from Germany
seen from Georgia
seen from Türkiye
seen from Morocco
seen from Sweden
seen from United States
seen from Kazakhstan
seen from China
seen from United Kingdom
seen from Sweden
seen from Germany
seen from United Kingdom
seen from Hungary
seen from China
seen from China
seen from Netherlands
seen from Hungary
seen from United States
seen from Singapore
in retrospect, a fairly obvious omission
anybody who is seriously concerned with "AI alignment" as a real problem should probably consider that they still haven't figured out how to ensure sam altman is broadly aligned with human values
[Longtermism] was developed over the past two decades by philosophers at the University of Oxford including Nick Bostrom, Toby Ord and William MacAskill. Bostrom is one of the leading advocates of the idea that advanced artificial intelligence is going to kill everyone on Earth, and he was the subject of harsh criticism earlier this year after an old email surfaced in which he claimed that “blacks are more stupid than whites”. MacAskill’s past is chequered too, as he was the moral “adviser” of Sam Bankman-Fried, the cryptocurrency billionaire and diehard longtermist charged with perpetrating what American prosecutors have called “one of the biggest financial frauds in American history”. Longtermism is the brainchild of these people – a group of highly privileged white men, based at elite universities, who have come to believe that they know what’s best for humanity as a whole. Longtermists ask us to imagine the future spanning millions, billions, even trillions of years, during which our descendants have left Earth and colonised other star systems, galaxies and beyond. Though Earth may only remain habitable for another one billion years or so, at which point the sun will grow too luminous for us to survive, the universe itself won’t stomp out the flames of life for an estimated 10¹⁰ years – that’s a 1 followed by 100 zeros, an unimaginably long time.
[...]
Bostrom, the father of longtermism, has written that we shouldn’t shy away from preemptive violence if necessary to protect our “posthuman” future, and he argued in 2019 that policymakers should seriously consider implementing a global surveillance system to prevent “civilisational devastation”. More recently, Bostrom’s colleague Eliezer Yudkowsky contended that pretty much everyone on Earth should be “allowed to die” if it means that we might still reach “the stars someday”. He also claimed in Time magazine that militaries should engage in targeted strikes against data centres to stop the development of advanced artificial intelligence, even at the risk of triggering a nuclear war. When I was a longtermist, I didn’t think much about the potential dangers of this ideology. However, the more I studied utopian movements that became violent, the more I was struck by two ingredients at the heart of such movements. The first was – of course – a utopian vision of the future, which believers see as containing infinite, or at least astronomical, amounts of value. The second was a broadly “utilitarian” mode of moral reasoning, which is to say the kind of means-ends reasoning above. The ends can sometimes justify the means, especially when the ends are a magical world full of immortal beings awash in “surpassing bliss and delight”, to quote Bostrom’s 2020 “Letter from Utopia”.
Really great news! They closed the stupid nazi Future of Humanity Institute at Oxford, and the lead guy quit.
"The closure of Bostrom’s center is a further blow to the effective altruism and longtermism movements that the philosopher has spent decades championing, which in recent years have become mired in scandals related to racism, sexual harassment and financial fraud. Bostrom himself issued an apology last year after a decades-old email surfaced in which he claimed “Blacks are more stupid than whites” and used the N-word."
Nick Bostrom’s Future of Humanity Institute closed this week in what Swedish-born philosopher says was ‘death by bureaucracy’
FYI this has been going on— (cw for screenshots from a racist rightwing blog)
[image IDs: several screenshots from a blog post by Noah Carl, a rightwing “researcher” of racial pseudoscience, regarding “Nick Bostrom’s pre-emptive apology”
2nd screenshot of Carl’s statement shown above:
Nick Bostrom is a philosopher at Oxford who works on topics like existential risk and human enhancement. I haven’t read much of his work, but people I respect rate it very highly. Anatoly Karlin (whom I had on the podcast recently) considers him “the greatest living philosopher”.
A few days ago, Bostrom posted a document titled ‘Apology for An Old Email’ on his website. The document was subsequently shared on Twitter by his colleague Anders Sandberg, apparently at Bostrom’s request. It begins:
> I have caught wind that somebody has been digging through the archives of the Extropians listserv with a view towards finding embarrassing materials to disseminate about people … I fear that selected pieces of the most offensive stuff will be extracted, maliciously framed and interpreted, and used in smear campaigns. To get ahead of this, I want to clean out my own closet, and get rid of the very worst of the worst in my contribution file.
The email in question, which was sent “in the mid 90s” as part of a discussion about “offensive communication styles”, is as follows:
> I have always liked the uncompromisingly objective way of thinking and speaking: the more counterintuitive and repugnant a formulation, the more it appeals to me given that it is logically correct. Take for example the following sentence:
> Blacks are more stupid than whites.
[I cut off the rest of Carl’s screenshot of Bostrom’s email, as it contained more unpleasant antiblack commentary]
3rd screenshot:
You don’t have to apologise for saying offensive things in a setting that people have selected into for the specific purposes of discussing offensive things. Stand-up comedians don’t need to apologise for telling jokes at their shows that it would inappropriate for them to tell in church. Moreover, Bostrom made the comments more than twenty years ago, and he “immediately apologised” at the time! End of story.
Yet as you well know, academia is crawling with offence archaeologists – low-lifes who spend their time combing through other people’s writing with the hope of finding something they can use to ruin their careers. They are not virtuous, and they do not care about the downtrodden. Their aim is simply to “take down” someone whose views they disapprove of – usually someone who contributes far more to society than they do.
In light of this, you can understand why Bostrom wanted to “get ahead” of the controversy by saying his piece pre-emptively. Unfortunately, what he said may have made things worse – not only for himself but for others who might find themselves in similar situations in the future.
4th:
Rather than making the points I made above (and perhaps apologising for needing to bring the admittedly provocative email to people’s attention), he issued an embarrassingly grovelling apology:
> I completely repudiate this disgusting email from 26 years ago … The invocation of a racial slur was repulsive. I immediately apologized for writing it at the time … and I apologize again unreservedly today. I recoil when I read it and reject it utterly.
As for his “actual views”, Bostrom thinks “it is deeply unfair that unequal access to education, nutrients, and basic healthcare leads to inequality in social outcomes, including sometimes disparities in skills and cognitive capacity”. And he wants you to know that he has given to charities “fighting exactly this problem”, including “the Black Health Alliance”.
The one saving grace of his apology – from the perspective of grown-up intellectual discourse – was that he didn’t denounce the hypothesis that genes contribute to group differences in cognitive ability. “It is not my area of expertise”, he wrote, “and I don’t have any particular interest in the question.” Note: the latter claim is likely to be false; how could you not be interested in it?
/end image ID]
https://twitter.com/rechelon/status/1615072322058076166
[image ID:
tweet by xriskology:
The real victimhood culture is on the political right:
[a screenshot from the same Noah Carl blog post shown above]
thread QRTing their tweet:
“That which can be destroyed by the truth should be.”
My line is that your career can be destroyed by exposing the truth (that you’re still soft on vapid racial pseudoscience 25 years after endorsing it), then it deserves to be.
I find it deeply disgusting and infuriating the way that so many “rationalists” are framing everything in terms of “he said a bad word and immediately apologized 25 years ago” when that’s not even fucking remotely the issue and would amount to almost no blowback on its own.
I think almost no one gives a shit about Bostrom using the n-word 25 years ago, his apology for that isn’t the issue. The issue is that his recent apology is stunning in what it studiously doesn’t renounce and the way it leaves claims about genetic racial intelligence open.
I’m well aware that Bostrom makes the perfectly fine transhumanist move that “even if racial intelligence disparities were a thing, they’re irrelevant because everyone can individually tinker with themselves in whatever direction” but it’s not enough to merely bracket the claims.
Bostrom’s “apology” sounds perfectly fine to him and his circles because “I am not an expert on racial science” sounds like a renunciation of his claim “blacks are stupider than whites” to them. But it’s anything but! It doesn’t address at all what he still believes.
The fact is that the specific measures the rationalist community chooses to valorize, certain limited performances of rationality, do not exclude shitbags willing to make some contortions so those shitbags flock into their circles, expelling others, and shaping background norms.
I’d be willing to bet at sharp odds that >50% of the self-identified rationalist community believes “there are significant genetic racial disparities in intelligence between the races and unfairly we can’t talk about this.”
That is the problem Bostrom’s non-apology exemplifies
Now there are myriad good mechanisms and reasons to dismiss the object-level claim, but sure, ghosts could exist and it could be true, the biggest problem is the way this “likelihood” has been drastically inflated within the rationalist community by their social dynamics.
The rationalist community is running the proverbial bar where they let nazis in. And if you fail to draw the line with one nazi, you very rapidly just have a nazi bar. Because nazis are large in number, have few other options, and are willing to go through a lot of contortions.
Nazis will absolutely shit themselves silly writing endless papers throwing chaff everywhere like the “200 proofs the earth is not flat” and credulous “rationalist” bros whose whole self-image prioritizes feeling smarter than everyone and holders of esoteric truths love that.
And so we see vast asymmetries and distortions in the epistemic frames of most “rationalists” as a consequence of their sociological dynamics. They proactively read anti-woke folks on the right and then pretty much never delve deep into radical leftist arguments.
They’ll delve into the most esoteric neoreactionary screeds with giant bibliographies of catholic and postmodern writers to find the secret actual argument for an inane conservative position, but then assume every leftist argument is the first related tumblr post they found.
Scott Alexander Siskind BRAGGED that his readership was fair and balanced because he had some socialists and only like 22% identified openly as alt-right or neoreactionary. That is not balance. That is a nazi bar. And that social fabric warps one’s epistemology.
Bostrom honestly thought he was addressing the problem with his racist email in the 90s. And Sandberg (god damnit dude) read it and was like “this is a knockout response I want to be associated with” and hordes of fuckers saw the same.
reply to the thread by zorangecats:
One observation I’ve made is I’ve never seen the rationalist “steelman” for the feminist worldview. An interesting omission.
/end image ID]
https://twitter.com/rechelon/status/1614428504581607424
[image ID:
tweet by Aella_Girl:
Lost some respect for EA due to the response to the Bostrom thing. I guess I shouldn’t be surprised? CEA has always leaned a bit too far towards PR at the expense of integrity. I guess I’d hoped to see some more bravery. Feel like the correct response would have had more nuance.
thread QRTing her tweet:
The lost “integrity” she’s bemoaning is that the Centre for Effective Altruism released a very thin anodyne statement condemning Bostrom’s “words” (presumably just his original email and no detail on whether the racial pseudoscience in particular sucks)
Imagine being so fucking twisted that you see condemning racist shit to any minor degree as a lack of integrity.
These people’s entire morality is about fiercely condemning and ostracizing anyone who ever dabbles in condemning or ostracizing people over shit like racism, rape, and transphobia.
It’s pure Coalition Of The Bad shit.
/end image ID]
Far from being the smartest possible biological species, we are probably better thought of as the stupidest possible biological species capable of starting a technological civilization - a niche we filled because we got there first, not because we are in any sense optimally adapted to it.
Superintelligence: Paths, Dangers, Strategies by Nick Bostrom
Far from being the smartest possible biological species, we are probably better thought of as the stupidest possible biological species capable of starting a technological civilization—a niche we filled because we got there first, not because we are in any sense optimally adapted to it.
Nick Bostrom, Superintelligence: Paths, Dangers, Strategies
Let's say you have been promoting some view (on some complex or fraught topic – e.g. politics, religion; or any "cause" or "-ism") for some time. When somebody criticizes this view, you spring to its defense. You find that you can easily refute most objections, and this increases your confidence. The view might originally have represented your best understanding of the topic. Subsequently you have gained more evidence, experience, and insight; yet the original view is never seriously reconsidered. You tell yourself that you remain objective and open-minded, but in fact your brain has stopped looking and listening for alternatives.
Nick Bostrom, Write Your Hypothetical Apostasy
Thread by @0x49fa98: Our ruling class has fully embraced the danger of information hazards. In fact, they have decided they are so dangerous that they now advise you to stop thinking all together. The WHO to advise...…
Our ruling class has fully embraced the danger of information hazards. In fact, they have decided they are so dangerous that they now advise you to stop thinking all together. The WHO to advise that you wear a mask on your brain.
…
An information hazard is a piece of true information that causes some harm to the person who learns it. Bostrom identifies six types hazardous information transfer: data, ideas, templates, signals, attention, and evocations.
…
In addition to the infohazard typology by information transfer, we can also classify infohazards by the type of risk they present: adversarial risks, market risk, error risk, psychological risk, information system risk, and development risk
…
When we evolved the power to understand and propagate arbitrary information, we became vulnerable to information hazards. This was a novel danger; only humans can be hurt by true information.
…
We can build AIs today that are capable of thinking, but no one would accuse them of reasoning. This is not because they lack intelligence, but because they lack motivation.
An AI was programmed to seek novelty, and this made it capable of solving a maze, but it encountered the distinctly human problem of procrastinating in front of a TV.
A cynical man might suggest that our motivation mechanisms are also this crude
…
In most cases, biasing hazard is less about things that pull you from the truth, and more about things that pull you from the center of social consensus. This is rule zero of power, which you've heard many times before.
…
Infohazards are so pervasive and dangerous that we have evolved a defense against them in the form of strategic epistemic failure.
It's hard to believe things that go against social consensus, when you know they are signaling hazards. If someone presents you with an ironclad case for a socially dangerous idea, you will tend to doubt it or dismiss it. You might agree one moment, and forget the next.
For example, with Gellman amnesia you forget a disturbing observation the moment it leaves your field of attention. How many other jarring revelations don't stick in your head this way? You'll never know
…
My favorite example is this study, which shows we don't believe scientific studies that come to negative conclusions about women. If this finding doesn't match your predilections, you will also surely dismiss it
…
And why not? The truth is all persuasion is grounded, not in reason, but in strength. When a logical argument convinces you of something, it's not the mechanics of reason that does it, it's the display of power that reason presents.
A skilled speaker not only projects power, but offers you power. "Think as I do, and you can wield some of my power." All persuasion is seduction. If the truth smells of weakness, most of us follow our nose. If the news is no longer convincing, that suggests a loss of power.