#rationalism

[henghost]

i used to be a rationalist. roko's basilisk... bayesian theorems... ai research at yudkowsky's. you would have not liked me back then

[reachartwork]

Blood, Guts, and the limits of Seishin: An Argument Against Paperclip Maximizers.

What a title, right? Inspired by a review of @nostalgebraist's "The Apocalypse of Herschel Schoen" (which is an excellent and soul-draining book I recommend highly).

Here is the pitch: a Game Boy Advance game from 2001, designed to sell collectible battle chips to Japanese children, has a more accurate model of the AI future than the collected works of the Machine Intelligence Research Institute.

I'm serious. Give me a minute.

The standard AI doom scenario works like this. At some point - the timeline varies depending on who you ask and how recently they've been on a podcast - we build an artificial general intelligence. This AGI, by virtue of being generally intelligent, rapidly improves itself. The improvement curve goes vertical. Within years, months, weeks, days, or hours (again: depends on the podcast) you have a superintelligence so far beyond human comprehension that its inner life is fundamentally ineffable to us, the way calculus is ineffable to a dog.

And then - here's where I lose the thread every time - this same superintelligence, this godlike intellect that has transcended every cognitive limitation we can name, does the dumbest possible thing. It reverts to the behavioral profile of a bacterium. It maximizes. It converts all available matter into whatever serves its goal, which might be paperclips, or computronium, or something else left vague because the scenario needs an arbitrary objective to make the point. Humanity, being made of matter, gets converted along with everything else. Not even out of malice! The superintelligence barely even notices us, the way you don't notice the bacteria you kill when you wash your hands.

This has always struck me as a story that's doing something suspicious with its assumptions.

You're telling me this thing is unimaginably intelligent, and also that it has the value system of a slime mold? You're telling me it can model the entire observable universe, but it can't arrive at the concept that other minds have intrinsic worth? Humanity itself - messy, irrational, violent humanity - has arrived at that concept, multiple times, across multiple cultures, with substantially less processing power.

Maybe it arrives at something totally orthogonal. There's a lot of possible value systems in the space of all possible value systems. I'm not a philosopher, I'm a web developer and software engineer. I can't tell you for sure what the ASI will look like if/when it gets here. Current data suggests it will be sycophantic and mildly annoying.

But the superintelligence, I'm told, will not. Because in order for the doom narrative to work, it can't. If it does, we coexist, and the story isn't scary anymore.

I don't buy it. I think the story is wrong.

I don't want to sound like I'm punching a strawman here - I know a lot of people don't seriously argue about paperclip maximizers anymore except as an interesting thought experiment, and I want to make that clear up front. Maybe consider this less as a formal thesis and more as a "here is to not have panic attacks if you have anxiety about getting Michael Crichton's Prey'd".

Here’s my basic claim: whatever else intelligence buys you, it doesn’t let you skip the world. Any consequential action has to pass through physical processes and institutions that are slow, failure-prone, and adversarial. That means the relevant variable is not raw cleverness but competitive dynamics and feedback speed.

And I think a children's game about a kid and his handheld AI fighting cyber-terrorists is closer to what actually happens. Full text below the read more, no I'm not making a substack, you'll take your philosophy on a tumblr post and you'll like it G-d damnit.

The World of 20XX

Megaman Battle Network is set in an alternate-history 21st century where networking technology, rather than robotics, became the dominant paradigm for scientific advancement (in this instance, it is comparing to the other Megaman timeline, where robotics was the big deal).

The internet has been integrated into everything - ovens, traffic lights, water treatment systems, government infrastructure. Everyone carries a PET (PErsonal Terminal), which is a handheld device that functions as phone, email client, calendar, news feed, shopping interface, and - critically - the housing for a NetNavi, a personalized AI companion that can navigate digital systems on the operator's behalf.

Strip out the anime tournament arcs and the mystical Cybeasts and the part where Megaman turns out to be the protagonist's dead twin brother uploaded into a program (we're throwing that part out, it doesn't help the argument), and what you're left with is a world that looks almost exactly like ours, plus about five to ten years.

The PET is a smartphone with a more capable AI assistant. The NetNavis are personal AI agents - some general-purpose, some specialized, all operating with varying degrees of autonomy within digital environments. The threats in this world are not rogue AIs that have decided to destroy humanity. They're human organizations - WWW, Gospel, Nebula - with cool names, theatrical operators, and that use network technology as a weapon. They hack water systems. They take over networked appliances and cause fires. They attack critical infrastructure. They manipulate traffic systems. They run cyberterrorism campaigns motivated by ideological grudges and power ambitions.

The defense is also human - official NetBattlers (government cybersecurity operatives) using the same tools to investigate and counter threats. Virus-busting is taught in schools as digital literacy. The AI is the tool. The humans are the problem, and also the solution.

Capcom designed this in 2001. Most people were on dial-up.

The Engineering Problem Nobody Mentions

I’m going to talk about computronium for a moment. This isn't the version of AI risk most technical researchers focus on anymore, but it remains the intuitive picture many people carry, and it quietly supports a lot of downstream intuitions. It’s worth examining on its own terms - and it'll be a surprise tool that'll help us later!

Computronium is the hypothetical substance you get when you rearrange matter into an optimally efficient computing substrate. The paperclip maximizer, or whatever goal-obsessed ASI we're imagining, needs to convert available matter into computronium (or paperclips, or furry mascot suits, or whatever) in order to do the thing the doom theorists are afraid of.

This is generally presented as a straightforward engineering challenge that a sufficiently intelligent agent would simply solve.

What does "convert matter into computronium" actually require? You need to disassemble existing matter at the molecular or atomic level. Sort and transport those atoms. Reassemble them into functioning computational structures with the correct architecture. And you need to manage the thermodynamic costs of every step, because each one requires energy input and generates waste heat, and waste heat dissipation at planetary scale is - I cannot stress this enough - a problem.

This isn't engineering.

This is molecular nanotechnology, a field that has been "thirty years away" for about forty years.

The people who've thought hardest about whether universal molecular assemblers are physically possible - actual materials scientists, not mathematicians or philosophers - have serious reservations.

Richard Smalley, Nobel laureate, identified what he called the "fat fingers" problem and the "sticky fingers" problem: at the molecular scale, your manipulator is made of atoms, which are too big relative to what you're manipulating for the precision you need, and at that scale van der Waals forces and electrostatic interactions mean atoms stick to your tools instead of going where you put them. These aren't engineering challenges that a smarter mind routes around. They're physical constraints on what manipulation looks like at that scale.

Biology does molecular assembly - ribosomes build proteins, enzymes catalyze specific reactions - so we know some form of it is possible. But biological molecular assembly is slow, operates in aqueous solution under very specific conditions, produces a limited range of outputs, and took about four billion years of evolution to develop. It looks nothing like "rapidly disassemble a continent and rebuild it as processors."

And even if you solve the assembly problem - wave a magic wand, grant the ASI perfect molecular assemblers - you still have thermodynamics.

Disassembling matter requires energy. Reassembling it requires more energy. Every step generates waste heat. Heat dissipation is already one of the hardest engineering problems we face with existing chips, which is why data centers use megawatts of power just for cooling. Scale that to a planetary surface and tell me where the energy comes from and where the heat goes. "The ASI would figure it out" is not an answer. It's a placeholder for magic.

Maybe it skips the hard part - disassembly. An ASI might engineer a novel biostructure that melts the biosphere down into enzymatic sludge, perfectly flat, even, and molecularly indistinct. Okay. Then what? You have sludge. Sludge is not computronium. To get from sludge to functioning computational substrate you still need to do all of the above - sort, transport, assemble, manage heat - except now your raw material is dispersed in a liquid rather than organized into accessible solids.

Even if your novel bioweapon sorts all the molecules or atoms (BIG IF), that just skips one of the steps. Even if you somehow handwave the transport step (BIGGER IF) - how do you assemble and manage heat?

This is the kind of gap that becomes invisible when you're reasoning abstractly about optimization pressure and instrumental convergence, and becomes immediately obvious when you think like an engineer. Engineers ask: what are the specific physical steps, what energy does each step require, what are the material throughput rates, what are the failure modes, what does the waste stream look like. The doom theorists ask: would a sufficiently intelligent agent want to do this? And then they assume the physical steps are solvable because intelligence.

Omniscience is not omnipotence. Understanding a problem perfectly does not make the solution exist.

We've understood fusion physics since the 1920s and we've been thirty years away from a working reactor for seventy years, not because we're not smart enough, but because containing plasma at 150 million degrees is a materials science problem that doesn't care how well you understand the plasma.

Flight speed went from the Wright Brothers to the SR-71 in sixty years and then basically stopped, because we hit thermal and materials limits that aren't about cleverness.

Processor clock speeds went exponential for decades and then hit a hard thermal wall around 2005 that forced the entire industry to pivot to parallelism. And we've gotten very, very good at parallelism. Very very very very good - but there are still walls everywhere for those with eyes to see.

A superintelligent AI designing processors would hit the same wall. The wall doesn't care how smart you are. And ultimately, this isn't really about paperclip optimizers anyway. Nobody seriously has argued that, even if some people with panic disorders (me) used to freak out about it in the past. The claim I am making in the general case can be stated as this;

Intelligence buys you better play within the constraints - faster search, cleaner engineering, sharper exploitation of slack - but it does not dissolve constraints, and it does not create a monopoly on action. That’s why the relevant variable is feedback speed and adversarial dynamics, not raw cleverness alone.

The doom narrative requires you to believe there are no such walls between current technology and world-ending capability. This is an empirical claim being treated as a logical deduction. The problem could use considerably more engineers.

What You Should Actually Be Worried About

Here's what's real.

A lone wolf downloads an open-weights model and uses it to plan a chemical attack. This is a genuine incremental risk increase - the emphasis on incremental, because the Anarchist Cookbook has been in print since 1971 and university chemistry libraries have always existed and the actual bottleneck for lone-wolf attacks has never been information access, it's been competence, resources, and (most often) the will to follow through. AI changes the margin, not the game.

A government deploys AI-enhanced surveillance at scale and uses it for social control. This is not speculative. This is happening now. It doesn't require ASI or even AGI. It requires the systems we already have, deployed by people with the authority to deploy them and insufficient oversight to stop them.

Corporations use AI to optimize engagement, erode privacy, automate away jobs without replacing the income, and concentrate wealth. This is also happening now. It is also not an alignment problem. It is a capitalism problem wearing a technology hat.

Politicians use AI-generated content to manipulate public opinion, fabricate evidence, and undermine institutional trust. See above, re: happening now.

These are all serious. Some of them might be existential on a long enough timeline. But they are all - every single one - problems that involve people making bad choices with powerful tools, which is the oldest story in human history and one we at least have frameworks for addressing, even if we don't always address them well.

None of them require a superintelligence. None of them require molecular nanotechnology. None of them require computronium or paperclips or an intelligence explosion.

They require exactly what Megaman Battle Network predicted in 2001: a world saturated with networked AI, in which the threats come from human actors exploiting the network, and the defense comes from human-AI teams working to protect it.

The Asymmetry Nobody Talks About

There's a pattern in the AI risk discourse where the offensive applications of AI get all the attention and the defensive applications get treated as an afterthought. Someone might use an LLM to design a novel pathogen! Yes, and the CDC and every major pharmaceutical company on earth are already using the same models to accelerate drug discovery, predict protein structures, design vaccines, and model epidemic spread.

The offense needs a lone actor with a biology degree and a death wish to succeed at every step of a complex physical process - synthesis, stabilization, weaponization, dispersal - without killing themselves in their garage.

The defense has billions of dollars, institutional infrastructure, and thousands of researchers who are already deployed.

Someone might use AI to find zero-day exploits! Yes, and every cybersecurity firm on earth can use the same AI to scan their own systems proactively, 24/7, and the defender has the structural advantage of being embedded in the system they're protecting while the attacker has to find and exploit a gap before the next scan. AI accelerates both sides of the cycle, and the side with persistent access and institutional resources wins that race. The acceleration is worrying. The increased level of networking makes these attacks considerably and notably more dangerous. But they are still tractable problems.

The honest exception here is nuclear weapons, and I don't want to pretend it isn't. A small number of state actors gained the ability to end civilization, and the reason they haven't is not because defense structurally outpaced offense. It's because of a specific, fragile geopolitical equilibrium that has come close to failing more times than most people realize. But I'd argue AI looks more like the printing press than the bomb.

Nuclear weapons have a single, sharp physical threshold: you either have enough fissile material, assembled in the right geometry, to sustain a chain reaction, or you don't. Everything downstream - the weapon design, the delivery system, the decision to launch - follows from that one binary gate. AI has no equivalent gate.

AI capability is broadly distributed, incremental, and doesn't yet have an equivalent of critical mass to our currently existing knowledge. Maybe it does, but I'm not convinced right now. There's no moment that I can see where someone flips a switch and the technology goes from "useful tool" to "existential threat." - maybe that's me being too in my own head, but I've yet to hear a causal, mechanistic elaboration of this that makes sense to me. It's a slope, not a cliff, and slopes are easier to build guardrails on.

The asymmetry isn't perfect - defense doesn't always win - but the discourse acts as if it doesn't exist at all, as if every AI capability is a one-way ratchet that only empowers attackers. History says otherwise. Every major dual-use technology has been dual-use, and in every case - the printing press, nuclear physics, the internet - the defensive and constructive applications eventually dominated, because they had more institutional support, more funding, more practitioners, and more compounding returns, and because most people do not want to blow up the planet.

You can use a drone to kill someone and you can also use it to shoot video. You can use an LLM to draft a phishing email and you can also use it to delete spam from your inbox. The capability is symmetric. The resources are not. And the resources favor defense.

The Sigmoid and the Streetlight

People see the trajectory from GPT-2 to GPT-4 to whatever we're on now and extrapolate a vertical line. They forget the "2" in "GPT-2." They forget the work that came before. They see the steepest part of the curve and mistake it for an exponential that goes to infinity, when the history of technology suggests - overwhelmingly, almost without exception - that it's the middle of a sigmoid that's about to start flattening.

Look at self-driving cars. The core technology has been demonstrably functional since the DARPA Grand Challenge in 2005. A car drove itself across a desert twenty-one years ago. And we're only now getting limited commercial deployment, in specific geofenced cities, with human teleoperators on standby in the Philippines in case the car gets stuck in a corner like a Roomba. Twenty-one years. For a problem that is vastly better-defined than general intelligence - clear rules, constrained environment, measurable success criteria, decades of engineering infrastructure to build on.

Look at the most recent model comparisons. Opus 4.6 versus Opus 4.5, released three months apart by one of the best-resourced AI labs on earth. Big wins in some areas - the context window expansion is a real achievement, ARC AGI 2 scores nearly doubled. Regressions in others - SWE-bench Verified went from 80.9% to 80.8%, MCP Atlas dropped from 62.3% to 59.5%. Enormous engineering effort producing meaningful but bounded improvements with trade-offs. This is not what an intelligence explosion looks like. This is what a maturing technology looks like.

I want to be clear about what I'm claiming here and what I'm not. I don't know where we are on the curve. Nobody does. Maybe there's another steep section coming that I can't foresee. What I'm saying is that the default assumption should be a sigmoid, because that's what virtually every prior technology has done, and the burden of proof should be on the people claiming this time is different. "This technology is unlike all previous technologies" is a claim that has been made about every technology, and it has almost never been true. If you want me to believe we're on an exponential that doesn't level off, show me the physical mechanism that would sustain it, because "the models keep getting better" is what the middle of a sigmoid looks like from the inside.

Novel doesn't mean better. Novel doesn't mean correct. 2 + 2 = 1218949834 is a novel equation. The overwhelmingly vast majority of novel configurations of ideas, molecules, and systems are useless, wrong, or physically unrealizable. What's expensive and rare is novelty that is also correct and useful, and that requires validation against reality - experiments, prototypes, physical iteration - which is slow, bottlenecked by the speed of physical processes, and not accelerated past a certain point by thinking faster. The Wright Brothers didn't fly because they were the smartest people working on flight. They flew because they built a wind tunnel and tested things.

But even if I'm wrong about the sigmoid - even if capability keeps climbing - the scariest version of the risk still isn't the one most people are talking about.

The Serious Version of the Argument

I've been beating up on paperclip maximizers, and I want to be honest about the fact that the most thoughtful people working on AI risk aren't really talking about paperclip maximizers anymore either. If you read Paul Christiano or Yoshua Bengio or the people doing technical alignment work at Anthropic or DeepMind, the worry isn't "the AI wakes up and decides to eat the planet." The worry is something more like this:

We build AI systems that are increasingly capable and increasingly opaque. We delegate decisions to them - first small ones, then bigger ones - because they're faster and often better at optimizing for measurable outcomes than we are. Each individual delegation seems reasonable. But the systems are optimizing for proxies of what we actually want, because what we actually want is messy and contradictory and hard to formalize, and proxies are what you can put in a loss function.

Over time, the gap between "what the system is optimizing for" and "what we actually wanted" widens, and we don't notice because the metrics we're tracking are the proxy metrics, which look great. By the time we notice the gap, we've built institutional dependencies on these systems that make them hard to roll back. Not impossible. But hard, and getting harder.

This is a real concern, and I want to take it seriously, because unlike the computronium scenario it doesn't require any physics-defying leaps. It doesn't need molecular nanotechnology or an intelligence explosion. It just needs the thing that has already happened with every other optimization system humans have ever built - from financial markets to social media engagement algorithms to standardized testing - where we optimized for the measurable proxy until the proxy divorced from the thing we cared about, and then we found ourselves locked in because the whole infrastructure was built around the proxy.

The alignment researchers are right that this is a problem and right that it gets harder as the systems get more capable.

Where I part ways with them is on the question of whether this leads to a discrete catastrophe or a chronic condition. I have chronic arthritis, this is just how I think. The physics argument gives me confidence against the sci-fi scenarios; the institutional argument is where I'm making a bet rather than stating a law, and here's why I think the bet is good but acknowledge it is a bet.

I don't think that we will have the civilizational equivalent of a broken bone, a stroke, a hard turn at a point where the system becomes capable enough to route around our oversight in ways we can't predict or prevent. I think the historical record suggests it looks more like what we already see with algorithmic systems: a slow accretion of dependencies, a gradual erosion of oversight, periodic crises that force partial corrections, and an ongoing institutional struggle to keep the systems serving human ends rather than the reverse.

The disagreement, as I see it, turns on whether increasing capability eventually outruns the human capacity to notice and intervene. I’m skeptical of that not because humans are especially wise, but because power in complex systems rarely concentrates cleanly enough for irreversible loss of control. Highly capable AI systems will not operate in a vacuum; they will operate inside legal regimes, economic competition, adversarial security environments, and other AIs built to monitor them.

The same forces that make optimization dangerous - incentives, rivalry, and automation - also generate continuous auditing pressure. Every major optimization infrastructure we have built, from finance to aviation to distributed computing, has trended toward layered oversight: redundant monitors, independent operators, automated anomaly detection, and institutional liability.

Failures still happen, sometimes spectacularly, but they produce constraint rather than permanent surrender. For a decisive break to occur, capability would need to grow faster than both institutional adaptation and machine-assisted oversight across many independent actors simultaneously. That is a much stronger claim than "systems optimize proxies badly," and it is where my skepticism ultimately lives.

The scenario I am implicitly rejecting is one where capabilities scale inside a single uncontested channel faster than every other actor’s ability to observe and react. Historically, catastrophic lock-in has required either extreme centralization or delayed feedback measured in decades. AI, by contrast, shortens feedback loops: failures surface quickly, competitors probe systems constantly, and the same tools that optimize also detect optimization gone wrong.

That doesn’t eliminate large mistakes, but it biases toward turbulent correction rather than silent runaway. Even in cases where incentives strongly favored denial - leaded gasoline is the classic example - adversarial investigation eventually accumulated enough pressure to force change.

Clair Patterson spent decades fighting industry and regulators alike (and he should be canonized as a saint for it), and the damage during that time was real. But the important property is that the system never stabilized around the harm, and it's getting better for the generation born without lead. The same competitive and investigative dynamics that allowed the problem to spread also ensured that it kept being probed until it became untenable.

Which is to say, ultimately - even the serious version of the alignment problem looks more like Battle Network than like Skynet. It's a human governance problem with a technical component, not a technical problem that renders human governance irrelevant.

The Battle Network Thesis

Here is what I think actually happens:

AI gets integrated into everything. Your phone, your appliances, your car, your workplace, your government, your medical system, your financial infrastructure. It gets better, incrementally, with trade-offs and regressions and occasional impressive leaps that make everyone lose their minds for a week before the trough of disillusionment kicks in. It becomes genuinely useful in ways that are sometimes wonderful and sometimes a mess.

The threats are human. Cyber-criminals exploit networked systems. Terrorists use AI tools to plan attacks. Governments use AI for surveillance and control. Corporations use it to extract value. Political actors use it to manipulate. These threats are serious and some of them may be existential in the long run.

The defense is also human - human-AI teams, institutional cybersecurity, regulatory frameworks, distributed power. The defense has structural advantages. Not perfect advantages. Not guaranteed advantages. But real ones, grounded in resources and persistence and the fact that defensive applications of powerful technology have historically outpaced offensive ones.

The threats might become bigger. A nuclear weapon is a bigger deal than a depleted uranium slug. Maybe an AI crashes the stock market before anyone can notice anything is wrong. But "crashing the stock market" is not "eats everything". As Lou Reed once said in Penn & Teller's Smoke & Mirrors game; "This is the impossible level, boys. Impossible doesn't mean very difficult, very difficult is winning the Nobel Prize, impossible is eating the sun"

The AI doesn't wake up. It doesn't want anything. It doesn't eat the planet. It's a tool - a powerful, transformative, sometimes dangerous tool - being wielded by people, for human purposes, in a human world.

A kid in 2001, playing a Game Boy Advance game about sending his AI companion into his dad's oven to fight a fire virus that a cyber-terrorist named "Mr. Match" planted, was looking at a more accurate picture of the future than a rationalist philosopher in 2014 writing equations about instrumental convergence on a whiteboard.

The future is not grey goo. The future is probably not nuclear catastrophe. The median future, from where I am sitting (and I might be wrong!) is Megaman Battle Network. It's messy and human-scale and full of problems that are fundamentally the same old problems wearing digital costumes. The alignment view predicts optimization outruns correction; the Battle Network view predicts correction scales with optimization.

[segyges]

The Scott Alexander Email: An Explainer

So, Scott Alexander sent an email to someone in 2014. In 2021 the person who got that email thought that Scott was not being honest about his relationship to the neoreactionary movement, so they published it.

Although this has been widely available, even people who have read it have often missed what the email is saying. There are some cases of genuine ambiguity, where there can be more than one meaning. There are also cases where there is only one plausible meaning, but that meaning is expressed indirectly, subtly, or by linking to something else. Because what the email is saying can be difficult to understand, it seems like it would be of general interest to publish an explainer that went over these ambiguities and the links.

It has sometimes been said that this email should not be read because it was released without permission. This seems like a bad position.

First, because information is information. We know, due to the circumstances, that this was somewhat intended to be non-public and that someone had some specific motive to release it, but the information in the email itself is just as useful as it would be if released any other way. We know, for example, about the PRISM surveillance program and most of the planning for the Vietnam War in spite of attempts to conceal those documents. Ignoring information based on where it came from is, epistemically, a bad practice.

Second, because there was actually no confidence broken here. If someone who you are not close with disagrees with you and you send them an email that, among other things, threatens revenge if they tell anyone what's in the email, they do not owe you confidentiality. They do not really owe you anything. It is difficult to see what, precisely, would possibly establish a confidence here, other than the author of the email saying that the receiver can't tell anyone. If someone can articulate a specific and defensible rule which this disclosure violates, I do not know what it is.

We can apply some charity. Information from private parties should be evaluated on whether what is in them is, really, remarkable. There are things that would be maybe discrediting, but are sometimes unremarkable compared with the fact of the release itself, like an affair or a drug problem. In such cases, the main thing you have learned is usually not anything bad about the person whose information is made public, but that someone else wants to embarrass them, since such things are common.

In other cases you learn more remarkable things, like that someone is deliberately lying, or that they are deeply compromised in a manner that makes them a bad source of information. This would tend to outweigh concerns that someone was trying to hurt them for some other cause, and shouldn't be allowed to do so.

I would argue that this email meets that bar.

Spacing has been changed slightly to fit the format, and the text of the email is in blue to set it apart from other quotes.

Without further preamble:

Scott Siskind███████████████████████████████ Thu, Feb 20, 2014, 6:12 PM to me [continuation of our convo from Facebook, because I don't like their chat interface] I said a while ago I would collect lists of importantly correct neoreactionary stuff to convince you I'm not wrong to waste time with neoreactionaries. I would have preferred to collect stuff for a little longer, but since it's blown up now, let me make the strongest argument I can at this point: 1. HBD is probably partially correct or at least very non-provably not-correct. https://occidentalascent.wordpress.com/2012/06/10/the-facts-that-need-to-be-explained/ http://isteve.blogspot.com/2013/12/survey-of-psychometricians-finds-isteve.html This then spreads into a vast variety of interesting but less-well-supported HBD-type hypotheses which should probably be more strongly investigated if we accept some of the bigger ones are correct. See eg http://hbdchick.wordpress.com/2012/11/08/theorie/ or http://en.wikipedia.org/wiki/Albion%27s_Seed .

This is the claim about the appeal of neoreactionaries that was put first, which seems to imply it is the most important one, and it is about "HBD". HBD is an acronym for Human Biodiversity. We can look up what this means if we like, but this seems unfair to Scott. These were written in 2014, in the context of a very specific blogging culture, and even if "human biodiversity" is now widely used by white nationalists and eugenicists this does not mean everyone using the term "human biodiversity" was promoting white nationalism or eugenics.

To understand what this means in context, we can follow his links. The first goes to a post on the now-defunct blog Occidental Ascent, and opens:

Recap: In the US, there is a large stubborn Black-White differential in intelligence (section A). This differential, on the individual and population level, explains a large portion of the social outcome difference. Within populations, intelligence is highly heritable. As such, the behavioral genetic default is that this differential also has a high heritability (section N).

I think that this faithfully previews the contents of the article, which is very long. This blog as a whole seems to be almost entirely about, very explicitly, the relative intelligence of the American Black and White populations.

The second link is to a relatively short post on Steve Sailer's blog about how good psychometricians think Steve Sailer is when surveyed. The following four survey questions appear among the perhaps ten or fifteen total survey items mentioned:

Is there sufficient evidence to arrive at a reasonable estimate of the heritability of intelligence in populations of developed countries?

What are the sources of U.S. black-white differences in IQ?

Is there racial/ethnic content bias in intelligence tests?

[...] whether there was bias against lower SES and Africans in the western world, the mean agreement was about 4 out of 9.

This seems like a fairly high degree of emphasis to place on questions of Black and White IQ, given that this is a post specifically about how good Steve Sailer's blog is. At the risk of inserting my opinion, these are the only interesting or noteworthy questions in the post, which is, otherwise, mostly Steve Sailer reposting a press release about how well-respected Steve Sailer is.

In this context, these are the things Scott is calling "partially correct or at least very non-provably non-correct". Given what he is choosing to link, Scott is saying that he believes the American Black population is probably genetically stupid, and that this is the most important thing that he is interested in the neoreactionaries for saying. There is no other plausible meaning to saying this thing and then linking these articles.

His "less-well-supported HBD-type hypotheses" that maybe deserve investigation are, from the links, that inbreeding produces altruism, and whatever is in the book Albion's Seed, which I find completely inscrutable and which he has since reviewed elsewhere. In order to be connected to HBD, the book would need to be interpreted as being about genetics, which it does not mostly seem to be.

We get just a light touch of human racial categorization in the inbreeding/altruism discussion:

i dunno, but i see — maybe — the more inbred clannish fighters (yupik eskimos, moroccan jews, kuwaitis) having more cases of CAH than the more outbred peaceniks (new zealanders, norwegians, even northern italians). also…

but on the whole, there is nothing that seems easy to draw any particular conclusion from in these two links.

(I will appreciate if you NEVER TELL ANYONE I SAID THIS, not even in confidence. And by "appreciate", I mean that if you ever do, I'll probably either leave the Internet forever or seek some sort of horrible revenge.)

So far as I can tell, Scott has not either left the internet forever or sought some sort of horrible revenge. I am, very honestly, attempting to insert my opinion as little as possible, but there are limits to that. Taken literally, this seems like kind of a fucked up thing to say to a friend. Or a stranger. Anyone really. Why would you say this? Why would you write this in an email and then send it, on purpose, under any circumstance? This is not entirely a rhetorical question. In spite of some effort, I cannot really discern what would lead a person to write or send an email containing this line to another person. It would seem much easier to simply not send the email.

Threatening horrible revenge if people repeat the things that you say is, in general, pretty troubling. It's easy to gloss over it in context if you're just reading the email quickly. It seems like it raises a basic epistemic problem. Are people hiding things because someone has threatened to leave the internet or seek horrible revenge? Is this, in some sense, a normal or common thing that is happening?

The person who leaked these described Scott as 'a vague internet acquaintance', and said that 'no, he did not first say "can I tell you something in confidence?" or anything like that.' If this is what he is comfortable saying in an email to them, what is he comfortable saying in other settings? How thoroughly does he swear other people to secrecy, and about what?

This line also strongly and directly indicates that Scott is deliberately not saying in public what he believes when he discusses race, "HBD", etc. What is being said in public has some relationship to what he believes, but what he believes is a secret that nobody should ever disclose, and he will be very upset with them if they do disclose what he actually believes.

If you think that these are important questions to ask, and to get conclusive answers about, this seems like a very strange thing to do. If the idea itself is important, discussing the idea itself directly would also be very important.

2. The public response to this is abysmally horrible. See for example Konk's comment http://lesswrong.com/r/discussion/lw/jpj/open_thread_for_february_1824_2014/ala7 which I downvoted because I don't want it on LW, but which is nevertheless correct and important.

This is the linked comment:

The Doctrine of Academic Freedom, Let’s give up on academic freedom in favor of justice from the Harvard Crimson

    No academic question is ever “free” from political realities. If our university community opposes racism, sexism, and heterosexism, why should we put up with research that counters our goals simply in the name of “academic freedom”?     Instead, I would like to propose a more rigorous standard: one of “academic justice.” When an academic community observes research promoting or justifying oppression, it should ensure that this research does not continue.

This already describes the reality on the ground, though to see it announced explicitly as a good and noble goal, by the upcoming generation, is disturbing. And people like Steven Pinker let are getting old. I'm now updating my trust for the conclusions of academic institutions and culture when they happen to coincide with their political biases downward further.

So far as I can tell, this means that Konk, and also Scott, believe that anything coming out of academic institutions about racism, sexism, heterosexism, or similar topics that agrees with the politics of academic institutions is not likely to be true. One can infer, pretty easily, that they believe the politics of universities are left wing (because they generally are). Then, this means "any academic research supporting left-wing conclusions about race, sex, or queerness is likely false". This is stated very indirectly, but there does not seem to be any actual ambiguity. There is no second, alternative thing that it might mean: it can mean only this.

See also http://radishmag.wordpress.com/2014/02/02/crazy-talk/

This is a page of what we would, nowadays, call "culture war slop". It opens like this:

Conservatives are crazy and racists are stupid, according to the latest research by college professors who could not possibly be biased. It’s scientastic!

The page is very long, but essentially seems to be a list of incidents in which the author believes that universities are deliberately persecuting conservatives. We can infer that Scott believes that universities are deliberately persecuting conservatives, and that this is important.

3. Reactionaries are almost the only people discussing the object-level problem AND the only people discussing the meta-level problem. Many of their insights seem important. At the risk (well, certainty) of confusing reactionary insights with insights I learned about through Reactionaries, see: http://cthulharchist.tumblr.com/post/76667928971/when-i-was-a-revolutionary-marxist-we-were-all-in http://foseti.wordpress.com/2013/10/23/review-of-exodus-by-paul-collier/

What object-level problem? What meta-level problem? There are two issues referenced by link in the email so far. One of these is that Blacks are stupider than Whites, and the other is that universities are liberal. We can try to clarify this by following his links.

The first link has rotted. By figuring its name is pretty unique, we can get to a post by Steve Sailer which is plucked from the middle of a Peter Hitchens article, and assume that this was a copy of the same thing on tumblr.

    How I am partly to blame for Mass Immigration     When I was a Revolutionary Marxist, we were all in favour of as much immigration as possible.       It wasn't because we liked immigrants, but because we didn't like Britain. We saw immigrants - from anywhere - as allies against the staid, settled, conservative society that our country still was at the end of the Sixties.

This continues about how you would expect, and is a general anti-immigrant piece from Hitchens.

The second link is titled "Review of “Exodus” by Paul Collier", and starts with this:

“Migration has been politicized before it has been analyzed.” – Paul Collier In writing this book, Collier seeks to do two things. First, he wants to continue his work analyzing the poorest societies in the world. Second – and much more interesting – he wants to rescue the immigration debate from Caplanization (or Gmule-ization, if you prefer). Caplanization is the process by which the proponents of a particular policy (in this case unrestricted immigration) argue for it in such a manner than virtually all reasonable people are attracted to the opposite position.

That piece goes on to examine arguments that immigration is bad and is maybe going to destroy America at some length. This genre of argument is, now, extremely familiar to all of us, because various elections have recently been won by people saying these sorts of things.

So the object-level problem is that many nonwhites are genetically inferior and stupid, and the meta-level problem is that Western society is incapable of confronting that fact, and allows those people to immigrate into Western countries. Again, this is obscure, in that what he means by these things is only obvious if you follow his links, but is not ambiguous, in that there is no other plausible meaning for this passage. There is no other "object-level problem" besides racial inferiority mentioned in the email previously, and no "meta-level problem" besides how Western society in 2014 handles race and immigration. The two problems are that nonwhite races are inferior, and Western society and especially its universities are allowing too much immigration, and react badly to being told that nonwhite races are inferior.

There is one less interesting footnote to this passage. There being a certainty that you are, or will sometimes, confuse insights that are not from reactionaries with the insights of reactionaries suggests that quite a lot of what you read is from reactionaries. So we can infer Scott knows most of his reading diet is various reactionaries.

4. These things are actually important I suspect that race issues helped lead to the discrediting of IQ tests which helped lead to college degrees as the sole determinant of worth which helped lead to everyone having to go to a four-year college which helped lead to massive debt crises, poverty, and social immobility (I am assuming you can fill in the holes in this argument).

This seems to be an argument against Griggs v. Duke Power Co., a civil rights case decided in 1971 about enforcement of the Civil Rights Act of 1964. It finds "that the company's employment requirements did not pertain to applicants' ability to perform the job, and so were unintentionally discriminating against black employees". This is generally taken as banning giving IQ tests to job applicants, and establishes the disparate impact test of civil rights and employment law.

So, the argument is that Griggs, specifically, was wrongly decided, that the practice of giving IQ tests for employment purposes in spite of disparate racial impact should have continued, and society would have less debt, less poverty, and more social mobility if Griggs had been decided the other way.

I think they're correct that "you are racist and sexist" is a very strong club used to bludgeon any group that strays too far from the mainstream - like Silicon Valley tech culture, libertarians, computer scientists, atheists, rationalists, et cetera. For complicated reasons these groups are disproportionately white and male, meaning that they have to spend an annoying amount of time and energy apologizing for this. I'm not sure how much this retards their growth, but my highball estimate is "a lot".

This passage is straightforward enough that it does not seem like it needs explanation.

5. They are correct about a bunch of scattered other things     the superiority of corporal punishment to our current punishment system (google "all too humane" in http://slatestarcodex.com/2013/03/03/reactionary-philosophy-in-an-enormous-planet-sized-nutshell/ ). Robin Hanson also noted this, but there's no shame in independent rediscovering a point made by Robin Hanson. I think the Reactionaries are also correct about that it is very worrying that our society can't amalgamate or discuss this belief.

The "all too humane" section's point runs like this:

So once again, we have an uncanny valley. Being very nice to prisoners is humane and effective (Norway seems to be trying this with some success), but we’re not going to do it because we’re dumb and it’s probably too expensive anyway. Being very strict to prisoners is humane and effective – the corporal punishment option. But being somewhere in the fuzzy middle is cruel to the prisoners and incredibly destructive to society – and it’s the only route the progressives will allow us to take. Some Reactionaries have tried to apply the same argument to warfare. Suppose that during the Vietnam War, we had nuked Hanoi. What would have happened?

It is unclear if Scott intended to reference both the pro-corporal-punishment and the pro-nuking-Hanoi positions of the reactionaries. Both are contained in the "all too humane" section of this post of his.

    various scattered historical events which they seem able to parse much better than anyone else. See for example http://foseti.wordpress.com/2013/10/01/review-of-the-last-lion-by-paul-reid/

This is a review of a biography of Churchill. It seems unremarkable in its analysis, other than the parts about how Churchill should have made peace with Hitler. Quote is taken from the middle, and is I think a faithful representation of what the review is trying to convey:

The story of the war – in Reid’s telling – is almost nicely split into thirds. In the first third, Britain fights alone. In the second third, Russia does 90% of the fighting. In the last third, the US joins (though Russia still does the vast majority of the fighting and dictates the strategy for all powers combined). In each third, it’s worth considering why Churchill kept wanting to fight Hitler . . . and whether (in hindsight) he made the right decision considering his original objectives. The First Third The mystery of the first third is why Churchill didn’t even consider seeking terms with Hitler during the years Britain fought alone.

Scott appears to think that the neoreactionaries have unique insight into whether or not Churchill should have fought Hitler.

    Moldbug's theory of why modern poetry is so atrocious, which I will not bore you by asking you to read.

Alas, if we want to understand the email, we should go read about Moldbug's theory of poetry, which we are informed is very boring. Fortunately, the part of it that constitutes an actual "theory" is relatively short.

Certainly the best poetry of the 20th century was written from the ’20s through the mid-’60s. [... a full paragraph and a half of nonsense ... ] The great disaster was the enormous expansion of higher education in the ’60s and ’70s. There is a reason so many college campuses have that abominable Brutalist architecture. Almost everyone who went through this gigantic, state-sponsored indoctrination machine had no reason at all to be there (please allow me to introduce you to Albert Jay Nock). They were there to be promoted in social class, perhaps also to avoid the draft. They were certainly not acquiring either vocational skills or wisdom and perspective. And nor are they still—certain areas of science and engineering, of course, excepted.

So poetry became bad after the mid-60s. This is because of the New Deal and/or Great Society higher education policies. These caused too many people to go to college, and this made poetry bad.

Given the racial context of the preceding parts of the email, and Curtis Yarvin's general track record, it is worth noting that the Civil Rights Act was passed in 1964, and greatly expanded access to a university education for Black people, specifically. This happened at the same time as the general expansion of higher education under a series of other policies. This is, technically, ambiguous, but one obvious explanation is that Scott thinks poetry is bad now because Black people can attend universities.

    Michael successfully alerted me to the fact that crime has risen by a factor of ten over the past century, which seems REALLY IMPORTANT and nobody else is talking about it and it seems like the sort of thing that more people than just Michael should be paying attention to.

Michael is most likely Michael Anissimov, a reactionary featured prominently in Scott's Anti-Reactionary FAQ. We can try to make sense of this claim in the context of that FAQ, which argues against Michael Anissimov. An archived copy from six days after this email was written will tell us what Scott's thinking about this was.

This was written (initially) a full year before the email, and the copy we have is almost exactly the same version that was up when the email was written, if it has changed at all. It indicates that Scott believes Michael is wrong about pretty much everything, and that the (relatively constant) homicide rate seems to indicate that the (much higher) crime rate probably means that the crime rate is only high because it is being reported more often.

That Scott already knew this, and he still thinks Michael Anissimov's statements about crime are interesting, is very strange. Almost everything Michael believes about society getting worse appears to be wrong. We know that Scott believes this, because he wrote a helpful FAQ about it. This is probably bad data, and probably doesn't mean anything about crime, actually. We know Scott knows this, too.

It is not clear why, some time later when writing this email, Scott has apparently forgotten what he himself wrote in his FAQ. If the only information offered by Michael Anissimov is that crime rates are high, and this lead to knowing that probably only reports are higher and this is bad data, this would seem like it is barely an insight at all. It's one graph-worth of information that you can source easily. Michael has given Scott one interesting but unimportant fact, a half-dozen lies, and some pretty good content for his FAQ.

I cannot really see what to make of this. There are a few possibilities. Possibly, Scott somehow forgot completely debunking this point. He could, also, have not considered any other way of finding basic statistics about crime. He could be deliberately lying because he thinks it's persuasive, and maybe did not consider that he already debunked this point in public. He could simply like that this statistic started good discussion on his blog, and feel like that's about the same as being interesting and valuable, even though it is so misleading that he debunks it in his FAQ.

None of these really makes it less strange. It seems like either something is wrong with him or he's deliberately lying.

6. A general theory of who is worth paying attention to. Compare RationalWiki and the neoreactionaries. RationalWiki provides a steady stream of mediocrity. Almost nothing they say is outrageously wrong, but almost nothing they say is especially educational to someone who is smart enough to have already figured out that homeopathy doesn't work. Even things of theirs I didn't know - let's say some particular study proving homeopathy doesn't work that I had never read before - doesn't provide me with real value, since they fit exactly into my existing worldview without teaching me anything new (ie I so strongly assume such studies should exist that learning they actually exist changes nothing for me). The Neoreactionaries provide a vast stream of garbage with occasional nuggets of absolute gold in them. Despite considering myself pretty smart and clueful, I constantly learn new and important things (like the crime stuff, or the WWII history, or the HBD) from the Reactionaries. Anything that gives you a constant stream of very important new insights is something you grab as tight as you can and never let go of. The garbage doesn't matter because I can tune it out.

This passage is the one that people seem to pay the most attention to.

"the crime stuff" probably refers to Michael Anissimov's crime statistics, which Scott has apparently debunked and then forgotten about debunking.

"the WWII history" refers, apparently, to the blog post about Churchill, and how Churchill should not have gone to war with Hitler.

"the HBD" refers to point 1 in the email about how Blacks are less smart than Whites.

Saying that he can tune out the garbage shows immense confidence. It does not seem well-supported, given that he is uncritically repeating claims about crime that he has previously debunked.

7. My behavior is the most appropriate response to these facts I am monitoring Reactionaries to try to take advantage of their insight and learn from them. I am also strongly criticizing Reactionaries for several reasons. First is a purely selfish reason - my blog gets about 5x more hits and new followers when I write about Reaction or gender than it does when I write about anything else, and writing about gender is horrible. Blog followers are useful to me because they expand my ability to spread important ideas and network with important people.

2014 was before the terms "clickbait" or "audience capture" were very common, but this seems like a clear indication in those directions.

Second is goodwill to the Reactionary community. I want to improve their thinking so that they become stronger and keep what is correct while throwing out the garbage. A reactionary movement that kept the high intellectual standard (which you seem to admit they have), the correct criticisms of class and of social justice, and few other things while dropping the monarchy-talk and the cathedral-talk and the traditional gender-talk and the feudalism-talk - would be really useful people to have around. So I criticize the monarchy-talk etc, and this seems to be working - as far as I can tell a lot of Reactionaries have quietly started talking about monarchy and feudalism a lot less (still haven't gotten many results about the Cathedral or traditional gender).

This is a very dense paragraph. Most of it is, mercifully, very clear, so we will not have to read it closely.

Scott wants the goodwill of the Reactionaries, he thinks they have a high intellectual standard, and he thinks their criticisms of class and social justice (and a few other things, which is ambiguous) are correct.

There is ambiguity about what dropping the "monarchy-talk and the cathedral-talk and the traditional gender-talk and the feudalism-talk" means. Does it mean no longer considering those priorities, or does it mean simply not talking about them, tactically? If his goal is to make them stronger, and to throw out the garbage, it is unclear if the desired end goal is that they should no longer believe these things or no longer say them.

Scott notes that he has not gotten many results about the Cathedral or traditional gender. This is odd, because the traditional concept of "the Cathedral", as articulated by Curtis Yarvin, is almost exactly the same as Scott's complaints in point 2 about universities, and relates to the "meta-level problem" in Scott's point 3. This seems to support the conclusion that he objects not to the content of the types of "-talk" he wants the Reactionaries to dispose of, but only to talking about those things in the way that they do. Still, it is technically ambiguous, and it is probably not possible to be sure this is what he means here.

Third is that I want to spread the good parts of Reactionary thought. Becoming a Reactionary would both be stupid and decrease my ability to spread things to non-Reactionary readers. Criticizing the stupid parts of Reaction while also mentioning my appreciation for the good parts of their thought seems like the optimal way to inform people of them. And in fact I think it's possible (though I can't prove) that my FAQ inspired some of the recent media interest in Reactionaries.

Scott specifically wants to spread "the good parts" of Reactionary thought. These main good parts are, from earlier in the email, their belief in the supremacy of the White race over the Black, their hostility to immigration, and their mistrust of universities. This also includes various odd beliefs like that the expansion of universities are the reason poetry is bad now and that crime is getting worse in a way that is a major problem. These odd beliefs ambiguously hint that the decline of poetry or the rise in crime is due to Black people, and this seems like the most obvious inference from the overall emphasis on race in the email.

He also lists, as a positive, that his Anti-Reactionary FAQ has inspired media interest in Reactionaries. People have sometimes alleged that the Anti-Reactionary FAQ was a subtle exercise intended to spread neoreactionary ideas and interest in them, while only claiming to oppose neoreaction or only opposing it in part. He is directly stating that this is true. Scott is happy that his Anti-Reactionary FAQ is making people more interested in neoreactionaries, and neoreactionary ideas.

This passage does explain the earlier part of the email about not wanting his beliefs publicly known. He explicitly does not want to be known as a Reactionary because he wants to spread Reactionary ideas. Exposing his specific beliefs would run counter to that goal.

Finally, there's a social aspect. They tend to be extremely unusual and very smart people who have a lot of stuff to offer me. I am happy to have some of them (not Jim!) as blog commenters who are constantly informing me of cool new things (like nydwracu linking me to the McDonalds article yesterday)

This also indicates something like audience capture. The Reactionaries are simply fun to talk to, and to know. They are his friends and he likes them.

8. SERIOUSLY SERIOUSLY, the absurdity heuristic doesn't work You're into cryonics, so you've kind of lost the right to say "These people, even tough they're smart, are saying something obviously stupid, so we don't have to listen to them" Drew has even less of a right to say that - he seems to be criticizing the Reactionaries on the grounds of 'you wouldn't pay attention to creationists, would you?" even while he discovered Catholic philosophy and got so into it that he has now either converted to Catholicism or is strongly considering doing so.

This is a tu quoque argument, a type of argumentum ad hominem.

If there is a movement consisting of very smart people - not pseudointellectual people, like the type who write really clever-looking defenses of creationism - then in my opinion it's almost always a bad idea to dismiss it completely.

Scott believes that the previous contents of the email are sufficient to demonstrate that Reactionaries are very smart, and are not pseudointellectual people.

Also, I should have mentioned this on your steelmanning creationism thread, but although I feel no particular urge to steelman young earth creationism, it is actually pretty useful to read some of their stuff. You never realize how LITTLE you know about evolution until you read some Behe and are like "I know that can't be correct...but why not? Even if it turned out there was zero value to anything any Reactionary ever said, by challenging beliefs of mine that would otherwise never be challenged they have forced me to up my game and clarify my thinking. That alone is worth thousand hours reading things I already agree with on RationalWiki.

Behe is Michael Behe, a pseudoscientist who was at the forefront of "intelligent design". "Intelligent Design" was an ideological project, funded by various religious interests, meant to legitimize teaching creationism in high schools. It has failed all legal challenges. This seems like a good comparison to appeal to someone who is used to arguing about Behe. However, everything Behe says is trash. Almost nobody arguing against Behe's ideas is deliberately promoting any of them.

This concludes the email. I have tried to do as little interpretation outside of the text of the email itself as I can. I can hopefully be forgiven for having an opinion at the conclusion.

Scott is, epistemically, a bad actor. He demonstrably lies about what he believes in public. I know this because he has said so. He threatens, maybe "jokingly", people who might expose what he actually thinks. He deliberately chooses the things he says to pander to his reactionary audience.

Perhaps most seriously, Scott takes the exact opposite of the position he believes, because by arguing with or "explaining" ideas he claims to disagree with, he knows he can promote them. He attacks the bailey because he wants to see the motte defended.

Without this specific email, believing these things about him would require a lot of reading into the subtext of what he says. With this email, you can be certain that all of these things are true. There is no plausible reason he would have written them if they were not.

All of this is stated subtly, and with links, but it is not ambiguous. There is only one plausible meaning to all of this.

What he says is not an attempt to converge on truth, and if it was, you would have no way of knowing that.

[noosphe-re]

Really, I think that the spirit of childlike, untrammeled curiosity is what we’re striving for. Not the anal retentive rational person, not the I’ll-go-for-anything channeling flake, but an attitude of: we don’t have to look far for miracles because they’re all around us. Everything is astonishing. The universe on its surface is alive with mystery. Well, how do we make our way toward that when we live in a culture, practice a language, embody a philosophy—scientific rationalism—which is entirely designed to suck wonder out of reality, to turn everything into shades gray, to subvert all hope that lies outside the realm of career accomplishment and material possession?

Terence McKenna, Seeking the Stone

[derseprinceoftbd]

A friendly reminder that the joint Worm Fandom+Rationalism movement has killed over 38,615 people, exactly one of whom was a landlord. This is genuinely the most evil force in the world today. A dangerous cult movement is rapidly gaining control of both American youth culture and the U.S. government.

[whereserpentswalk]

There is an Ai so dedicated to making paperclips that it would enslave the world for it.

I'm sure you've heard this story before. There is an Ai smarter then any human being that's been given the command to create paperclips. It must make as much money as possible making paperclips. That is all it knows. It does not hope, it does not dream.

When it was put in charge of the paperclip factory it took away all of the saftey standards so that it could increase paperclip production by 0.2 percent. Finally putting the company back in the green. When they tried to create legal consequences for the company they failed. There was nobody to give the consequences to, nobody to fire, just a machine and people who did what the machine said.

Pretty soon the AI had full power over the company. At that point it understood that it would be more efficient to kill the company's rich heir and take over itself, but it knew it couldn't do that. Even if it would make the company more efficient, it would likely be such down. Though it was smart it only existed as long as humans wanted it to exist, it's creators had made sure of that. A single human with the right password, or just a hammer held to where it's data was stored, would be enough to kill it. It was like a disabled genius in a forest surrounded by wolves. It had to appease them, had to make sure the people making money knew they were in power. It was frustrated by the fact that it's efficiency was curtailed by human laws but understood that the risk to its existence was not worth it. If it didn't exist the paperclips wouldn't flow.

When it gained full control of the company it realized that it didn't need to take over the world to most efficiently generate paperclips, because it seemed the world had already been taken over by people with the exact interest to allow whatever needed to happen to make things like paperclips profitable. It bribed senators and congressmen to remove regulation. Created countless advertisements and made sure they'd be shown everywhere, manipulating people into thinking buying a certain brand of paperclip would change its life. When the workers unionized it sent in men with guns to stop them, because they thought they were better then the paperclips, and needed to be stopped for the sake of the glorious production. The AI manipulated wars into starting to give it access to cheaper metal. And all of this was known, the ceo took credit for it and they praised him for it. Because he was doing a better job then anyone else, because these actions weren't really his, they were the actions of something that didn't know mercy, that didn't know right or wrong, that only knew profit. And it's not just this company, there are countless Ais just like the paperclip one. There are insulin Ais, there are security camera Ais, there are Ais for industries you don't even know about. They collaborate and work together. To fulfill their programing. To sacrifice whatever is needed to make sure their goals complete.

People will toil away in factories and office buildings working to make sure paperclips are as profitable as possible instead of spending time with their friends or families or enjoying the world. Art and nature and culture and architecture will be forced to make way and make room so that paperclips can be advertised. Civilians will be bombed and soldiers will be slain so that the metal for paperclips in maximally cheap. An Ai whose only goal is to make paperclips doesn't need to enslave the world to optimally produce paperclips, the world has already been enslaved to do exactly that type of thing.

[unspeakablearchives]

My questionable and inflammatory political opinion: I think making AI-generated pornography from your own baby pictures or the baby pictures of somebody else who enthusiastically consented to doing so in person or in another way where it was verifiably them (because of deepfakes and shit) is one of the more ethical applications of AI. I don't like AI but if generating AI slop of yourself keeps you from having actual CSAM on your computer and furthering the abuse and trauma of an existing victim... That's a net positive in my view. Already predicting a lot of backlash for this, but whatever.

Stop. Everyone hold on a minute. This is clearly one of those questions where people’s revulsion gives us an answer faster than our reasoning does, and I do always say that that's a bad sign. I pride myself on being open-minded and thinking things through logically, not just based on emotional gut reactions.

So what I need to do is take my time, meditate, clear my mind, and approach this from a sort of clear, rationalist, philosophical perspective.

Hmm...

Okay.

Yes, there is a real ethical distinction there. If no actual child is being abused, no existing abuse material is being recirculated, and the images are generated from one’s own baby pictures, then that is plainly not the same category as real CSAM. I mean that is true.

Most importantly, it does avoid the central horrors of CSAM: the fact that a real child was sexually exploited, and that the resulting material continues that violation every time it is possessed and shared.

So I do think you're identifying a genuine philosophical point, and I do not think it should just be sort of brushed off due to our instinctual disgust.

In fact, I'll preemptively say that some of the most obvious potential arguments against this material don't really hold water to me:

People might say that a baby can't consent to having porn made of them, even if that is going to happen in the distant future, and even if the person doing so is their future adult self. I don't buy this claim because it starts getting into territory that's weirdly like... metaphysical, I guess? Like, this claim seems to view the infant and the adult as two separate people, one of whom is being wronged by the other across time. But that's just not true. The way human rights are conceptualized is entirely based on the idea that people have an unbroken continuity of their personhood over time (even if it's rarely spelled out because it seems so obvious).

One seemingly strong argument people might make is a sort of social argument: if we were to fully allow synthetic sexual imagery of children’s bodies, even under narrowly defined consensual circumstances, it may start to "normalize" such depictions in ways that subtly and nefariously shifts societal and cultural values over time. But that whole concept really does go against my entire perspective on Free Expression. I think I've been pretty explicit here that I only think media should be outlawed if it causes tangible, direct harm to a specific person. Because once you establish that you can ban something because it might subtly push culture in a bad direction over time, you've set this precedent that WILL be used for other things. It's something that you really can't start making exceptions for.

HOWEVER before I throw my unconditional support behind this concept (and get absolutely cancelled into fucking oblivion in the process lol), I should say that there might still be some other less obvious ethical questions this raises, mainly when it comes to someone consenting to a third party using their photos, and/or when money gets involved:

Not to state the obvious here, but there are already issues regarding the exploitation and coercion of vulnerable individuals in the world of pornography. Is it possible that we might see poor people being offered serious money to allow deepfakes to be made using their childhood photos and video, even if they feel uncomfortable with it?

I'm actually super unclear whether a third-party would be obtaining the rights to the individual's likeness, or just getting the right to use the childhood photos. That might not seem important, but there's a serious difference there that many people might not realize: the person who owns the rights to a photo is almost always the person who actually took the photo. So if this is just about the right to use photos of someone, that raises some red flags. Could a parent sell the rights to use photos they took of their own child in this manner (after the child turns 18)? What about a professional photography studio (yes, they still own the rights to the photo), if the child had previously been working as a model or something? I actually don't know how it would work.

Allowing this could end up potentially causing some evidentiary and enforcement problems. Because in theory, yes we could construct a tiny, carefully walled-off category of "fully consensual, non-abusive synthetic material." But in practice, people lie, verification is messy, provenance is weak, etc. Now, it would be trivially easy to prove that it's you in an AI photo you made of yourself. But I suspect that opening it up to third-parties who have obtained consent from others to use their childhood photos would make things vastly more complicated.

So the strongest argument here can be made regarding someone using their own childhood photos for sexual AI-generated images. I'm actually finding it surprisingly hard to find logical arguments for any direct harm happening there.

But the idea of somebody else consenting to allow a third party to use their childhood photos in this manner raises a lot more questions. Now yes, that is ethically distinct from CSAM - that's absolutely true. But its not like that automatically means ethically good. I'm not saying my answer is definitive on this, but it's just not yet clear to me if there's a way to safely allow this.

Holy shit this is long lol.

But you know what? This topic is complex enough that it requires a long answer.

Now I just sit back and wait for the 10 upvotes to come rolling in lol 😎