A miserable little pile of opinions.

[official-kircheis]

if terry tao thinks GPT-5 is useful for maths and linus torvalds thinks Claude is useful for coding. you look pretty embarrassing claiming there’s no use. and future ai will only be better.

[escuerzoresucitado]

[profound-yet-trivial]

[femmenietzsche]

I was planning to make a post to the effect of:

The vampires in Blindsight are a bit of a contrivance tbh. The aliens and the transhuman brain mods are all plausible-feeling - you can imagine aliens with a distributed non-conscious intelligence, and you can imagine how these very drastic modifications would shift people away from human baseline in a big way - but the specific tradeoffs of the vampires - hyperintelligence at the cost of amoral carnivory - come off as arbitrary. These very very advanced humans couldn't salvage the good parts and ditch the bad, really? It's a way to add drama in a thematically relevant way that doesn't hold together as well as the rest of it.

but after further consideration I'm not sure how true that is. The transhuman characters (all of them) are strange alien to us, but in a way that makes emotional sense. Oh, of course the guy whose consciousness is partly distributed across a bunch of machinery would be alienated from his body. Oh, of course the guy who lost half his brain and had it replaced by whatever would come off like he's simulating being human. These tradeoffs *feel* right to me, in a way that the vampirism doesn't, but the "why doesn't this future society just engineer people better" objection that I raised to the vampires applies just as much here. The tradeoffs are just as arbitrary on a technical level, since it's fiction. It doesn't *feel* like you get to have characters who are both superhuman and normal, but that's not backed by anything. That said, for fiction intuitive plausibility still matters very much, so I'm only half withdrawing my complaint. It's still a little goofy, especially the right angles thing

[femmenietzsche]

deadgenerations

the vampires were actually the original idea for the book. he wanted to make SciFi vampires for his first contact novel and so everything about them is, well, "how do I do SciFi vampires". it's why it feels less natural. the other stuff in the novel, namely everything about the aliens, arose organically as he wrote. basically everything about the aliens came as he wrote, often times looking for extant biological mechanisms for things he wanted in his book, but most significantly the thesis for the book - consciousness is useless cognitive baggage - wasn't what he went into the book with. it was actually supposed to be the opposite - about the evolutionary or cognitive utility of consciousness. as he wrote, he discovered that there wasn't any research which demonstrated a use for consciousness despite looking quite hard - and so that became the conclusion of the novel.

This makes sense to me. That's a pretty funny reversal though, given how bone deep the cynicism is in this book

[petluck]

insane news, today a man in my city stole a bus and just. kept making all the stops.

[petluck]

the Assman rides again

[petluck]

an upstanding citizen of our great city

[umpireofpopculture]

[rustingbridges]

I do find it a bit tragic that this is a whole type of guy. like that guy who kept getting arrested for sneaking into and driving nyc trains and buses normally

[profound-yet-trivial]

Don't kinkshame

[intimate-mirror]

discourse chain/thread where at some point you start saying unreasonable stuff so you can tell which of your followers has good taste and which just agree with you by which like the correct posts but not the incorrect ones

[profound-yet-trivial]

what do you think of followers who approve of you going shallowly meta in replies?

[femmenietzsche]

Pick an ethical system

Fanaticism

Nihilism

Crippling doubt

Telling people what they want to hear

Voting ended onJul 13

[femmenietzsche]

transgenderer

Crazy fucked up fanaticism isn't winning....no wonder everyone on tumblr wants to kill themselves...

Just themselves instead of millions of others 🙂‍↕️🙂‍↕️🙂‍↕️

[profound-yet-trivial]

The thing about fanaticism is that it at least has a chance of being a useful reaction to reality

[catasters]

[chameleonsynthesis]

xkcd 325

[gridmaster5000-deactivated20241]

you disagreed w me in an intellectually honest way. we're friends now.

[librarycards]

just got kicked out of the omelas DEI office for asking why there are no former torture kids on the board

[librarycards]

they told me they had a "lived experience" subcommittee but it was just the torture kids' parents. tf.

[ms-demeanor]

Maybe it's because I'm more of an infosec person than a cybersec person but everything I hear about Mythos and AI-based security threats makes me feel like an alien.

[profound-yet-trivial]

Spiky intelligence is counterintuitive. At this point, the top LLMs aren't reskinned text predictors—that's only their larval stage. More and more lifting is being done by massive amounts of reinforcement learning, and for some tasks that works much better than others.

So Mythos/Fable can both give wrong answers on everyday questions (though far less often than its predecessors) and be superhuman at finding full zero-day exploits, because RL is far more reliable on coding prowess than on human interaction.

(But also, the paid-only models have become drastically better at everyday questions than anything that preceded them; anyone who's only using the free models will have deeply incorrect intuitions about what's already possible.)

[intimate-mirror]

This is also my take on @bambamramfan's quiz.

[profound-yet-trivial]

When the positive-negative axis lumps together existential risk with the fake water usage crisis, the quiz is worthless.

[penny-anna]

Do Eridians know they are different colours. I bet Grace's alien kids love finding out what colour they are. It means absolutely nothing to them but they're like :O :O

[penny-anna]

Some of them think he's making this whole 'colour' thing up to mess with them and try to catch him out by asking again on a different day to prove he's just saying random noises but he's like you are still blue buddy and they're like :O :O

[penny-anna]

Like if we met an alien species who had extra senses & they said that some humans felt spingly and some humans felt spoingly I bet we'd all want to know if we were spingly or spoingly humans

[profound-yet-trivial]

Eridians perfectly understand "these are the wavelengths of electromagnetic radiation that reflect off a body and the wavelengths that are absorbed by it". The thing that would be mysterious to them is the way that human perception integrates several different wavelengths lossily into a single "color", and even generates fake composites like "magenta" that don't correspond to any single wavelength.

[raginrayguns]

i like the MIRI "modal combat" setup where programs prove things about each other to decide whether to cooperate or defect in a prisoner's dilemma

But I don't actually like the definition of a "modal agent" in the paper

consider FairBot. "I cooperate with you if and only if there's a proof you cooperate with me". Put another way: "I cooperate if there's a proof that you cooperate, and I defect if there's no proof that you cooperate".

Why not make it symmetrical? If there's a proof you cooperate, then I cooperate. If there's a proof you defect, then I defect.

This sounds like it should be equivalent, but it's not. First of all, if there's no proof of either we just don't halt I guess.

But... even leaving that case aside, the system we're doing proofs in (let's say Peano arithmetic) can't rule out that both proofs exist. This gets us into a conflict with the definition of a modal agent. Like, what makes a program a FairBot is that there's a proof in Peano arithmetic that it cooperates if and only if there's a proof that the opponent cooperates. More generally, a modal agent cooperates if and only if some condition in provability logic.

This is actually not at all like the kind of thing i really want to model, which is two programs searching for proofs and doing something based on the first proof they find. Now that I think about it, i've never seen this in the MIRI research papers. Like in proof-based UDT they loop over outcomes from best to worst, and check if there's any proof that one of the actions available leads to that outcome.

Maybe there's some good reason for that, maybe it just doesn't work. Like, if PA can't prove I cooperate, how is my opponent supposed to prove I cooperate? But maybe it can be made to work, and it's just that I'm the only person who thinks that agents shouldn't be able to prove what their own actions are when searching through proofs.

[profound-yet-trivial]

Not sure I understand the critique. Do you mean that if your modal agent is A ↔ f(A,B) for a modal formula f, you want the symmetry f(x,y) = f(x,~y) for all x, y? Because I think you'd see the problems with that.

In general, though, I long ago came to the conclusion that modal logic is the wrong instrument for representing decision theory, because it's insufficiently expressive to even capture CDT.

[raginrayguns]

anthropic.com:

The net effect of this order is that we must abruptly disable Fable 5 and Mythos 5 for all our customers to ensure compliance.

...

If this standard was applied across the industry, we believe it would essentially halt all new model deployments for all frontier model providers.

Ha! AI paused!

[profound-yet-trivial]

I forget who I was telling that the SCR designation was not the last trick Trump's DOW would try in order to strongarm Anthropic into removing all safeguards for the USG.

My guess at the time was that they'd threaten to revoke the visas of all foreign national Anthropic employees. This is actually a more crushing blow to Anthropic than that would have been.

Anthropic's response was to disable Fable/Mythos for all customers, including the USG. We'll see how the DOW escalates next.

[intimate-mirror]

Many people accuse rationalism of being a cult about yudkowsky, and the common meme response is that a rationalist is someone who disagrees with yudkowsky. I agree with the essence of this response - it is considered very fine and expected to criticize yudkowsky.

I think a better choice if you are looking to call someone a cult leader is Oliver Habryka.

He is in charge of the physical conference location, and is thereby positioned to (and in practice does) ban people from all major rationalism-associated events for many reasons, including on the basis of personal feuds.

He is also in charge of lesswrong, the blog where a large chunk of rationalism-specific discourse takes place, and is thereby positioned to (and in practice does) ban people from the primary hub of rat discussion. More subtly, even when explicit moderation action is not taken, the way karma works on lesswrong serves to make his particular feelings on a post have a giant effect on its visibility.

Finally, criticism of him tends to be socially punished in a way that criticism of yudkowsky is not. He reacts to criticism with acrimonious insults and threats, and his personal friends jump in to defend him.

I would definitely not call him a cult leader, but he is like 15% of the way there, whereas yudkowsky is like 3% of one. Yud has the name cachet, but Habryka has the actual broad organizational power, and that is what causes bad behavior.

[profound-yet-trivial]

There's a difference between a petty tyrant and a cult leader; your points support an accusation of the former, not the latter. (Go ahead and make that accusation! I'm not keen to defend Oli on that charge.)

People defend Oli fiercely, but I haven't seen anyone treat him as a charismatic guru in person or online, as they did with Geoff Anders or Brent or Vassar. Have you?

"Cult leader" shouldn't be used to mean "guy who's wielding social power harmfully"—it's more specific than that.

[profound-yet-trivial]

The thread blaming electromagnetic background noise for the Fermi paradox shows a staggering lack of imagination about what a technological civilization could do.

We can fully rule out any hypothesis under which hundreds of interstellar spacefaring civilizations would predate us in the Local Group by millions of years, because someone would have been expanding and harnessing Kardashev 1.5+ levels of solar energy as they went along, and we would definitely notice a sphere thousands of light-years in diameter in which a significant portion of the stellar spectra had been replaced with waste heat.

[argumate]

one thing LLMs and people have in common is if you ask them "why did you do that" even their sincere answer will often be complete garbage confabulated after the fact.