#replication crisis

[todays-xkcd]

Maybe encouraging the publication of null results isn't enough--maybe we need a journal devoted to publishing results the study authors find personally annoying.

Replication Crisis [Explained]

Transcript Under the Cut

[Megan, Ponytail, and Cueball are standing at a lectern. Ponytail is talking into the microphone.] Ponytail: In the early 2010s, researchers found that many major scientific results couldn't be reproduced.

[Ponytail turns slightly to look around the room.] Ponytail: Over a decade into the replication crisis, we wanted to see if today's studies have become more robust.

Ponytail: Unfortunately, our replication analysis has found exactly the same problems that those 2010s researchers did.

[This panel shows a newspaper, with title "NEWS" surrounded by flourishes. There is a photo of the three researchers at the lectern taking up about half the front page. The headline reads: "REPLICATION CRISIS SOLVED".]

[mostlysignssomeportents]

Marshmallow Longtermism

The paperback edition of The Lost Cause, my nationally bestselling, hopeful solarpunk novel is out this week!

My latest column for Locus Magazine is "Marshmallow Longtermism"; it's a reflection on how conservatives self-mythologize as the standards-bearers for deferred gratification and making hard trade-offs, but are utterly lacking in these traits when it comes to climate change and inequality:

https://locusmag.com/2024/09/cory-doctorow-marshmallow-longtermism/

Conservatives often root our societal ills in a childish impatience, and cast themselves as wise adults who understand that "you can't get something for nothing." Think here of the memes about lazy kids who would rather spend on avocado toast and fancy third-wave coffee rather than paying off their student loans. In this framing, poverty is a consequence of immaturity. To be a functional adult is to be sober in all things: not only does a grownup limit their intoxicant intake to head off hangovers, they also go to the gym to prevent future health problems, they save their discretionary income to cover a down-payment and student loans.

This isn't asceticism, though: it's a mature decision to delay gratification. Avocado toast is a reward for a life well-lived: once you've paid off your mortgage and put your kid through college, then you can have that oat-milk latte. This is just "sound reasoning": every day you fail to pay off your student loan represents another day of compounding interest. Pay off the loan first, and you'll save many avo toasts' worth of interest and your net toast consumption can go way, way up.

Cleaving the world into the patient (the mature, the adult, the wise) and the impatient (the childish, the foolish, the feckless) does important political work. It transforms every societal ill into a personal failing: the prisoner in the dock who stole to survive can be recast as a deficient whose partying on study-nights led to their failure to achieve the grades needed for a merit scholarship, a first-class degree, and a high-paying job.

Dividing the human race into "the wise" and "the foolish" forms an ethical basis for hierarchy. If some of us are born (or raised) for wisdom, then naturally those people should be in charge. Moreover, putting the innately foolish in charge is a recipe for disaster. The political scientist Corey Robin identifies this as the unifying belief common to every kind of conservativism: that some are born to rule, others are born to be ruled over:

https://pluralistic.net/2020/08/01/set-healthy-boundaries/#healthy-populism

This is why conservatives are so affronted by affirmative action, whose premise is that the absence of minorities in the halls of power stems from systemic bias. For conservatives, the fact that people like themselves are running things is evidence of their own virtue and suitability for rule. In conservative canon, the act of shunting aside members of dominant groups to make space for members of disfavored minorities isn't justice, it's dangerous "virtue signaling" that puts the childish and unfit in positions of authority.

Again, this does important political work. If you are ideologically committed to deregulation, and then a giant, deregulated sea-freighter crashes into a bridge, you can avoid any discussion of re-regulating the industry by insisting that we are living in a corrupted age where the unfit are unjustly elevated to positions of authority. That bridge wasn't killed by deregulation – it's demise is the fault of the DEI hire who captained the ship:

https://www.axios.com/local/salt-lake-city/2024/03/26/baltimore-bridge-dei-utah-lawmaker-phil-lyman-misinformation

The idea of a society made up of the patient and wise and the impatient and foolish is as old as Aesop's "The Ant and the Grasshopper," but it acquired a sheen of scientific legitimacy in 1970, with Walter Mischel's legendary "Stanford Marshmallow Experiment":

https://en.wikipedia.org/wiki/Stanford_marshmallow_experiment

In this experiment, kids were left alone in a locked room with a single marshmallow, after being told that they would get two marshmallows in 15 minutes, but only if they waited until them to eat the marshmallow before them. Mischel followed these kids for decades, finding that the kids who delayed gratification and got that second marshmallow did better on every axis – educational attainment, employment, and income. Adult brain-scans of these subjects revealed structural differences between the patient and the impatient.

For many years, the Stanford Marshmallow experiment has been used to validate the cleavage of humanity in the patient and wise and impatient and foolish. Those brain scans were said to reveal the biological basis for thinking of humanity's innate rulers as a superior subspecies, hidden in plain sight, destined to rule.

Then came the "replication crisis," in which numerous bedrock psychological studies from the mid 20th century were re-run by scientists whose fresh vigor disproved and/or complicated the career-defining findings of the giants of behavioral "science." When researchers re-ran Mischel's tests, they discovered an important gloss to his findings. By questioning the kids who ate the marshmallows right away, rather than waiting to get two marshmallows, they discovered that these kids weren't impatient, they were rational.

The kids who ate the marshmallows were more likely to come from poorer households. These kids had repeatedly been disappointed by the adults in their lives, who routinely broke their promises to the kids. Sometimes, this was well-intentioned, as when an economically precarious parent promised a treat, only to come up short because of an unexpected bill. Sometimes, this was just callousness, as when teachers, social workers or other authority figures fobbed these kids off with promises they knew they couldn't keep.

The marshmallow-eating kids had rationally analyzed their previous experiences and were making a sound bet that a marshmallow on the plate now was worth more than a strange adult's promise of two marshmallows. The "patient" kids who waited for the second marshmallow weren't so much patient as they were trusting: they had grown up with parents who had the kind of financial cushion that let them follow through on their promises, and who had the kind of social power that convinced other adults – teachers, etc – to follow through on their promises to their kids.

Once you understand this, the lesson of the Marshmallow Experiment is inverted. The reason two marshmallow kids thrived is that they came from privileged backgrounds: their high grades were down to private tutors, not the choice to study rather than partying. Their plum jobs and high salaries came from university and family connections, not merit. Their brain differences were the result of a life free from the chronic, extreme stress that comes with poverty.

Post-replication crisis, the moral of the Stanford Marshmallow Experiment is that everyone experiences a mix of patience and impatience, but for the people born to privilege, the consequences of impatience are blunted and the rewards of patience are maximized.

Which explains a lot about how rich people actually behave. Take Charles Koch, who grew his father's coal empire a thousandfold by making long-term investments in automation. Koch is a vocal proponent of patience and long-term thinking, and is openly contemptuous of publicly traded companies because of the pressure from shareholders to give preference to short-term extraction over long-term planning. He's got a point.

Koch isn't just a fossil fuel baron, he's also a wildly successful ideologue. Koch is one of a handful of oligarchs who have transformed American politics by patiently investing in a kraken's worth of think tanks, universities, PACs, astroturf organizations, Star chambers and other world-girding tentacles. After decades of gerrymandering, voter suppression, court-packing and propagandizing, the American billionaire class has seized control of the US and its institutions. Patience pays!

But Koch's longtermism is highly selective. Arguably, Charles Koch bears more personal responsibility for delaying action on the climate emergency than any other person, alive or dead. Addressing greenhouse gasses is the most grasshopper-and-the-ant-ass crisis of all. Every day we delayed doing something about this foreseeable, well-understood climate debt added sky-high compounding interest. In failing to act, we saved billions – but we stuck our future selves with trillions in debt for which no bankruptcy procedure exists.

By convincing us not to invest in retooling for renewables in order to make his billions, Koch was committing the sin of premature avocado toast, times a billion. His inability to defer gratification – which he imposed on the rest of us – means that we are likely to lose much of world's coastal cities (including the state of Florida), and will have to find trillions to cope with wildfires, zoonotic plagues, and hundreds of millions of climate refugees.

Koch isn't a serene Buddha whose ability to surf over his impetuous attachments qualifies him to make decisions for the rest of us. Rather, he – like everyone else – is a flawed vessel whose blind spots are just as stubborn as ours. But unlike a person whose lack of foresight leads to drug addiction and petty crimes to support their habit, Koch's flaws don't just hurt a few people, they hurt our entire species and the only planet that can support it.

The selective marshmallow patience of the rich creates problems beyond climate debt. Koch and his fellow oligarchs are, first and foremost, supporters of oligarchy, an intrinsically destabilizing political arrangement that actually threatens their fortunes. Policies that favor the wealthy are always seeking an equilibrium between instability and inequality: a rich person can either submit to having their money taxed away to build hospitals, roads and schools, or they can invest in building high walls and paying guards to keep the rest of us from building guillotines on their lawns.

Rich people gobble that marshmallow like there's no tomorrow (literally). They always overestimate how much bang they'll get for their guard-labor buck, and underestimate how determined the poors will get after watching their children die of starvation and preventable diseases.

All of us benefit from some kind of cushion from our bad judgment, but not too much. The problem isn't that wealthy people get to make a few poor choices without suffering brutal consequences – it's that they hoard this benefit. Most of us are one missed student debt payment away from penalties and interest that add twenty years to our loan, while Charles Koch can set the planet on fire and continue to act as though he was born with the special judgment that means he knows what's best for us.

On SEPTEMBER 24th, I'll be speaking IN PERSON at the BOSTON PUBLIC LIBRARY!!

If you'd like an essay-formatted version of this post to read or share, here's a link to it on pluralistic.net, my surveillance-free, ad-free, tracker-free blog:

https://pluralistic.net/2024/09/04/deferred-gratification/#selective-foresight

Image: Mark S (modified) https://www.flickr.com/photos/markoz46/4864682934/

CC BY 2.0 https://creativecommons.org/licenses/by/2.0/

[nonenosome2]

I know this is a meme, but it got me thinking.

Why is this on the men for not wanting to go to therapy and not on the therapists for making men not want to go to therapy?

Like, I know it is a "haha, men would rather X than go to therapy" thing, but if men would rather die than come see you, maybe, just possibly, you are doing something wrong?

I know the Replication Crisis has hit psychology hard, so maybe once they catch up with the scientific method, and don't just go with "This is true. Source: Trust me bro" they will finally figure this out.

[thefloralmenace]

TDLR: Don't distrust science; Be skeptical of areas with no research and studies without any follow-ups

Since my angry post about how science journals tend to work (with a few nice exceptions as I've found out via some lovely Tumblr users) has gotten widespread, I want to clarify something related to mistrust of science:

Almost all individual scientists are trustworthy. I've worked with a bunch of them, including a guy known as "One of the Four Horseman of Chemistry" at my old university (for being a pain to work with), and I've gotten along with them excellently. They have never asked me to p-hack, they have never asked me to manipulate data. Bad scientists who are constantly majorly manipulating their data are a rarity. The two I can think of right off hand are that guy at Standford who got exposed by a student journalist this year and some dude in the fly science area who keeps publishing (and vehemently defending) a bunch of stuff that no one can replicate. Also the good news here is that they do tend to get caught, if not by a student journalist, by their peers, and this is usually because bad scientists who are motivated solely by ego and puffing up the importance of their careers tend to eventually publish something that seems so ground-breaking that other scientists get incredulous, and that experiment bypasses the replication crisis [the phenomenon where experiments aren't repeated by different groups of scientists because they've already been published and won't get republished or contribute to anyone's career]. My science professor friend told me about this happening with a Korean lab that swears they can make superconducting happen at room temperature. That's a big claim. Other scientists were immediately like "Hold up," tried to repeat their methods, and no one can make it work, so that lab has lost a lot of credit in the science community.

Individual scientists, the scientific process, scientific methods, and peer review are generally very trustworthy. Science as a whole is generally very trustworthy.

What you have the right to be suspicious of is 1) Lack of published research on a given thing, 2) Studies with important results that you cannot find any follow-up studies on, especially if they've been widely spread around as "proof" of something.

And THOSE are the problems that come from the fact that very few journals and pretty much no highly prestigious journals publish negative or null results (i.e. "We found nothing interesting") or repeated trials of experiments that have already been published. This disincentivizes reseachers repeating the work of other research groups (the replication crisis). This is not a moral failing on the part of the scientists; they just know that even if they do do repeats of other people's work, it will likely never see the light of day, so there is literally no point in spending years doing that when funding and resources are limited (and that's pretty much always the case in research). Some labs have taken on the responsibility of repeating other people's experiments, but most of that gets shared internally among science groups. This helps improve science from the inside, but because journals won't publish these studies, the public is denied these same confirmations. Scientists aren't hiding things from you; the journal publishing industry is forcing information underground. Then not replicating experiments can result in a single study finding something "interesting" by chance rather than because there's actually a causatory relationship and then becoming accepted science when it's just a fluke: Like the study showing differences in men's vs. women's brains. Not true, just got a statistical difference by chance that one time, which then took a very long time to realize. This situation is once again, not a result of malice or manipulation by the scientists - it's just something that happened by accident.

And since very few people publish null results, scientists probably frequently spend a lot of time accidentally replicating experiments that turned out "boring" and were never published thinking it has simply never been done (which is a waste of time because they won't be published, no one will ever know it's been repeated a billion times, and if it HAS been repeated a billion times, we really can just move on and direct resources to other projects). The professor I know says that scientists generally assume that if the experiment would be easy to set up and has nothing published on it means that it's been tried and found to be uninteresting... which adds the additional problem of experiments never being done because people assume they're too easy.

Once again, notice that at every stage of the game here, the issue is the publishing conventions, not the morals or intentions of the scientists.

Now when you combine these conventions with limited funding that is often allocated based on how prestigious the lab and its work are, which is frequently determined by how many publications and citations it has, WHICH IS IN TURN affected by whether or not you can get your work into high impact-factor journals, there starts to be an incentive to overstate mildly significant results or even mess with the statistics in pursuit of trying to keep finding interesting things to publish, so that you will keep getting funding, so that you can continue your research.

And here I would also like to lay way more blame on the publishing conventions than the scientists (a lot of people quit academia rather than exaggerate results). If the literal survival of a lab was not directly tied to how much and where they publish, there would be no incentive to exaggerate. Scientists that give in to that are also not necessarily horrible and immoral either - they may be banking on their line of research being really important, so they feel the need to overstate initial results so that people will keep giving them money to keep on going. These people also have livelihoods, and "publish publish publish or lose your job" is a really tough spot to put people in then expect them to behave perfectly. Scientists are human, and humans put under immense pressure will break down and make decisions in their own self-interest or in the interest of what their fallible human judgments think is the greater good. That's not malice, that's not bad people trying to fool you, that's not people using science to push an agenda. That is a direct result of the strangle-hold the publishing industry has on science.

Oh and also: There are predatory journals that will publish anything if you pay them a huge amount of money, so now people also have to look up THE JOURNAL to see if it's legit - that's a big thing scientists are doing a good job of warning young scientists about. The legit journals, which sit behind a paywall, also do not pay anything to the scientists they publish. That paywall goes 100% to the publisher and not the authors or their lab.

The whole system keeps science labs underfunded and desperate and promotes major gaps in our collective knowledge and fact-checking machine. In short, don't distrust science. Major manipulators are found out, The Journals of Negative and Null results have been established because people recognize the issues above, some labs have taken on the responsibility of repeating other people's experiments even if these results are never published, and peer review does ferret out a lot of potential manipulation before it is ever published. It is the publishing conventions created by the journals that are preventing us from attaining the highest possible quality of science that would absolutely occur if scientists were relieved of all these pressures.

[alpaca-clouds]

The Science Problem

My good people of tumblr,

We do need to talk about science. Even though I fear a post like this will once again fail to break containment. Still, I do see from time to time posts go around citing scienctific papers - but doing it either in bad faith or not understanding a specific issue.

This issue is that science can be wrong. Even if it was super influential.

I hope that most people on this (usually fairly left leaning) website will generally agree that manmade climate change is real, and that vaccination is indeed something that helps to prevent the spread of disease. This is generally something that the vast majority of science agrees on by now. However, you will be able to find papers - even recent papers in at least medium sized journals - that seek to disprove this.

In these cases we all know the main reason for it: most people trying to disprove manmade climate change are paid off by the fossil lobby. Meanwhile, most people trying to disprove vaxination are some "alternate healthcare" nutjobs far down the crunchy pipeline. If you look at their studies, you usually find a lot of issues with the way they are set up.

Positively we have to say: yes, the majority of science does agree in both cases. Climate Change is real, and vaccination is good for many people. Great. Hooray.

However, here is the thing: just because a lot of scientists agree on something does not make it right. Sadly, there is this phenomen in science where sometimes something is assumed that just makes somewhat makes intuitive sense. And then there is the phenomen where things are argued in bad faith that when argued make inuitive sense.

Several good examples of this are brought in by Naomi Klein in Merchants of Doubt, where she goes through examples of companies investing in science to get specific outcomes and how it shaped the idea of doubting science.

But there are also other examples. One of my favorite is the concept of IQ. The IQ test was not invented with bad intentions (it was a pre-school test in France), but it was wider dispersed as an "intelligence test" by an eugenicist, and then people just ran with it being a test for intelligence until recently. Till this day A LOT OF PSYCH STUDIES are constructed around IQ tests. But that does not make it anymore true that the IQ test tells you a whole lot about intelligence. It might be able to tell you some limited how much talent and training a person has in specific areas (math, logical reasoning, spatial reasoning, memory, and some specific linguistic skills), but that is just a tiny part of what makes intelligence.

Either way: sometimes it just turns out that for one reason or another there is this one thing in science that EVERYONE THOUGHT was a given and that eventually turns out to be wrong - with this meaning that a ton of research that might have build on it might be wrong or at least not fully right or needing to be reconsider. (For example: some studies build on IQ still have interesting observations outside of corellation to this IQ - but still need to be reevaluated in this regard.)

Another good example is the entire thing with fatness. You know, for a long while it was common sense that "obesity causes diabetes". But then fairly recently we have learned "actually, people with the genetic predisposion to develop diabetes are more likely to be fat due to their metabolism working different".

Then there is also the entire replication issue, the Bernoulli fallacy, the automtic bias of many studies (due to WEIRD subjects) and so on and so forth.

Why am I talking about this?

Because at times I see both people I agree with, and people I disagree with bring up studies that have clear issues - and go unchallenged.

You all know the examples where I get annoyed about this: People sharing studies that assume heteronormativity, studies that assume biological monogamy in humans, studies that very clearly leave out alternate explanations, studies that assume that correlation is causation and so on.

And also some of them do rely on research that has been debunked.

Yes, generally there is things we can assume is real: Climate change, vaccination, smoking is bad and so on and so forth. Oh, trans people exist, is also one of those.

But not everything that studies say is always true. And you need to apply critical thinking to it.

[jadagul]

I am spectacularly offended by this Matt Levine reader email about using astrology in consumer finance prediction.

This was a machine learning model – the job of the data scientist was, put everything in, see what's significant, of that discard everything that's discriminatory, the rest is your model. Ultimately with twelve astrological signs it's over 50/50 that one will come out significant at 95%. I thought it was elegant. "Astrological signs? Do you believe that?" my boss said. I said it wasn't a question of belief, I was a statistician and was going to follow the numbers rather than letting anyone's preexisting theories about the stars and planets influence the data science. I think he believed that meant I'd agreed to take it out.

Like, the guy literally said "We're very likely to have a false positive here by chance, but since we got one we have to take it seriously. I'm a statistician."

He's fully aware that he's p-hacking and garden-pathing. He's fully aware of the multiple comparisons problem. And then he endorses the conclusion anyway!

(And, as a side note, it's not over 50/50; If you do twelve tests the chance of one coming out significant by chance is about 46%. So he fucked up the arithmetic too!)

[serialport0]

j.d. vance talking about the replication crisis? wow. maybe things are going to be alright after all

[probablyasocialecologist]

Recommendations to meditate, exercise, and pursue nature for happines…

Recommendations to meditate, exercise, and pursue nature for happiness rely on weak evidence

According to a new paper in Nature Human Behavior, media reports most often recommend five strategies for happiness: expressing gratitude, increasing social interaction, practicing mindfulness meditation, exercising, and spending time in nature. Over the past decade, journalists have repeated these recommendations so often that they have begun to sound almost like simple common sense. But the media has relied on studies published before the field of psychology began to reckon with a so-called “replication crisis,” write the authors of the paper, Canadian psychologists Dunigan Folk and Elizabeth Dunn from the University of British Columbia in Vancouver. Around 2011, scientists began to notice that a majority of psychology studies could not successfully obtain the same results when repeated. Researchers in the field, it turned out, consistently relied on weak research methods, such as selective reporting of results, exclusion of certain participants, and small sample sizes—too few participants to yield statistically significant results. “It would be easy to assume there is a strong base of evidence for these [happiness] strategies,” write Folk and Dunn, “given the frequency with which they are recommended.” But while the five happiness tips may not qualify as snake oil, exactly, their benefits at this point are theoretical at best and their success with any one person may depend heavily on individual personality or preferences. To assess the validity of the big five happiness strategies, Folk and Dunn identified 532 studies that experimentally investigated them with a non-clinical population. When they reviewed this batch of studies, they found that the vast majority used too few participants to yield reliable findings and failed to “pre-register” their studies—a research method that aims to improve data quality, prevent cherry-picking, and reduce publication bias, which is the publication of studies that fail to find an effect—among other things.