In my field (cognitive behavioural psychology, and i guess psychology more generally too) this is known as the "file drawer problem", i.e. the vast majority of non-significant findings are left in some researcher's file drawer never to see the light of day because it's not as publishable as the ones that are significant. It's a key factor in what we call the Replication Crisis, where researchers completely failed to replicate several foundational studies on which like half the field was based. Once the first couple effects couldn't be replicated (social priming, ego depletion, power posing, the marshmallow test, etc.), it caused a massive domino effect that revealed widespread replicability issues across the entire field of psychology. Half the famous pop-psychology stuff that gets passed around is based on studies that can't be replicated (and trust me, people have tried really hard to replicate them, it really is egg on our faces for all the most famous bits of our field turning out to be fake). Effects we all thought were really solid turned out to have been statistical anomalies concealed by the file-drawer problem. Stuff that was published in places like Nature and Science. Prominent researchers were exposed for faking huge amounts of their data. Meta-analysis showed suspicious gaps in the distributions of published p-values, indicating people were lying about marginally-significant results (i.e. if the data gave a p-value between 0.05 and 0.06, people were "rounding down" to 0.05 so it technically met the significance threshold). Evidence emerged of researchers collecting massive amounts of data and arbitrarily including or excluding participants until their data showed significant results, or simply collecting more and more data until they had significant results (p-hacking). Papers got retracted, funding was pulled, people got fired. It was a total fucking mess that almost collapsed the entire field, and it happened because of a couple of factors:
a) you can't "prove" anything from non-significant results (i.e failing to find evidence of a link between purple jelly beans and acne is not the same thing as finding evidence *against* a link between purple jelly beans and acne) and so you can't really reach any definitive and interesting conclusions. If you find significant evidence of something you can make all kinds of interesting claims and discuss how it fits or conflicts with existing literature, and propose ideas to explore how and why it works like that, etc. If your findings are nonsignificant, the best you can do is be like "idk man, it's weird it didn't work though, right? Maybe our methods were flawed, or maybe our participants are weird, or maybe there just is no effect here - I sure as fuck couldn't say!" and keep it pushing (while hoping people continue to pay you to do this).
b) Academic journals gain their prestige/value from how often their articles are cited (with for-profit, pay walled journals this is also their main source of revenue bc people/institutions will pay them for access to the articles others have cited, etc.), and so they are motivated to publish things people will want to cite lots. People typically cite articles in order to reference a specific finding or theory it shows evidence for. Therefore, papers with non-significant findings are unlikely to be cited nearly as often as those with significant findings (on account of how they generally don't prove anything) and are thus less valuable for the journal to publish. This maintains the journal's reputation for being difficult to get published in and also for publishing only the most high-quality, "impactful" work (bc everyone is citing it, so it must be important!), ergo the prestige thing. Open-access journals and those dedicated to tackling the file drawer problem are available but are far less prestigious, owing to the lack of citations they get.
c) Academics are judged in hiring and grant application processes by how many papers they have published, how well-cited their work is (ditto above), *and* how prestigious the journals they've published in are. This is the "publish or perish" problem yarning mentioned. In my field applicants for academic positions etc. are often ranked by a combination of Journal Impact Factor (average number of citations received within a two-year window by articles published by the "best" journal you have also published in) and h-index (highest number, h, for which your top h most-cited papers have at least h citations. E.g. if you have 20 papers, and the top 7 are all cited at least 7 times but the next most-cited is only cited twice, you would have a h-index of 7). As you can maybe see, this makes resisting the file-drawer problem (and, to an extent, the other far more unethical methods I mentioned above) quite difficult on the part of individual academics, who need to prioritise publishing work that will get cited if they want to compete for jobs and grants. As an early-career researcher with relatively few published papers, pushing to get your non-significant results published is practically career suicide in a lot of fields especially if you're not working on something that gets a lot of funding opportunities to begin with. It can sometimes look worse to hiring/grant committees to have 6 papers with no citations than to have no papers at all bc the default assumption is often that this means your work is low-quality in some way (this perception has started to change, thankfully). It can be a bit easier for more established researchers for whom publishing low-impact papers doesn't really matter so much and who are probably not having to scrabble for funding etc. as much anyway, but they are also less motivated to push for publication in general.
There isn't a super easy solution to the problem tbh, and it is only compounded by the pressure Derin mentioned to twist whatever you actually want to study to include whatever the hot new analysis/theory/whatever is favoured by your friendly neighbourhood funding-bodies. If people are paying you to look into The Thing, then you do kind of have to try to get it published if you find it, even if it is a bullshit half-baked addition tacked onto the end of the bit you're actually interested in. And conversely, the people invested enough in The Thing to pay you to study it probably don't particularly care if you publish papers that don't find evidence for it, in fact they would probably rather you didn't. Citations are also an abysmally poor metric of the value of a given piece of work to the field, but we don't really have a better one. It *is* becoming more understood how flawed the system is though, and work on large-scale multi-lab replication studies and meta-analyses is gradually being valued more, which is good to see. More journals are also opting for a pre-publication system, where you submit your introduction, rationale, and planned methodology for approval *before* you collect data and get your results. If your pre-pub is accepted, the journal agrees to publish the work regardless of whether the results themselves are significant, which avoids the file drawer problem while also allowing the journal to maintain certain standards of quality based on the methodology and theoretical foundations. Open access, and especially the proliferation of predatory pay-to-publish journals, has gotten a reputation of letting literally anyone publish whatever they like with very poor peer review processes, which contributes to the lack of prestige associated with publishing outside the more traditional journals. The pre-pub system helps to combat that reputation, which makes it more of an option for early-career researchers. Pre-pubs are also often made public in some capacity, which helps to prevent things like p-hacking and retroactively changing hypotheses to fit findings - a solid improvement in transparency imo. Doesn't solve the problem of people potentially straight-up faking their data, but it reduces the incentive to do that stuff if you know you'll get a publication regardless. Funding bodies are springing up whose whole thing is giving grants for replication work across various fields, with expectations that researchers will publish whatever they find, significant or not. There's not a lot of them so far, but at least there are some ways to fund research that aren't so biased towards reinforcing the file drawer problem for whatever the grant committee's pet theory is. It's still a mess, but recent shifts have given me some hope tbh.
Side note bc i can't help myself: The whole "absence of evidence is not evidence of absence" thing is a deliberate feature of how hypothesis testing works, not a bug. You predict what will happen *if the theory you are testing is true* (e.g., if our hypothesis, H¹, is that purple jelly beans are associated with acne, we would predict that participants who eat purple jelly beans would develop more acne than those who don't), and then you run statistical tests to see whether this hypothesis explains the data you actually collect better than assuming there is no effect (which we call a null hypothesis or H⁰) within some pre-determined confidence interval, which is usually 95% or 99% for my field but the standard for statistical significance varies a lot. All your basic, garden-variety statistical hypothesis tests (t-tests, ANOVAs, correlations, regressions, etc.) have a built-in assumption that there is no effect and then test for evidence to contradict that assumption. Because of this, it's impossible to tell a false negative, or the presence of a conflicting effect in the opposite direction, from the actual absence of an effect based on one of these tests. The idea is that we would rather accept a false-negative and dismiss an effect that does exist (which others will surely find evidence of on replication if it is actually reliable) than risk a false positive where we believe in an effect that doesn't exist. In some cases, we use another H⁰, depending on the specific setup of the experiment, whether it's reasonable to assume *no* baseline effect vs. an established level of effect, etc. but it always reflects some baseline assumption that we are comparing H¹ to. Each time you run these tests you can only check for evidence of one H¹ at a time, so failing to find that evidence doesn't mean H⁰ is true necessarily (it could be any number of other alternate explanations), it just means we haven't found enough evidence to suggest H¹ is sufficiently more likely than H⁰, so we can't be confident H¹ is true. If you are testing multiple different hypotheses, and therefore running the same test multiple times, you are *supposed* to add a statistical correction for multiple-comparisons to avoid the inflating family-wise error rate (there are a couple different methods depending on how conservative you want to be with your confidence intervals etc. I won't get into all of that here unless anyone actually wants to know, lol.) This doesn't always happen, especially if multiple different papers are analysing the same dataset since technically, each person is only doing the test once, even though overall you do get the same issue of family-wise error rates. Meta-analyses, where you look at the papers that have already been published on a given topic and analyse what the cumulative evidence says, typically do correct for multiple comparisons in some way, though in my experience.
dailybridgerton
titsay
annaxmalina
kaledo-art
stormborns
izzy806
ellievsbear
