The Replicability Crisis in Science
The world of science is in the midst of unprecedented soul-searching at present. The credibility of science rests on the widespread assumption that results are replicable, and that high standards are maintained by anonymous peer review. These pillars of belief are crumbling. In September 2015, the international scientific journal Nature published a cartoon showing the temple of “Robust Science” in a state of collapse. What is going on?
From Nature, September 15, 2015. Reproduced by courtesy of Nature Publishing Group.
Drug companies sounded an alarm several years ago. They were concerned that an increasing proportion of clinical trials was failing, and that much of their research effort was being wasted. When they looked into the reasons for their lack for success, they realized that they were basing projects on scientific papers published in peer-reviewed journals, on the assumption that most of the results were reliable. But when they looked more closely, they found that most of these papers, even those in top-tier academic journals, were not reproducible. In 2011, German researchers in the drug company Bayer found in an extensive survey that more than 75% of the published findings could not be validated.
In 2012, scientists at the American drug company Amgen published the results of a study in which they selected 53 key papers deemed to be “landmark” studies and tried to reproduce them. Only 6 (11%) could be confirmed.
In 2012, the governments of the world’s richer countries spent $59 billion on biomedical research, one justification for which is that basic-science research provides the foundations for work by private drug companies. So this is not a trivial problem. Meanwhile, by 2013, in the realm of experimental psychology, as in other branches of science, there were alarming signs that much of the published research could not be replicated. A large-scale replication study by psychologists published last year sent further shock waves through the scientific world when it turned out that around two-thirds of the published studies in top psychology journal were not reproducible.
In the late nineteenth century, many scientists adopted a style of writing using the passive voice, “A test tube was taken….” instead of “I took a test tube…” to create as impersonal a style as possible, a world of emotion-free events unfolding spontaneously in front of a detached objective observer.
In reality, of course, scientists are people, and like other people have different temperaments and personalities from each other, are often competitive, and prefer their own hypothesis to be right rather than wrong. In most branches of science, scientists publish only a small percentage of their data, 10% or less, and obviously select the “best” results to publish, leaving inconvenient or inconclusive data unpublished. The problem is made worse by a systematic bias against replications within the sciences. Researchers who replicate other people’s work find it hard, if not impossible, to get their papers published, because replication is not deemed to be original, and most journals pride themselves on publishing original research.
Unfortunately, personal advancement in the world of science depends on incentives that encourage these questionable research practices. Professional scientists’ career prospects, promotions and grants depend on the number of papers they have published, the number of times they are cited and the prestige of the journals in which they are published. There are therefore powerful incentives for people to publish eye-catching papers with striking positive results. If other researchers cannot replicate the results, this may not be discovered for years, if it is discovered at all, and meanwhile their careers have advanced and the system perpetuates itself. In the world of business, the criteria for success depend on running a successful business, not on whether business plans are ranked highly by business academics, and whether they are often cited in business journals. But status in the world of science depends on publications in scientific journals, rather than on practical effects in the real world.
Meanwhile, the peer-review system is falling into disrepute. The very fact that so many unreliable papers are published shows that the system is not working effectively, and a recent investigation by the American journal Science revealed some shocking results. A member of Science’s staff wrote a spoof paper, riddled with scientific and statistical errors, and sent 304 versions of it to a range of peer-reviewed journals. It was accepted for publication by more than half of them.
Obviously the present system of academic research encourages the publication of false positive results. At the same time, the huge financial incentives that underlie the multi-billion dollar drug industry encourage the suppression of negative results. Many drug companies simply do not publish the results of negative studies that show their drugs are ineffective. On the other hand, of course they publish the results of positive studies that favour their drugs. Insofar as “evidence-based medicine” relies on published studies, it creates a very misleading impression of scientific objectivity, reflecting a strong bias based on the commercial self-interest of pharmaceutical corporations.. Such practices are all too common, as Ben Goldacre shows in his book Bad Pharma: How Drug Companies Mislead Doctors and Harm Patients (2012).
The psychologist Nicholas Humphrey has compared this “sub-prime science” crisis to the financial crisis of 2008. The implications of this crisis are far-reaching, because science is so important for our civilisation and economy. There is now an unprecedented mood of humility within the sciences. Whether there will be serious changes, or simply a reversion to business-as-usual, remains to be seen.
2087
origamiaround
titsay
tangentwithtanner
dailybridgerton
turnnoffyourmind
andulkaphoto
