#p hacking

[inspired-lesson-plans]

A few years ago, I got briefly obsessed with (and had a Tumblr post break containment about) the then-ongoing downfall of Cornell marketing professor Brian Wansink, an influential food psychology researcher who turned out to have been doing scientific fraud on a massive scale. This ranged from having grad students record every random thing they possibly could about a study and then slicing and dicing the data until they found a correlation between two random things that met a statistical significance threshold and only reporting that, to impossible data, to copy-pasting thousands of words between allegedly unrelated papers, to extremely basic arithmetic errors like sample sizes that don't add up. The scientists who exposed him dissected several dozen of his papers, but he's published over 200, and pretty much all of them seem to be like this. I've thought ever since that it would make a great school activity to bring in a stack of random Wansink papers and challenge the class to see how many problems they can find.

Oh, I've heard about this guy! I think they talked about him on an episode of Freakonomics, or else somewhere on NPR.

Academic dishonesty, especially regarding scientific publications, it's something very near and dear to my heart. And the fact that this guy produced so many in easy, delightful, daytime talk show style. Scientific headlines means that I think I know the angle I would want to take with this just because somebody says that something is science, doesn't mean it is. It's not a matter of you. Can't trust the mainstream media, it's a matter of people need scientific literacy in order to be trustworthy when reporting science.

But the reason why I'm responding to this ask instead of just making a lesson plan, Which I do intend to do, is that I never ever actually took any classes on scientific methodology. All I know about p-values I learned from Khan Academy and Wikipedia.

So do you think we could collaborate?

If I draft up a lesson, could you proofread it for me to make sure that it's got all the information that you would like to see students learn?

And if anyone else knows any way they can help, please leave a comment or DM me. I'll be grateful to gain a better matter of this topic.

[system-of-a-feather]

Wanted to share this very good video on the Replication Crisis as someone who took a whole college course on this taught by one of the researchers dedicating her career to this topic ^^

[teachanarchy]

[warmhappycat]

[psychbuilder]

The Reformation: Can Social Scientists Save Themselves?

By Jerry Adler

Pacific Standard: The Science of Society

Originally posted April 28, 2014

Here is two excerpts to a long, yet exceptional, article on research in the social sciences:

OUTRIGHT FAKERY IS CLEARLY more common in psychology and other sciences than we’d like to believe. But it may not be the biggest threat to their credibility. As the journalist Michael Kinsley once said of wrongdoing in Washington, so too in the lab: “The scandal is what’s legal.” The kind of manipulation that went into the “When I’m Sixty-Four” paper, for instance, is “nearly universally common,” Simonsohn says. It is called “p-hacking,” or, more colorfully, “torturing the data until it confesses.”

P is a central concept in statistics: It’s the mathematical factor that mediates between what happens in the laboratory and what happens in the real world. The most common form of statistical analysis proceeds by a kind of backwards logic: Technically, the researcher is trying to disprove the “null hypothesis,” the assumption that the condition under investigation actually makes no difference.

(cut)

WHILE IT IS POSSIBLE to detect suspicious patterns in scientific data from a distance, the surest way to find out whether a study’s findings are sound is to do the study all over again. The idea that experiments should be replicable, producing the same results when run under the same conditions, was identified as a defining feature of science by Roger Bacon back in the 13th century. But the replication of previously published results has rarely been a high priority for scientists, who tend to regard it as grunt work. Journal editors yawn at replications. Honors and advancement in science go to those who publish new, startling results, not to those who confirm—or disconfirm—old ones.

The entire article is here.

[leetakeuchi-blog]

My current Astronomy professor is responsible for the satellite that took this image.