Discover Top Posts Tagged with #reproducibility

Popular Recent

I’m not gonna deny that there are valid criticisms of Mythbusters from the standpoint of science education, but when you come right down to it, the show is about assessing reproducibility – and whatever issues one might have with their methods, that they’re concerned with reproducibility at all puts them head and shoulders above a lot of the stuff that makes its way into the published journals these days.

#media #television #mythbusters #science #reproducibility

i'd love to see a site which lists as many scientific studies as possible and presents some clean info about them. not even containing the studies themselves (since those are always behind a paywall) but just some meta info about things like the sample size, P-value, reproducibility, etc. just all the stuff you'd need to know about whether it's good quality or not. being able to search for a study and quickly know whether it's a good one that has a good P-value, large sample size, and has been reproduced or if it was one done on 9 people once is really important to know if it's worth listening to

#i could make something like this #like technically #i can make sites #but it's the collecting of papers and cataloging them that's the issue #i couldn't do that myself #for it to be of any real use you'd need a bunch of people #tbf i could just make it and see #but there's other stuff i could work on that is more immediately rewarding for me #science #studies #papers #reproducibility #lizabeth talkabeth

A rose by any other name would smell as sweet

What they say:

“Social science is in crisis! We ought to take measures to improve the state of our field!”

What you think they mean:

What they actually mean:

[...] Lincoln and Cannella argued that championing a rigid set of “gold standard” methods reflects dangerous epistemologies shaping academic discourse. These rigid norms—what they called “methodological conservativism” at the time—have profound implications for research, including which methods are taught and supported, which research imaginaries are cultivated, which projects are valued, which are funded, and which understandings of the world are nourished. They note a key danger that is worthwhile to consider in a methods collection because of its salience to researchers’ very identities. [...] All researchers are vulnerable to shifting forms of normalization and prescription, to discourses that champion hegemonic standards, or even “new” and “better” methods, including those oriented toward social justice. Claims about any methods can become prescriptive, associated with a “right” and “good” identity to embrace as a researcher, whether through dominant or feminist and critical approaches.

This is from Research Methods for Social Justice and Equity in Education, which I acquired via media. The focal point is clear from the title, and so it might be low-hanging fruit, but let’s see what this book has to say about “truth”. It gets 52 matches, whereas something like “qualitative” nets you 241 results. What’s more interesting, though, is the context of the results, namely which words surround the word “truth”. In the first three results we find: the word “truth” literally in inverted commas (“scare quotes”, they’re sometimes called, to be used as a sign of irony); “multiple truths”; the sentence “[life] history challenges the idea of a universal truth—stemming from Eurocentric positionalities.”

There’s a whole chapter called ““Objectivity”, Truth and Multiple Realities”, and yes, there are again scare quotes around a word. Let’s see some interesting excerpts from this chapter. For fun, let’s emphasise the claims to objectivity within the text with italics.

As a critical scholar and a qualitative researcher, my approach to “objectivity” is often unlike approaches of other faculty many of my introductory qualitative students have encountered or have studied under. Because we begin with a history (and within a present) of marginalization, that is, a structure of inequality—where individuals and groups have been and continue to be denied access to and the opportunity to contribute to “official knowledge” as we think of it through “research”—attaining “objectivity” is not possible. Even striving for such a goal seems not only improbable but unjust in a variety of ways, in my opinion. This does not mean that we should not adhere to principles of sound research—that is, we should do whatever we like without regard to established practices for ethical research. It does mean though that participants’ realities are impacted by social, economic, political, and educational structures that have historically and presently limit access and opportunity. That is, the powerful exist, as do the disempowered. If we reflect on our history, the powerful have had access and opportunity to conduct research and create “knowledge.” Thus, truth is relative, and a single truth is not attainable. What is true for one participant or group may not be true for another participant or group. Regardless of the approach to research—critical or not—the realities of power, and relatedly, access and opportunity, remain.

That’s a lot of claims to truth in a single paragraph about how truth is relative. Let’s move on, but before we do, another great sentence:

“Objectivity,” as they have been taught, is something—perhaps if not wholly attainable—is always a thing to strive for.

The next part is where the author ostensively answers a question I, and perhaps you, have had in my head the whole time. Let’s see:

Another concept that introductory qualitative students struggle with is generalizability. They often come to class with ideas that suggest that the only research that is worth conducting is research that can be generalized to people outside of the sample. When I tell them that the intent of qualitative research is to better understand the particular, and that qualitative work is not meant to be generalizable—they often ask, “Well, then, why would anyone conduct a qualitative study?”

Great question! Why, indeed, especially as actually finding the truth (or even approximating it) appears not to even be a central aim in this type of “research”. So, let’s see what she has to say about that.

I ask them—“Have you read any quantitative studies that suggested that the findings are true for everyone?”

Okay... Not a great start, but let’s see if she improves.

Really, in my opinion, with any approach to research, one can only generalize to their own data—which is exactly what qualitative studies do.

I admire your commitment to saying “my opinion” to avoid the nasty situation where you might *gasp* lay claim to the truth, but this is patently false. Every single quantitative study will try to generalise to the “population”, and this is, in fact, the entire purpose of sample-based research. So, this remains purely an opinion, and this is fortunate, because opinions cannot be wrong. You could point out the irony in saying that no research is generalisable in a book about methodology, but let’s not.

This amounts to beating a dead horse, but since this chapter is such a trainwreck, I cannot help but include a few more excerpts. I will skip the author’s poor analogy and go straight fo the money shot, namely where the author talks about generalizability and objectivity, but just with different words!

Citing Lincoln and Guba, Merriam and Tisdell (2015) state:

the notion of “transferability,” in which “the burden of proof lies less with the original investigator than with the person seeking to make an application elsewhere. The original inquirer cannot know the sites to which transferability might be sought, but the appliers can and do.” The investigator needs to provide “sufficient descriptive data” to make transferability possible. (p. 254)

Gosh, transferability... Sure sounds a lot like generalisability. But she means of research methodology, so it’s totally fine, right?

Thus, “validity” has traditionally been understood by qualitative researchers as “credibility.” Because qualitative researchers will “never capture an objective ‘truth’ or ‘reality,’ the field has established a variety of strategies to increase the credibility or the “correspondence between research and the real world” (Wolcott as cited in Merriam & Tisdell, 2015, p. 244).

Wait a second... “correspondence between research and the real world”? You mean... objectivity? What is this positivist bullshit? What “real world”? Which of the multiple realities might you refer to?

That’s it, I’m out. Clearly, these researchers are just that kid trying to stuff a square peg into a round hole: they do care about success, they just want to do it badly, as long as they get to use their own methods.

#opinion #opinions #social science #methodology #replication crisis #science #nonscience #education #research #rant #reproducibility

New report examines reproducibility and replicability in science While computational reproducibility in scientific research is generally expected when the original data and code are available, lack of ability to replicate a previous study — or obtain consistent results looking at the same scientific question but with different data — is more nuanced and occasionally can aid in the process of scientific discovery, says a new congressionally mandated report from the National Academies of Sciences, Engineering, and Medicine. Reproducibility and Replicability in Science recommends ways that researchers, academic institutions, journals, and funders should help strengthen rigor and transparency in order to improve the reproducibility and replicability of scientific research.

#neuroscience #science #research #scientific research #research methods #neuroscience research #reproducibility #replicability

Meta-Science Articles

Here are some interesting paper on scientific publication standards, reproducibility and data analysis:

Metastudies for robust tests of theory. Baribault, B., Donkin, C., Little, D. R., Trueblood, J. S., Oravecz, Z., van Ravenzwaaij, D., … Vandekerckhove, J. (2018). Proceedings of the National Academy of Sciences of the United States of America, 115(11), 2607–2612.

Issues with data and analyses: Errors, underlying themes, and potential solutions. Brown, A. W., Kaiser, K. A., & Allison, D. B. (2018). Proceedings of the National Academy of Sciences of the United States of America, 115(11), 2563–2570.

Transparency in authors’ contributions and responsibilities to promote integrity in scientific publication. McNutt, M. K., Bradford, M., Drazen, J. M., Hanson, B., Howard, B., Jamieson, K. H., … Verma, I. M. (2018). Proceedings of the National Academy of Sciences of the United States of America, 115(11), 2557–2560.

An empirical analysis of journal policy effectiveness for computational reproducibility. Stodden, V., Seiler, J., & Ma, Z. (2018). Proceedings of the National Academy of Sciences of the United States of America, 115(11), 2584–2589.

#scientific publications #science #meta-science #data analysis #reproducibility

New Paper: Reproducible research in linguistics: A position statement on data citation and attribution in our field

I’m so excited to share this paper on data citation in linguistics, published in Linguistics. Not just because it was co-authored with 13 other linguists, including some of my linguistics heroes, but because it articulates so much of what we need to do to improve the way linguists think about data. (it’s Open Access, so you can read/download and share as you wish!)

I’m often reminded of something Nicholas Evans likes to say:

“Linguistics has a unique but unfulfilled destiny, as the most scientific of the humanities and the most humanistic of the sciences.”

It captures what I love about linguistics, but I think it also explains why we have a complicated relationship with the idea of data, and its management. This position paper outlines why it is important that linguists think about the data on which linguistic claims are made.

My Favourite line

Linguistic data are the very building blocks of our field. Given that linguistic theories need to be borne out through data, we believe that linguistic data are important resources in their own right and represent valuable assets for the field. Therefore, our field needs to accept responsibility for the proper documentation, preservation, attribution, and citation of these assets. The responsibility to do so is an integral part of linguistic research, and it should be collectively shared by individual scientists and researchers, data stewards, research institutions, and funding organizations.

Article writing Trivia

I’ve been working on some parts of this paper with Andrea Berez-Kroeker since 2013. Andrea has been working on some of the central ideas in this paper for even longer. Academia is a slow game sometimes, but this paper says a lot of things that a lot of the authors have been thinking about for a long time.

A lot of ideas in this paper formed the initial motivation for the Austin Principles of Data Citation in Linguistics

There is a reference to work on bacteria in human navels as an example of replicable research because I had read about Carl Zimmer’s participation in this research on his blog a couple of years early and it amused me greatly.

Abstract

This paper is a position statement on reproducible research in linguistics, including data citation and attribution, that represents the collective views of some 41 colleagues. Reproducibility can play a key role in increasing verification and accountability in linguistic research, and is a hallmark of social science research that is currently under-represented in our field. We believe that we need to take time as a discipline to clearly articulate our expectations for how linguistic data are managed, cited, and maintained for long-term access.

#reproducibility

Trending Tags

Recently Viewed Tags

#reproducibility