#understanding science

[thefugitivesaint]

'The Ear', ''Understanding Science'', Vol. 2, 1963

[drferox]

Science Literacy Lesson #1

Saying “There is no evidence that children transmit the virus or suffer the same mortality as adults” is NOT the same as saying “There is evidence that children do not transmit the virus and do not suffer the same mortality as adults.”

The first is saying there is a lack of information, that we don’t know. The second statement has gathered evidence.

[drferox]

Hierarchy of Evidence

Science Literacy Lesson #2

Here’s the thing about science: it’s trying to prove itself wrong. All the time. And if it fails to prove a hypothesis wrong, then it concludes it must be getting closer to the truth.

Not that it has discovered the truth, just that it is closer. This is why important concepts in science are termed Theories, they are inviting you to test them, to challenge them, to prove them wrong. Calling something a Theory does not mean it’s unproven. The Theory of Evolution is on equal footing as the Theory of Gravity, but you don’t see many people jumping off rooftops because ‘Gravity is just a Theory!’

(Compare this to dogma that actively discourages questioning or challenging it)

But how we gather that evidence is subject to a hierarchy. Not all evidence is equal, not all ‘proof’ is valid and this information gets ranked. This is infuriating for science people talking to non-science people who don’t know why these things are ranked as they are.

So since it’s topical, and in some discussions it’s always topical, here’s one basic Hierarchy of Evidence in ascending order, with examples below the cut:

An anecdote (doesn’t really count tbh)

Expert Opinion

Case Study

Series of Case Studies

Case Control Studies

Randomized Controlled Trials

Double blind randomized controlled trials

Systemic reviews of those controlled trials

Details below the cut

Anecdote: An Anecdote is the personal experience of one person. It does not have enough case numbers behind it for statistical analysis (1 in 1000 odds still happen sometimes, does not mean it’s common) and often not enough detail to extrapolate onto the wider population. For example: A dog doing well on a particular type of food, one person winning tattslotto, one person attributing their survival of an illness to eating only fruit.

Expert Opinion: An expert opinion is the personal opinion of one person who is highly educated and well read on the topic, providing it is on the topic they are an expert in. For example, while I might provide an expert opinion on a general veterinary medicine topic, I wouldn’t be qualified to provide an expert opinion on the relative benefits of one chemotherapy agent versus another in feline lymphoma. You have to keep the scope in context. But this is just a statement, an opinion. It’s not backed up by experiments or data unless that opinion specifically references them in which case you’re really deferring to the experimental data, not the expert’s opinion. Experts are informed, but they don’t make reality. And expert opinion should change in the face of higher ranked evidence. For example: Immunology professor saying you should vaccinate your kids, 99% of scientists saying climate change is a real effect and serious problem

Case Study: Now we’re getting somewhere! Not far, but somewhere! A Case Study is kind of like an anecdote, except somebody has actually written it up in detail to share with the wider scientific community, and open it up to criticism. It’s not enough evidence on it’s own to usually form a conclusion, but it useful for things that are rare (eg endangered species, rare cancers)  or very, very new when there hasn’t been time to do a more in depth study. For example, a case study about dog food causing morbidity in captive red pandas, case studies on an emerging condition.

Series of Case Studies: Similar to above, but where the case study might only have one data point, a series of case studies has multiple data points, all written up in detail. When patterns are playing out in multiple instances, it allows some trends to be guessed at, but the evidence provided is still so-so. You are still only observing and writing down what has happened, this is still not actually an experiment as you’d know it. For example, a case series of multiple rare cancers treated at the same hospital over a 10 year period.

Case Control Studies: More observations over larger numbers of subjects attempting to narrow down one factor. For example, you might take a group of people with lung cancer and compare them to a similar group of people matched for age and sex without lung cancer, and look for lifestyle or exposure differences, eg smoker status. Or looking at dogs and matching two groups for age/breed/sex and comparing illnesses with the age of desexing.

Randomized Controlled Trials: Finally we’re getting to the real experimenting! You have two groups, one gets the treatment/experimental variable (a drug, food, etc) and the other gets a placebo. Subjects get randomly assigned into each group and you look for statistical differences between the two groups. This is how most basic experiments are done.

Double Blind Randomized Controlled Trials: The good stuff! This is where the hard science is at! Double Blind means that not only are your subjects randomly assigned to different groups (the thing you are testing and placebo), but the subjects do not know which group they are in and the researchers do not know who is in which groups. This is to eliminate reporting bias. This might be done to test therapeutic applications of a medication, or pet food diet. It’s typically something you can administer and measure the results. A properly done Double Blind Randomized Controlled Trial is very strong evidence.

Systemic Reviews: What’s better than a DBRCT? Multiple Double Blind Randomized Controlled Trials that have had their results compared and analyzed together. For example, a systemic review of multiple pet feeding trials, or a systemic review of all treatment papers for a particular condition in the last 20 years.

Now you know what these things are and mean, it gives you an idea of how much faith to put into any given report. A case study? Interesting, might be worth a look but could go either way. Randomized controlled trial? Pretty good, probably useful data. Systemic reviews? We can have a lot of trust in conclusions from a thorough systemic review.

[drferox]

The Null Hypothesis & Significant Difference

Science Literacy Lesson #4

Let’s say you have two dice, a pair of d20s. 

The Null Hypothesis says there is no (zero) correlation between the numbers rolled by each dice, that they are both completely independent of each other.

You roll them together and they both come up with 20! Super lucky!

Or is it? How many times would you need to roll those dice to prove that they are linked to each other somehow?

The Null Hypothesis is the default assumption in science that two variables (things) just don’t affect each other. It’s the science equivalent of ‘innocent until proven guilty’. You need to prove a link between two things, you don’t assume it’s there.

So in this case we are assuming that the number rolled on the first dice will have no effect whatsoever on the number rolled on the second dice.

Ah! But both dice rolled a 20. So what does that mean?

It means your sample size was small.

You can’t reasonably make a conclusion about these dice based on a single roll. You can’t make a reasonable conclusion based on 5 rolls. The number of times you roll the dice is your sample size (often just called N when we’re discussing the experiment. So five subjects is N=5). If your sample size is unreasonably small, then it weakens your experiment and makes it harder to justify conclusions.

The sample size you require to prove your point varies depending on what you’re testing, but generally larger numbers are always better. An experiment with a sample size of 10,000 is significantly more meaningful than an experiment with a sample size of 12. This is quite simply a numbers game.

(That said, context is also important. If you were publishing a study about the Norther White Rhino, for example, and your sample size was only 2 that may still be significant as you do your best with what you’ve got.)

Now you know, as a reasonable human being or corvid-in-a-trenchcoat, that rolling the dice once doesn’t prove they’re linked somehow, because you know how dice work. A single incident is not any kind of useful evidence. Yet people use single incidents as anecdotes as ‘proof’ all the time! Can you see how frustrating that is?

You rolled the dice once and thought maybe the two dice were linked. You roll the dice 100 times and you will probably see those dice are not linked. The strength of your evidence will only grow as your sample size does.

But what if, hypothetically, you rolled this pair of dice 5 times, and each time you did so they came up with the same number?

Statistics in scientific experiments get complicated and you will completely fall asleep if I try to explain it, but experiments should be looking for a significant difference which is an indication of how likely or unlikely you would be to get the same result if the Null Hypothesis is true.

In this case, rolling your two d20 and them coming up with the same number each time for all 5 times in a row is something like 64 million to 1, if it’s just random chance. That’s a pretty significant difference.

But if you rolled the dice a thousand times and only 5 of those times came up with the same number? Not so significant. Another example of why you need to include your negative results.

Most results report their significant difference to a confidence value of 95 or 99. That means they’re either 95% or 99% sure that their results could not have been achieved by random chance alone. You can’t get to 100 of course, random chance is like that, but the closer you get, the stronger the evidence to reject the Null Hypothesis.

And, look, I’m more of a visual person, so have a graph. If your data produces a confidence value of 95, it means that there is a 95% chance the data didn’t just come from random luck. That the data wouldn’t fall under the blue bit of this curve if the Null Hypothesis is true.

And Confession: I don’t find these numbers fun. If I was running an experiment I’d totally pay a lovely and clever statistician to tell me what sort of sample sizes I would need to achieve my desired confidence interval. I’m not going to run down how you calculate them, just what these numbers are and why they matter.

The larger your sample size, the easier to prove a significant difference with a sigh confidence value. The weaker any of these factors are, the weaker the experiment in general.

Did that give you a headache? It gave me a headache. But hard science often involves some serious maths.

[drferox]

Control Groups and why they matter

(Also why we sometimes don’t use them and why that can be valid)

Science Literacy Lesson #3

Let’s say you have a brand new product that you think is wonderful (a new pet diet, an essential oil treatment, a new human antiviral, etc) and you want to do an experiment to prove it to the world.

You give it to a bunch of test subjects and 60% of them improve in whatever you’re measuring (pet health, anxiety, and patient survival in these examples). That’s great right? Proves it works?

Nope! Completely meaningless without a control group to compare it to.

You might have a hypothesis that your variable (the new food, treatment, medicine, etc) is effective, but you need to prove it against a population without that variable.

The saying in medicine is that about 2/3 of patients will recover no matter what you do. So it’s really, really important to know that a treatment you are recommending, or the variable you are testing in this experiment, performs better than ‘literally nothing’.

That’s where your control group comes in. If you gave 50 participants your experimental variable (food, essential oil, antiviral etc) and 30 improved in whatever you were measuring, but you also have a group of 50 participants that you didn’t give it to and 30 of them also improved, then your product is not as good as it initially looked.

So what might constitute a ‘control’ group?

An ideal control group receives no treatment. For some types of tests that’s not completely feasible as subjects are aware of whether they’re taking a pill or not, so that’s where a placebo is used. Both subjects are taking a pill, but only one is real and the other is known to produce no measurable results.

Using a placebo for the control group is ideal for something like the essential oils example, because you are testing an active product against nothing. You want to know whether that essential oil has any better results than the placebo.

This gets a little harder when testing a diet, because you can’t feed your control group literally nothing. Obviously the group being fed literally nothing for 12 months will be worse off than the group being fed the experimental diet! You have to feed them something, and this would either be a ‘market leader’ diet or an equivalent diet without the one variable you are trying to test (protein level, specific ingredient, etc).

Testing medicines, especially in humans, can get tricky again, especially if you’re testing a treatment for something critical where patients who are untreated are likely to die or suffer. So for these types of experiments the control group will be given the current standard treatment to see how that compares to the experimental one, instead of no treatment at all. You can’t ethically not treat your cancer/pain patients because they were randomly assigned to a control group.

And specifically in medical experimental trials, some will have a type of escape clause where if the tested medicine is proving amazingly, super effective, to the point where it would be definitely cruel to deny the control group that treatment, the control group is sort of disbanded and given the treatment anyway. This is one of the few situations where it’s acceptable to not have a control group. You had one, but they were doing so much worse than the experimental group that it wasn’t fair for them to continue.

Next time: Placebos, blinding and reporting bias

[drferox]

Placebos, Blinding and Double Blinding

Science Literacy Lesson #5

I hope you’re all enjoying these introductions to science literacy, especially if you’ve never come across these terms before or haven’t had a scientific education.I also hope they’re easy enough to understand, though the statistics one is a bit of a PITA.

This one is a lot more conceptual though, so no numbers as such to think about.

You’ve all heard of Placebos before, right?

Placebos are things (a treatment, diet, ritual, etc) that are inactive, meaning they should do basically nothing in regards to the effect you are measuring. Common placebos used in medicine are starch pills or saline injections. Any perceived improvement or result caused by these placebos is termed the Placebo Effect,

The Placebo effect is an important phenomenon in experiments because if you don’t account for it, you may well end up falsely concluding that sugar pills provide pain relief or that homeopathy works!

So, many types of experiments will include a placebo as a type of control group, and you might still see some results from that group, but your experimental group must demonstrate a significant difference from the placebo to be meaningful.

Now, what happens when the people in your experiment know they are taking the experimental treatment? If they know, that can produce a type of placebo effect too, because the subject expect results. People only need to believe they are taking something to report it has effects.

As a side note this can still occur in animal studies, as owners of those animals expect and report on a result subject to the placebo effect.

We attempt to counteract the placebo effect by Blinding.

Blinding means that experimental subjects have no idea whether they are in the placebo/control group, or the experimental group. They don’t get to know whether the pill they’re taking is a placebo or the real deal.

Double Blinding is even better, as it removes the placebo effect from the experimenters. In a double blinded study, the participants do not know whether they are getting the real treatment or the placebo, and neither does the experimenter. The only person who knows is a helpful pharmacist who has a list of participant numbers and who got what, revealed after the study data is collected.

Yes, the pesky placebo effect can influence the experimenters too, especially when the measures results are subjective (eg brighter, happier, shinier coat, clearer skin) and not numbers (eg weight, blood counts).

If you have not at least blinded, or preferably double blinded, your study, then the results are potentially very easy to dismiss with the placebo effect, especially if you haven’t demonstrated a massive difference.

[scripture-pictures]

[scripture-pictures]