Today I'm more than usually annoyed with a pop science article, so I'm going to talk about reading these sorts of articles, why you should always be skeptical of claims in them, and some of the ways you can tell the article's author didn't understand what they were reading and told you the wrong thing.
I clicked on an article in Eating Well about low bone density and dementia, because my mother has both. There's not a lot we can do for her now, but I am a curious person. I know Eating Well isn't great at science interpretation and communication, so I'm anticipating that I'm going to need to read the original study already, going in. (How do I know Eating Well isn't a great source usually? Well, I have read it before, and it has some really clear biases if you read a few articles that aren't science communication, and so you get to know a source over time like that. Regardless of how, I'm already suspicious they're not going to do a great job.)
The article is talking about research that shows low bone density may be predictive of dementia risk. It is written by a journalist and reviewed by a dietician. Now, I don't know what review the dietician did, but she did a bad job, and also, so did the journalist, because THE FIRST red flag that goes up is pretty quick: the math is very, very clearly wrong.
This says there are 3651 participants, and that over 11 years, 688 of them developed dementia. This is 18.8% and the article calls it 19%. That's fair! Not a red flag so far, just rounding. Then it says that of the 1211 people with lowest bone density at the start, 90 people (7.4%) developed dementia, and of the 1211 with highest bone density, only 57 (4.7%) did.
This IS a red flag. It's a GIANT red flag. This red flag can be seen from SPACE by anyone who knows how percentages work.
Here's how: You have 3651 people. 1211 of them are in the low bone density group, 1211 of them are in the high bone density group, leaving 1249 people. You have 688 total dementia cases, but your high and low groups account for only 147 of them, leaving 541 cases for that middle group. That's a LOT of cases. That middle tertile, just eyeballing it, has to have about 40% of its people with dementia -- that makes low bone density look like it predicts LOWER dementia risk relative to the middle group.
I can write out the equations for you two ways:
3651 - 1211 - 1211 = 1249
688 - 90 - 57 = 541
541/1249 = 0.433
0.433(100) = 43.3%
Because I am someone who does a fair amount of stats for a living, though, what I noticed was pretty much this equation:
0.074(1211) + x(1249) + 0.047(1211) = 0.19(3651)
and I knew immediately that x had to be MUCH bigger than it should, which indeed the math bears out:
x(1249) = 0.19(3651) - 0.074(1211) - 0.047(1211)
x(1249) = 694 - 90 - 57
x = 547/1249 = 0.438
0.438(100) = 43.8%
That 694 is because the authors rounded 18.8 to 19 earlier, not because I can't math. So, due to rounding, you get slightly different answers -- but BOTH of them point to something SERIOUSLY WRONG with the reporting. What is actually going on in that middle tertile? Where do these numbers come from?
Well, lucky us, they mention the name of an author, a journal, and a date. Always be wary of pop sci articles that don't give you a way to track down the original, but giving you that way to track things down doesn't mean they aren't still doing a crummy job with their reporting, as we see here.
The original paper is Association of Bone Mineral Density and Dementia: The Rotterdam Study, published March 2023 in Neurology. This is a pretty technical article with a fair amount of math and things in parens etc. etc. and tables and lots of measurements. The table captions are often not the greatest, which makes it a bit harder to read and interpret. For example, in Table 1, items are listed as number(number)
and this can be any of:
count (percent) -- this one's usually labeled in the table itself
mean (standard deviation)
median (interquartile range) -- these last two are NOT labeled in the table, so we don't know which set of numbers is which.
Great. Thanks guys. Assuming what's called a "normal distribution" mean (SD) and median (IQR) numbers will be similar, but they're not the same and I'm irritated they're conflated but OK. Soldiering on!
The original study looked at several different measures of bone density, and found only ONE of them to show predictive ability for dementia: the density of the femoral neck. This means that for their article, Eating Well should have looked at the results for femoral neck bone density, which we find in Table 2:
You have the actual numbers for 5 years, 10 years, and study end, as well as the hazard risk (HR) for each bone density tertile, with the highest tertile set as the standard. Numbers in the HR column have 1 as a reference point -- lower than 1 is lower risk than the highest tertile, and higher than 1 is higher risk.
The first thing I noticed is that neither 57 nor 90 occur in the femoral neck section at ALL. Those numbers from the Eating Well article are just not there. I also notice that the other numbers don't align even one little bit -- the number of total cases of dementia is different, for example. I do notice that the column with the 10 year followup has numbers in it close to 57 and 90 (49, 67, 86, totaled to 202) and that the overall numbers for the total study are much higher -- 201, 236, 229. Interesting.
At this point, I just straight-up search the paper for "90", and I find it in Table 2....in the total bone density section, which the paper's authors have said is NOT the section that showed possible predictive results. I search for "57", and also find that in total bone density, and also....wow the EW author straight up failed to read. This is actually worse than I thought.
Read across, these are the 5 year followup numbers (first 2 columns - count and HR), 10 year (middle 2 columns), and total followup numbers (last 2 columns).
We see our friends 57 and 90 in the 10 year columns. 90 is, as described in the EW article, in the lowest bone density tertile, but 57 is NOT in the highest bone density tertile. It's in the middle tertile. The actual number for the highest tertile is 68. Additionally, the total cases for 10 years is nowhere near that 688 number -- it's 215. We only get total case numbers close to 688 when we look at the study end numbers: it's 686, in this particular group. If we look at the study end case numbers for highest, middle, and lowest tertiles, we see WHY this particular measure can't be used to predict anything: they are 227 (highest), 227 (middle), and 232 (lowest) -- not significantly different from each other.
We can also see here that this group of people -- people who had total bone density measurements -- is not 3651, but 3633, which is listed across the bottom row. The overall STUDY had 3651, but not all of them had total bone density recorded.
Now we know that the author of the EW article did all of the following:
read the wrong part of Table 2
mixed up middle and high tertile results
reported 10 year results mixed with total followup results (this resulted in the weird math that alerted me something was very very wrong in the first place).
and the person who was supposed to review the article didn't have even the basic math skills to catch the problem -- which she absolutely should have, as a registered dietician. For giggles, I looked up program requirements for a BS in Dietetics. Programs require things like statistics and precalc -- not math heavy, but the math that alerted me to this problem is VERY basic statistical knowledge, like the kind they teach in 6th grade level statistics, which I know because it was literally in my 6th grader's curriculum this past school year. So a registered dietician DEFINITELY had enough math to catch this problem, and should have, and Eating Well should be ashamed of itself.
SO. What can we learn from this?
Well, science communication is a skill set. Some people have worked very hard to develop that skill set and are excellent at it -- but lots of people do not have it, and even those who do can make mistakes. Many, many pop sci articles are not written by trained science communicators, or people with any education in how to read scientific articles, or people with good reading comprehension, even. It's very common for pop sci articles to have these sorts of errors in them. Therefore:
Always read pop sci articles with a skeptical eye. Ask yourself:
Do these numbers line up? Usually the math in pop sci articles is not very complex -- you can often do some basic arithmetic to make sure it even makes sense, as was the case here.
Does one part of the article seem to contradict another part of the article?
Do I feel confused about what exactly I'm being told? What's not clear about it?
Am I being told about HOW something works or WHY it works or both? Are those two things being conflated somehow?
Is there a link or way to find the original research? If not, my advice is to throw the whole article away. If yes, you can go check it out -- often just looking at the abstract or results section will be enough, and abstracts usually aren't paywalled even if the rest of the article is. You would be surprised how many times the abstract says "we found X" and the pop sci article says "the researchers found Y".
Could I explain this article to someone and have it make sense? If not, why not?
Is the article confusing correlation (these things happen together) with causation (one of these things causes the other)?
Pop sci articles, like other journalistic articles, are extremely subject to bias issues from the publication they're in. A lot of people tend to read pop sci articles as neutral, factual reporting, but they aren't! I mentioned EW's biases earlier -- the one I think is most relevant to how their article is written is a pervasive belief that if you just eat the right things in the right amounts you will be thin and healthy and stave off all kinds of problems. They close their article by mentioning that, although the study's authors are clear that this connection is unlikely to be causative, and that risk factors for low bone density and dementia have substantial overlap, readers should act like it might be causitive with diet and exercise choices that promote bone health. They were so excited to get to their point about fixing your diet that they didn't pay attention to the actual science they were reporting on. (Sidenote: actual scientific journal articles are supposed to be neutral, factual reporting. They also aren't actually that, but there are some measures in place around this to try to prevent the worst effects of bias.)
It's worth brushing up some basic math skills. You don't need to know a lot! Very basic information will help you better understand a lot of articles -- both ones that are accurate and well-written, and ones that are shoddy and should not have been published. I really like Larry Gonick's The Cartoon Guide to Statistics but if your grasp of percentages is shaky, it will be too advanced. A good option might be something like The I Hate Mathematics! Book, which is pretty old but really accessible, but there's probably some newer great ones out there that I just don't know about.
