Here
are the first three paragraphs of a story from the medical news site, MedPage
Today.
"ORLANDO
– Patients on kidney dialysis who are infected with Clostridium difficile
appeared to have a greater risk of infection relapse and also appeared to have
a higher all-cause mortality that patients who do not have kidney disease,
researchers said here.
"Mortality
related to C. difficile infection was 3.8% among the 104 patients with
end-stage renal disease (ESRD) and 1.46% among 300 controls without ESRD, said
Massini Merzkani, MD, resident in internal medicine at the Albert Einstein
School of Medicine's Jacobi Medical Center in Bronx, N.Y. (No data as to
significance were presented.)
"In
his poster presentation at the National Kidney Foundation 2013 Spring Clinical
Meetings, Merzkani told MedPage Today that the relapse rate in severe C.
difficile infection was 34.7% in the controls and 45.2% in the patients
with ESRD. (No data as to significance were presented.)"
Does it make you nervous that "No data as to significance were
presented"?
It should.
The authors didn't analyze the data for statistical
significance. Would there be any way to do it yourself?
Yes, if you knew which statistical test to use.
Should a science reporter know something about statistics?
Yes, and the story was reviewed by an emeritus professor of
medicine at an Ivy League medical school who should have known too.
In addition, there is a rather interesting math error. The
mortality rate of 1.46% for the 300 controls doesn't compute. [300 x 0.0146 =
4.38] Unless 4.38 people died, the figure must be wrong.
Since both the mortality and relapse rates are categorical
(yes or no) variables, the correct statistical test to use is Fisher's exact
test.
The p value for mortality is 0.21 and for relapse is 0.061.
Neither difference is statistically significant which means that based on this
study, one cannot say that "C. diff is dangerous in ESRD."
You might point out that a p of 0.061 is pretty close to the
magical value of 0.05. That is true, but there is another major flaw in the
study. The article says the ESRD patients "were compared with patients
without chronic kidney disease who were admitted with C. difficile infection
during the same time period. The researchers calculated that randomly selecting
300 of the 2,400 control patients would produce a valid comparison of
outcomes."
Despite that "calculation," the comparison is
invalid. One cannot simply compare ESRD patients to random patients. They would
at least need to be matched for age, sex, co-morbidities other than ESRD and
perhaps other variables to eliminate confounding.
It is possible that ESRD patients will have worse outcomes if
they contract C. diff colitis. But this study doesn't prove that, and the story
is misleading.
It's 2013. I agree with The Guardian's Observer column which
says that Nate Silver's accurate predictions highlight "the importance of
statistical literacy in our data-heavy age."
11 comments:
Having ESRD and being on dialysis is often not compatible with having robust health. Neither is having a bad C.diff infection. The thesis of this "research" is, in my mind, akin to saying: Patients with congestive heart failure do worse when in septic shock. Was there ever actually a scientific question in debate here? Even if there was, answers to these simple 'association' questions are best done in the large patient care databases out there (e.g. NSQIP), rather than small single-institution reviews like this.
There needs to be academic reform and housekeeping in our medical community. There are too many journals and too many meetings that feature posters and abstracts. All this does is pollute the airwaves for physicians already struggling to keep up with medical science, and it allows the blatantly inadequate scientific articles you frequently highlight to get published/promoted without merit. I get that students/residents/fellows benefit from these outlets to further their careers, but if scientific integrity is lacking isn't it ultimately bad for everyone?
Josh, well said. Thanks.
I agree. If you search my blog for the term "journal," you will find some posts about the excess of journals. I have also written about the absurdity of some societies that accept every abstract, no matter how bad, as a poster.
By the way, the poster I wrote about above has very little relationship to the abstract that was submitted. You can view the submitted abstract here--http://ww3.aievolution.com/nkf1301/index.cfm?do=abs.viewAbs&abs=1481
[sarcasm] I wonder if elderly people with COPD have a higher mortality from community-acquired pneumonia than non-age-matched healthy controls... I think I'll look at some records and make a poster [/sarcasm]
Josh and Neuro beat me to the punch...as I like to say in the kitchen, 'everything tastes better with butter', and every disease is worse with ESRD added in.
Scalpel, is this a failure on the part of academia to allow the proliferation of weak studies? I remember journal club in residency, and being told to read journals beyond my training, as textbooks would be outdated by the time they were published. Now even journals are suspect. It's kind of like watching/reading the news...you have to look at the source with a skeptical eye (intended choice of words there).
Neuro, I think you're on to something. I'll alert the Nobel Prize Committee.
Artiger, It is a complete mess and what's worse is that we now have the Internet to disseminate this stuff quicker.
Isn't it nice that students get to make a poster of their learning?
I've only been a RA on one major research project (a maternal-infant study-maternal mental health r/t baby's development) so I don't have a lot of experience & knowledge about this stuff but apparently I'm thinking like Neuro. It seems logical that someone with end stage anything would have a higher risk factor for any relapse if they are infected with anything, correct?
I do wonder though, how can they relapse if they are infected? Don't you have to be clear of the infection first in order to relapse? So, if they are infected then they still have the nasty little critters in their system causing further damage to a damaged system so it really isn't a relapse right? Just a continuation of the downward progression of the end stage process.
This is a big issue and the old marketing technique of "throw numbers at them" still exists. You are correct in what is put out there today and it does need to be verified. There's an item called a "P" value and just the other day one of the executives from the FDA made a comment stating he does not rely on these only. It makes you crazy. I have had some conversations with the NISS (national institute of statistical science) and they get studies too that they can't replicate as well, frustrating as today more than ever you need to verify as there's all kinds of models and studies out there and which one has accuracy tied to it?
They build what is called a "model" to use for research and if it can't be verified sometimes there's more than a million models that can be used so in other words it leaves you to guess.
This a good video presentation brought down to layman's terms to where you can hear a scientific mathematician debunk a lot of this, Everything is Context at the link. I keep it on my footer area content so others can see it, and Charlie is a bit entertaining too, so it's not boring but he explains how much of this gets taken out of context and not only in studies but in many other areas, things are being take out of context, and most of time, making money is the root of this.
http://ducknetweb.blogspot.com/2012/01/context-is-everythingmore-about-dark.html
Professor Siefe also wrote the book called "Proofines the Dark Arts of Mathematical Deception" and I recommend reading it too if you want to dig deeper because there's a lot of it out there and it helps you understand how it gets started and the money motivating factors. It's a shame we have to decipher and find out what is good and what is not today and we are flooded with so much of this
Libby, it's true that patients with more co-morbidities usually have more trouble if they get sick.
I think by relapse, they may mean recurrence of symptoms. Patients usually aren't retested for C. diff if they feel better with treatment.
Quack, thanks for commenting. I've seen the video before. It is very interesting and entertaining.
I agree that there was probably some shoddy stats work going on here. However, if you wanted to give the benefit of the doubt, the mortality calculations could have been actuarial (eg Kaplan-Meier), potentially leading to mortality percentages as the authors report (that is, you couldn't just say 1.46%x300=4.38; you'd have to see the curves, and at what point in time they're reporting the instantaneous survival).
I doubt that very much. Patients were not followed over time. Since no significance stats were presented, it is very unlikely that they did something as sophisticated as actuarial survival.
Post a Comment
Note: Only a member of this blog may post a comment.