I heard it frequently when the infamous Propublica Surgeon Scorecard first appeared three years ago. Back then I blogged about it saying “To me, bad data is worse than no data at all.”
A recent study in BJU International confirmed my thoughts about this type of publicly posted data and identified a previously unreported issue. The paper attempted to determine whether the public was able to accurately interpret statistics used in the Surgeon Scorecard. It turns out they were not very good at it.
Investigators from the Department of Urology at the University of Minnesota surveyed 343 people who attended the Minnesota State Fair in 2016. Those who took the survey had a median age of 48, were 60% female, 80% white, and 60% college educated. Their median annual income was $26,550 with an interquartile range of $22,882-$32,587.
The authors showed individuals the figure below on a tablet computer with the accompanying statement “This graph shows the individual surgeons’ complication rates after 28-35 cases. Surgeons A, B and C raw complication rates are A = 1/35 or 2.9%, B = 1/34 or 3.8% and C = 1/28 or 3.6%.”
In case you aren’t sure, understand the complication rates for these three surgeons are not significantly different due to the small numbers of cases and complications.
The most surprising finding of the study was although the surgeons’ complication rates were clearly stated above the figure, just 15.2% of the participants could correctly identify surgeon C’s complication rate. The participants thought the average complication rate for surgeon C was 25% (range 3.6% to 50%). Regarding surgeon B, they were better at estimating the complication rate, but still only 34.9% got it right.
The subjects were asked multiple-choice questions related to the surgeons’ complication rates. When asked to choose a surgeon for a hypothetical procedure, 192 (56%) picked surgeon A, 30 (8.7%) picked B, and 19 (5.5%) selected C; 102 (29.7%) said they didn’t have enough information to decide.
Here’s the new wrinkle on the potential harm of misinterpreting data. The subjects were then told that their insurance would only pay if they used surgeon C, and if they wanted to use one of the other surgeons, they would have to pay out of their own pockets. Almost two-thirds said they would pay an average of $5754 in order to have their surgery done by surgeon A or B.
Those willing to switch were significantly poorer, had a significantly higher incidence of a history of cancer, and misinterpreted the complication rates significantly more often.
What this means is that the people who could least afford to switch surgeons were the most likely to do so.
Bottom line: People may misinterpret published data on surgical complication rates which could result in financial harm to them.
5 comments:
I used to be a Social Science Analyst for HHS...and would tend to avoid using percentages with such small numbers. People do indeed mis-remember, round up, make unwieldy assumptions as you point out. And charts, especially given that so many people skim nowadays, can confuse the issues further. Actually, this chart to me is a bit overwrought given such small numbers. Interesting about the last point, though I wondered how level of education factored in, and how much that 'significant' experience with cancer did as well. And whether, if the stats 'really' said what the misinterpretors thought, if more affluent folks would also opt to pay so much out-of-pocket. Never assume, etc.!
But how then do patients find out about doctors who may not have as good success rate with a surgery. I'm very involved in the kidney cancer patient community and for a surgery like a partial nephrectomy the word is to look for a surgeon who does at least 100 of these per year. How do you get that info? Obviously ask, but how do you know that the surgeon isn't inflating their numbers? Rely on recommendations? That can work but could leave out many qualified doctors just because you don't talk to the right people. I managed to not have surgery with someone who clearly didn't have experience to do the surgery (and he was chair of dept and came highly recommended) I couldn't stand the guy. Good thing. Found a different surgeon who did have experience but probably didn't meet the 100 cases per year by a long shot!
Cassie, thanks for commenting. The paper is what it is. You can quibble about the percentages but most people didn't understand what they were looking at. I agree the chart is overwrought if not downright misleading. It is a screen shot from the Surgeon Scorecard.
Debra, I don't think many surgeons consistently do 100 partial nephrectomies per year. That is more than 1 every 3 weekdays, How to find an experienced and capable surgeon is a good question for which I do not have the answer.
There was drama locally over a surgeon/dept outcomes. Apparently this data is public and reporters caught wind of below average outcomes and published a bunch of stories on it, smearing the surgeon and dept. His stats showed below average outcomes but using a small sample size, basically the surgeon had poor outcomes on 4 more patients than expected, out of a sample size of 70. There was no analysis of case difficulty or extenuating circumstances outside the surgeon/hospitals control. Basically the dept had to shut down temporarily and insurance started playing hardball with reimbursements.
Can you send me a link to one of the stories? I am very interested to read about this surgeon's situation. If you don't want to post it here, please email me at SkepticalScalpel@hotmail.com. Thanks.
Post a Comment
Note: Only a member of this blog may post a comment.