It doesn't surprise me, but apparently a lot of people were taken aback by an Annals of Surgery paper published online last week stating just that.
The authors gave 767 surgeons four brief complex clinical scenarios and asked whether they would operate on each patient. The vignettes were purposely designed to not have "correct" answers.
In response to the question would you recommend an operation, the surgeons could choose one of the following responses: very likely, unlikely, neutral, likely, or very likely.
If you were in the emergency department with mesenteric ischemia, would you want a surgeon who responded "neutral"?
Why the authors selected five possible choices is puzzling. In real life when you are faced with a difficult decision in the middle of the night, you don't have five options. You have only two—operation or no operation.
More about that later.
The surgeons' estimation of the risks of each procedure varied widely, and most of them agreed about recommending surgery only for the patient with a small bowel obstruction.
In a Vox story about the paper, a Harvard health policy expert, Dr. Ashish Jha, said the findings were “disturbing."
I would call the findings "expected." These were difficult cases with no right answers.
A second paper in the same journal by the same investigators came up with somewhat different results. It randomized 779 surgeons into two different groups. One group had access to the American College of Surgeons operative risk calculator score and the other did not. For the same four clinical scenarios, surgeons who were given the risk calculation score estimated risks significantly closer to what the calculator’s values were—another non-surprise.
The difference between the estimated risks between the two groups was statistically significant but probably not clinically significant. For example, surgeons who used the risk calculator score estimated operative risk for the small bowel obstruction patient at 13.6% compared to 17.5% for the surgeons who didn't know risk calculator score, p < 0.001. Would the 3.9% difference between the two estimates really change a surgeon's mind about operating? I doubt it.
The effort to quantify risk so precisely may not only be wrong; it could be impossible.
Radiologist Saurabh Jha blogged about this two years ago. He wrote, “Numbers are continuous. Decision-making is dichotomous. One can be at 15.1%, 30.2% or 45.3% risk of sudden cardiac death. But one either receives an implantable cardioverter defibrillator (ICD) or does not. Not a 15.1% ICD.”
Jha concluded, “You can remove the burden of judgment from a physician but then you will no longer have a physician.”
As the authors of the two papers pointed out, among other points to be considered is that the risks of not operating are unknown. The topic has not been studied and probably never will be.
The most important finding of the second paper was that "averaged across the four vignettes, the two groups did not differ in their reported likelihood of recommending an operation, p = 0.76."
Since the first paper portrayed surgeons as wildly erratic at estimating risk, it of course received all the attention.