Tuesday, August 23, 2011

Fun with Statistics: Straw Man, Hawthorne Effect & Power Debunk Cholangiogram Study

Major bile duct injuries can be prevented by implementation of routine intraoperative cholangiography [an x-ray of the bile ducts] say the authors of a paper published in the August 2011 issue of the Journal of the American College of Surgeons. The Dutch researchers established a policy of routine intraoperative cholangiography during laparoscopic cholecystectomy and looked at the incidence of bile duct injuries three years before and three years after it was instituted. Selective [at the discretion of the surgeon] intraoperative cholangiography was performed in 421 patients and routine intraoperative cholangiography was to have been in 435. Bile duct injury occurred in 1.9% of patients before the routine use of intraoperative cholangiography and in no patients after. The difference was statistically significant, p = 0.004.

Therefore, everyone having a laparoscopic cholecystectomy should undergo routine intraoperative cholangiography, right?

Not so fast. There are a few problems with the study. Let’s take the “Straw Man” issue first. A “Straw Man” is the establishing of a false premise and then defeating it with an argument. I have blogged about this before [here, here, and here]. The “Straw Man” in this case is the bile duct injury rate of 1.9% in the pre-routine intraoperative cholangiography cohort. Many  large series of laparoscopic cholecystectomies report rates of bile duct injury of well under 1%. An Egyptian study of 2,714 laparoscopic cholecystectomies found only 5 [0.18%] bile duct injuries. A Swiss study of 31,000 laparoscopic cholecystectomy patients noted a similar rate of bile duct injury of 0.3%. A third study reviewed 234,220 laparoscopic cholecystectomies done in Florida over a 10-year period and found that 0.25% resulted in a bile duct injury.

The next problem is called the “Hawthorne Effect,” which is the well-known finding that behavior improves when subjects know they are being watched. It is named for a factory near Chicago where several such experiments were done 80 years ago. Workers’ productivity increased no matter what changes were made in their environment. The surgeons in the routine intraoperative cholangiography study were given extra training in a skills laboratory and were aware that their performance was being monitored.

Despite the policy, only 59.8% of the patients in the routine intraoperative cholangiography group actually underwent routine intraoperative cholangiography. Even at the end of the three years, more than 23% of patients were still not undergoing routine intraoperative cholangiography. This suggests that the surgeons themselves were not totally convinced that the procedure was worth the extra time involved to complete the x-ray.

Another problem that is true of all so-called “before-and-after” studies is the fact that the “after” group has the benefit of the surgeons becoming more proficient simply because they have been performing the procedure longer.

The correct way to perform this investigation would be to randomize patients with one group having mandatory cholangiography and compare them to patients randomized to not have cholangiography. Such a study would be very difficult to do because the incidence of bile duct injury is so small. In order to achieve adequate statistical power, one would need more than 1000 patients in each group. 

Meanwhile, I will continue to perform selective intraoperative cholangiography.

4 comments:

Anonymous said...

Dear Scalpel: Sorry for reading the archives and commenting so long after the fact. Couldn't you agree that routine cholangiography, while not necessarily reducing the incidence of bile duct injury, might reduce the severity of the injury and the delay in diagnosis? I speak from my N of one experience.

Skeptical Scalpel said...

I can't say whether the severity would be less. No one has studied that. The x-ray must be done before the injury occurs or else it can't prevent an injury. Therefore, I don't see how it could reduce the delay in diagnosis.

Anonymous said...

In my case, I put the cholangiogram catheter in what I (falsely) believed to be the cystic duct. My anatomy was confusing -- there was this "duct of Luschka" at least that I can remember now 3 year later. I couldn't really get a good critical view. The cholangiogram did not show the cystic duct but did show filling of the common hepatic and common bile ducts. I had been flailing around for far too long at this point. In frustration I called my partner; he takes 3 seconds to say, Open. I really didn't want to open but took his advice (and was so glad in retrospect that I had only dinged the CBD and not severed it in half.)

Skeptical Scalpel said...

I have no problem with doing a cholangiogram in selected cases, especially if there is a question about the anatomy.

Your partner gave you good advice.

The paper reviewed in this post was advocating routine use of cholangiography for all cases, which I do not feel is necessary or useful.

This post might interest you if you have not already read it. http://skepticalscalpel.blogspot.com/2012/08/evidence-based-surgery-what-evidence.html

Post a Comment

Note: Only a member of this blog may post a comment.