Fun with Statistics: Straw Man, Hawthorne Effect & Power Debunk Cholangiogram Study

Major bile duct injuries can be prevented by implementation of routine intraoperative cholangiography [an x-ray of the bile ducts] say the authors of a paper published in the August 2011 issue of the Journal of the American College of Surgeons. The Dutch researchers established a policy of routine intraoperative cholangiography during laparoscopic cholecystectomy and looked at the incidence of bile duct injuries three years before and three years after it was instituted. Selective [at the discretion of the surgeon] intraoperative cholangiography was performed in 421 patients and routine intraoperative cholangiography was to have been in 435. Bile duct injury occurred in 1.9% of patients before the routine use of intraoperative cholangiography and in no patients after. The difference was statistically significant, p = 0.004.

Therefore, everyone having a laparoscopic cholecystectomy should undergo routine intraoperative cholangiography, right?

Not so fast. There are a few problems with the study. Let’s take the “Straw Man” issue first. A “Straw Man” is the establishing of a false premise and then defeating it with an argument. I have blogged about this before [here, here, and here]. The “Straw Man” in this case is the bile duct injury rate of 1.9% in the pre-routine intraoperative cholangiography cohort. Many  large series of laparoscopic cholecystectomies report rates of bile duct injury of well under 1%. An Egyptian study of 2,714 laparoscopic cholecystectomies found only 5 [0.18%] bile duct injuries. A Swiss study of 31,000 laparoscopic cholecystectomy patients noted a similar rate of bile duct injury of 0.3%. A third study reviewed 234,220 laparoscopic cholecystectomies done in Florida over a 10-year period and found that 0.25% resulted in a bile duct injury.

The next problem is called the “Hawthorne Effect,” which is the well-known finding that behavior improves when subjects know they are being watched. It is named for a factory near Chicago where several such experiments were done 80 years ago. Workers’ productivity increased no matter what changes were made in their environment. The surgeons in the routine intraoperative cholangiography study were given extra training in a skills laboratory and were aware that their performance was being monitored.

Despite the policy, only 59.8% of the patients in the routine intraoperative cholangiography group actually underwent routine intraoperative cholangiography. Even at the end of the three years, more than 23% of patients were still not undergoing routine intraoperative cholangiography. This suggests that the surgeons themselves were not totally convinced that the procedure was worth the extra time involved to complete the x-ray.

Another problem that is true of all so-called “before-and-after” studies is the fact that the “after” group has the benefit of the surgeons becoming more proficient simply because they have been performing the procedure longer.

The correct way to perform this investigation would be to randomize patients with one group having mandatory cholangiography and compare them to patients randomized to not have cholangiography. Such a study would be very difficult to do because the incidence of bile duct injury is so small. In order to achieve adequate statistical power, one would need more than 1000 patients in each group. 

Meanwhile, I will continue to perform selective intraoperative cholangiography.