Evidence-Based Surgery: What Evidence?

Here is a study that illustrates everything that is wrong with the current status of evidence-based surgical practice.

Many of the standard procedures we perform in general surgery are based on observational studies, expert opinion or my favorite “That’s the way I was trained,” and not randomized controlled trials. Although some such research has been done, subjecting patients to sham operations raises ethical issues and would expose patients to anesthesia unnecessarily.

But some topics could be studied prospectively. A recent paper [Variation in the use of intraoperative cholangiography during cholecystectomy. Sheffield KM et al. J Am Coll Surg. 2012;214:668-79] highlights the problem of insufficient evidence in some areas of surgery.

A group from the University of Texas Medical Branch in Galveston looked at differences in the rates of operative cholangiography in their state. They analyzed data from 212 hospitals in which more than 160 cholecystectomies had been done including almost 177,000 such operations over the 8 years from 2001-2008. The surgeon was identifiable in 89% of cases.

Despite the known pitfalls of basing clinical research on administrative data, several interesting findings of this paper are worth discussing.

Operative cholangiograms were done on 44.6% of the total cohort. By hospital, the operative cholangiogram rates ranged from 6.0% to 98.2%. The breakdown by surgeon was limited to the 706 who had done at least 40 cases. The range of operative cholangiogram use by individual surgeons was 0% to 100% with a median of 39%. Medians were higher for patients with gallstone pancreatitis (69%) and lower for those with acute cholecystitis (25%).

Uninsured patients were only slightly less likely to have operative cholangiography than those who were insured. A puzzling finding was that of those patients who had both ERCP and operative cholangiography, 37% had undergone the ERCP before the cholecystectomy and operative cholangiogram. Why would an operative cholangiogram be necessary after an ERCP had already been done?

The authors found that the variation in rates of operative cholangiography was attributed much more strongly to the surgeon and the hospital rather than the indication for surgery. They concluded that the likelihood that a patient would undergo operative cholangiography depended on the hospital she arrived at and who the surgeons was.

The extent of the variation in the use of operative cholangiography could hardly be greater. It is difficult to believe that there is no agreement on the indications. I don’t think this is unique to Texas either.

The literature is conflicting. One can find multiple papers to support any position. Some claim that operative cholangiography helps prevent common bile duct injuries and reveals unsuspected stones. Others say false positive operative cholangiograms result in more procedures and that most asymptomatic stones discovered by cholangiography never cause symptoms. Surgeons who routinely perform operative cholangiography say it does not waste time while those who don’t do them say it does.

False negatives can occur. I have seen patients with negative operative cholangiograms readmitted within a few weeks because of symptomatic common bile duct stones.

I firmly reside on the low end of the operative cholangiography spectrum. I never perform one unless there is a specific indication as dictated by the liver function tests or a significant question involving the anatomy in the operating room.

A large, well-designed randomized controlled trial would help settle the issue, but it will probably never be done.

Who would sponsor such a study? The companies that manufacture the equipment for cholangiography certainly would have no incentive to fund it. Maybe the best we can hope for is a consensus statement from a group of expert surgeons.

Will it ever be forthcoming?

A version of this post appeared in General Surgery News in June of 2012.

Fun with Statistics: Straw Man, Hawthorne Effect & Power Debunk Cholangiogram Study

Major bile duct injuries can be prevented by implementation of routine intraoperative cholangiography [an x-ray of the bile ducts] say the authors of a paper published in the August 2011 issue of the Journal of the American College of Surgeons. The Dutch researchers established a policy of routine intraoperative cholangiography during laparoscopic cholecystectomy and looked at the incidence of bile duct injuries three years before and three years after it was instituted. Selective [at the discretion of the surgeon] intraoperative cholangiography was performed in 421 patients and routine intraoperative cholangiography was to have been in 435. Bile duct injury occurred in 1.9% of patients before the routine use of intraoperative cholangiography and in no patients after. The difference was statistically significant, p = 0.004.

Therefore, everyone having a laparoscopic cholecystectomy should undergo routine intraoperative cholangiography, right?

Not so fast. There are a few problems with the study. Let’s take the “Straw Man” issue first. A “Straw Man” is the establishing of a false premise and then defeating it with an argument. I have blogged about this before [here, here, and here]. The “Straw Man” in this case is the bile duct injury rate of 1.9% in the pre-routine intraoperative cholangiography cohort. Many  large series of laparoscopic cholecystectomies report rates of bile duct injury of well under 1%. An Egyptian study of 2,714 laparoscopic cholecystectomies found only 5 [0.18%] bile duct injuries. A Swiss study of 31,000 laparoscopic cholecystectomy patients noted a similar rate of bile duct injury of 0.3%. A third study reviewed 234,220 laparoscopic cholecystectomies done in Florida over a 10-year period and found that 0.25% resulted in a bile duct injury.

The next problem is called the “Hawthorne Effect,” which is the well-known finding that behavior improves when subjects know they are being watched. It is named for a factory near Chicago where several such experiments were done 80 years ago. Workers’ productivity increased no matter what changes were made in their environment. The surgeons in the routine intraoperative cholangiography study were given extra training in a skills laboratory and were aware that their performance was being monitored.

Despite the policy, only 59.8% of the patients in the routine intraoperative cholangiography group actually underwent routine intraoperative cholangiography. Even at the end of the three years, more than 23% of patients were still not undergoing routine intraoperative cholangiography. This suggests that the surgeons themselves were not totally convinced that the procedure was worth the extra time involved to complete the x-ray.

Another problem that is true of all so-called “before-and-after” studies is the fact that the “after” group has the benefit of the surgeons becoming more proficient simply because they have been performing the procedure longer.

The correct way to perform this investigation would be to randomize patients with one group having mandatory cholangiography and compare them to patients randomized to not have cholangiography. Such a study would be very difficult to do because the incidence of bile duct injury is so small. In order to achieve adequate statistical power, one would need more than 1000 patients in each group. 

Meanwhile, I will continue to perform selective intraoperative cholangiography.