Reduced Rate of Infections in Colorectal Surgery: Real or Hawthorne Effect?

There was some buzz last week on medical news sites like MedPage Today, Fierce Healthcare and Science Codex, about a paper from Johns Hopkins describing how the institution reduced its surgical site infection (SSI) rate for colorectal surgery. A multidisciplinary project called the Comprehensive Unit-Based Safety Program (CUSP), involving 36 people including a “team coach” and a hospital executive, began when the SSI rate was 27.3%. After meetings and suggestions for change, they came up with six interventions that they felt would make a difference. They were

• Standardization of skin preparation with chlorhexidine
• Administration of preoperative chlorhexidine showers
• Selective elimination of mechanical bowel preparation
• Warming of patients in the preanesthesia area
• Adoption of enhanced sterile techniques for skin and fascial closure
• Addressing previously unrecognized lapses in antibiotic prophylaxis

In the year following the implementation of the changes, the SSI rate dropped to 18.2%.

Sounds great, doesn’t it? Let’s take a closer look at a few of the interventions.

The issue of chlorhexidine for skin preparation is not quite settled. One of this paper’s own references cites a study which says that povidone iodine use leads to significantly fewer SSIs than did chlorhexidine.

The bowel prep intervention is very confusing. To quote the paper, “In December 2011, the CUSP group reviewed the literature again and decided to change from no mechanical bowel preparation to mechanical bowel preparation with oral antibiotics beginning in February 2012.” This is interesting since the paper was submitted to the journal in January of 2012.

Regarding enhanced sterile techniques for skin and fascial closure (which means they changed gloves and some instruments after the intestinal anastomosis was completed), there is no evidence that doing so prevents wound infections. The only reference they provided was to a book of standards from the Association of periOperative Registered Nurses (AORN), a notoriously non-evidence-based document. [See my previous blog on rules without foundations.]

The lapses in antibiotic prophylaxis involved patients who were allergic to penicillin, a condition that affects only about 10% of the population. There were 602 patients in the study of whom about 60 would probably have been allergic to penicillin. That’s hardly enough to make a big difference in the infection rate.

These policy changes reduced the infection rate of colon surgery from 27.3% to 18.2%. A quick search reveals that in 2010, the colon surgery SSI rate for all hospitals in the state of New York was 5.3%. That’s all hospitals, not just university medical centers. The respected medical resource UpToDate also says the expected SSI rate for colon surgery is 5%.

Despite the well-known effectiveness of 36-person committees (and wouldn’t you have liked to have attended those meetings?), I think there is another explanation for these results. It’s called the Hawthorne effect, which is the name given to the fact that the behavior of experimental subjects changes if they know they are being watched. For example, if you tell the staff of an ICU that there will be changes to the hand washing protocol and compliance will be monitored, whatever the changes are, the rate of compliance with hand washing will increase.

In summary, the authors have turned an abysmal colon surgery SSI rate into a merely bad one by observing it. Medical news sites reported the findings without questioning any of the conclusions. One wonders why the paper is receiving any attention at all or even why it was published.