Monday, August 6, 2012

Reduced Rate of Infections in Colorectal Surgery: Real or Hawthorne Effect?


There was some buzz last week on medical news sites like MedPage Today, Fierce Healthcare and Science Codex, about a paper from Johns Hopkins describing how the institution reduced its surgical site infection (SSI) rate for colorectal surgery. A multidisciplinary project called the Comprehensive Unit-Based Safety Program (CUSP), involving 36 people including a “team coach” and a hospital executive, began when the SSI rate was 27.3%. After meetings and suggestions for change, they came up with six interventions that they felt would make a difference. They were

  • Standardization of skin preparation with chlorhexidine
  • Administration of preoperative chlorhexidine showers
  • Selective elimination of mechanical bowel preparation
  • Warming of patients in the preanesthesia area
  • Adoption of enhanced sterile techniques for skin and fascial closure
  • Addressing previously unrecognized lapses in antibiotic prophylaxis

In the year following the implementation of the changes, the SSI rate dropped to 18.2%.

Sounds great, doesn’t it? Let’s take a closer look at a few of the interventions.

The issue of chlorhexidine for skin preparation is not quite settled. One of this paper’s own references cites a study which says that povidone iodine use leads to significantly fewer SSIs than did chlorhexidine.

The bowel prep intervention is very confusing. To quote the paper, “In December 2011, the CUSP group reviewed the literature again and decided to change from no mechanical bowel preparation to mechanical bowel preparation with oral antibiotics beginning in February 2012.” This is interesting since the paper was submitted to the journal in January of 2012.

Regarding enhanced sterile techniques for skin and fascial closure (which means they changed gloves and some instruments after the intestinal anastomosis was completed), there is no evidence that doing so prevents wound infections. The only reference they provided was to a book of standards from the Association of periOperative Registered Nurses (AORN), a notoriously non-evidence-based document. [See my previous blog on rules without foundations.]

The lapses in antibiotic prophylaxis involved patients who were allergic to penicillin, a condition that affects only about 10% of the population. There were 602 patients in the study of whom about 60 would probably have been allergic to penicillin. That’s hardly enough to make a big difference in the infection rate.

These policy changes reduced the infection rate of colon surgery from 27.3% to 18.2%. A quick search reveals that in 2010, the colon surgery SSI rate for all hospitals in the state of New York was 5.3%. That’s all hospitals, not just university medical centers. The respected medical resource UpToDate also says the expected SSI rate for colon surgery is 5%.

Despite the well-known effectiveness of 36-person committees (and wouldn’t you have liked to have attended those meetings?), I think there is another explanation for these results. It’s called the Hawthorne effect, which is the name given to the fact that the behavior of experimental subjects changes if they know they are being watched. For example, if you tell the staff of an ICU that there will be changes to the hand washing protocol and compliance will be monitored, whatever the changes are, the rate of compliance with hand washing will increase.

In summary, the authors have turned an abysmal colon surgery SSI rate into a merely bad one by observing it. Medical news sites reported the findings without questioning any of the conclusions. One wonders why the paper is receiving any attention at all or even why it was published.

18 comments:

paleohappy said...

Now, what might have worked better is if they had a committee oversee the infection committee, then you'd get the Hawthorne effect squared. Or, better yet, a super committee to oversee the two sub-committees, then you'd get Hawthorne effect cubed. Keep going at that rate and pretty soon you'd get Federal-government-like results, which I'm sure would get plenty of well-deserved, and well-reported media coverage...and outstanding results in the OR, or course...or not.

P.S. Just found your blog. Love it.

Skeptical Scalpel said...

Well-said. Maybe a task force would have been better.

Dr Skeptic said...

Good post, and well dissected. This highlights one of the big problems in surgical research: so much of it relies of before-and-after studies that universally show improvement in the "after" group, and there is very little understanding of the inherent biases in these studies. Studies that test similar practice-change programs comparing one institution to another, or use randomisation, rarely show such improvements.

Anonymous said...

Dear Sceptical Scalpel, here you really hit the nail !

Quote:

"These policy changes reduced the infection rate of colon surgery from 27.3% to 18.2%. A quick search reveals that in 2010, the colon surgery SSI rate for all hospitals in the state of New York was 5.3%. That’s all hospitals, not just university medical centers. The respected medical resource UpToDate also says the expected SSI rate for colon surgery is 5%."

Skeptical Scalpel said...

Dr. Skeptic, I appreciate your remarks. You have made similar observations in your own blog, which is worth following at http://doctorskeptic.blogspot.com.au/.

Anon, thanks for the comment and for reading my blog.

Steve, Kel, Rob & Lara said...

Thanks for another insightful post. Wondering if you would mind sharing your thoughts on laparoscopic inguinal hernia repairs, and whether it is a worthwhile exercise? are there any real advantages over an open mesh repair or is it just a fashionable approach?

Skeptical Scalpel said...

There's a lot of literature to support laparoscopic inguinal hernia repair. I've seen a few disasters and heard of many more, but it seems that in the hands of an experienced surgeon, the results are OK. If I had an inguinal hernia, I would opt for an open mesh-plug repair.

Rob said...

Another great post. Here's a good resource about things that work and don't work related to reducing SSI: www.stopwoundinfection.com

Steve, Kel, Rob & Lara said...

Thanks!

Pat said...

Dear Skeptical Scalpel,

You make excellent points. I just beg to differ when it comes to chlorhexidine.

"The issue of chlorhexidine for skin preparation is not quite settled"
Your statement is valid. But most of the "issues" in medicine are "not settled" either.

In my opinion the current evidence suggests that alcoholic chlorhexidine is the best agent for skin prep, as per the RCTs below.
Moreover, chlorhexidine has been shown to be superior in other settings, such as the insertion of central venous catheters.

The study that suggested chlorhexidine to be inferior is seriously flawed. It was not randomized, the procedures performed were not comparable across different study periods, no details in regards to diagnosis of infections are provided (and it was funded by a company that does not sell chlorhexidine).

In the light of the currently available data, alcoholic chlorhexidine seems to be the best available agent for skin preparation.

(I'm a general surgery resident and don't have any financial interests in this topic)

Paocharoen V, Mingmalairak C, Apisarnthanarak A. Comparison of surgical wound infection after preoperative skin preparation with 4% chlorhexidine [correction of chlohexidine] and povidone iodine: a prospective randomized trial. Journal of the Medical Association of Thailand. 2009 Jul;92(7):898–902.
Darouiche RO, Wall MJ, Itani KMF, Otterson MF, Webb AL, Carrick MM, et al. Chlorhexidine-Alcohol versus Povidone-Iodine for Surgical-Site Antisepsis. The New England journal of medicine. 2010 Jan 7;362(1):18–26.

Skeptical Scalpel said...

Pat, thanks for the thoughtful comment. You make some good points. Maybe what I should have said was I can't believe the difference between povidone iodine and chlorhexidine is that great. Also, it is not stated whether some surgeons were using chlorhexidine prior to the CUSP project's inception.

Anonymous said...

wwound infections are a big topic at ouur hospital and Hospital system right now. Every article that I have seen in the last 2 years from all over the world has shown infection rate of 12-15%. Our hospital which has all of the general surgeons on Epic in the hospital and in the office has been showing an infection rate of 10-15%. When we started studying it closely we dropped it down to 8%. We have 100% capture of all wound infections due to the fact that all of our postoperative notes are inn the system and all of her cultures are sent to our main hospital. The other for hospitals in our system all have rates listed as 1-5%. Interestingly enough none of them have their postoperative/office notes in the Epic system and their system does not have as strong past postoperative surveillance as we do. Might take on this has been that if you do surveillance right he will see that you have a infection rate of about 10% whereas if you only only count infections that arise in the hospital you'll have an infecction rate of about 5%.

Skeptical Scalpel said...

That's an excellent point. Most hospitals rely on surgeons self-reporting of wound infections once the patient has left the hospital and many are missed.

Skeptical Scalpel said...

Addendum to my last comment.

The Johns Hopkins paper does not say how infections were tracked. However, after thinking about the issue, I would point out that most infections after colon surgery are discovered during the initial hospitalization. This is because the average hospital length of stay is 6 or 7 days postop.

Matthias Maiwald said...

It is correct that the question of chlorhexidine (CHG) for surgical skin preparation is not settled. We recently performed a systematic review (1) and found that clinical trial outcomes from chlorhexidine-alcohol are often incorrectly attributed to CHG alone in the literature on skin antisepsis. This makes CHG appear more efficacious in authors' perception than it is in reality. This phenomenon is happening on a very large scale, affecting primary clinical trials, systematic reviews, clinical practice recommendations and evidence-based guidelines. What is clear is that the combination of CHG and alcohol is superior to povidone-iodine (PVI) alone. But that is a comparison of two antiseptics against one. The question of CHG-alcohol versus PVI-alcohol for surgical skin preparation is clearly unresolved. In fact, there is no evidence to support the use of CHG alone for preparation of superficial skin in surgery, and CHG alone commonly fails US regulatory requirements for skin antiseptics. On microbiological grounds, alcohols have about 10 times greater immediate efficacy than CHG (2), while CHG has residual activity that alcohol does not have. For most operations up to a few hours, on microbiological grounds, the alcohol should be able to contribute more to the prevention of surgical site infections than the chlorhexidine. Alcohol is also microbiologically significantly more effective than PVI. On this basis, the results of the often-cited Darouiche et al. study (3) were predictable before the study even started: a comparison of two agents against one, where already one of the two was known to have significantly greater efficacy than the entire competitor arm. And, by the way, in response to the other comment, chlorhexidine alone has never reached statistical significance for the prevention of catheter-related bloodstream infections (CR-BSI) (1).

1. Maiwald M, Chan ESY (2012) The Forgotten Role of Alcohol: A Systematic Review and Meta-Analysis of the Clinical Efficacy and Perceived Role of Chlorhexidine in Skin Antisepsis. PLoS One 7(9): e44277. PMID:22984485.

2. Larson E (1988) Guideline for use of topical antimicrobial agents. Am J Infect Control 16: 253–266. PMID:2849888.

3. Darouiche RO, Wall MJ, Jr., Itani KM, Otterson MF, Webb AL, et al. (2010) Chlorhexidine-alcohol versus povidone-iodine for surgical-site antisepsis. N Engl J Med 362: 18-26. PMID:20054046.

Skeptical Scalpel said...

Dear Dr. Maiwald,

Thanks for the comment supporting my opinion. You added a great deal of evidence. I appreciate your taking the time and for reading my blog.

Matthias Maiwald said...

P.S. Based on the above-mentioned microbiological superiority of alcohol-containing skin antiseptics and the fact that two substances against one were tested, I am surprised that an ethics committee even approved the Darouiche et al. 2010 study.

Skeptical Scalpel said...

Dr. Maiwald, thanks again for commenting. As you know, some IRBs and ethics committees are more diligent than others.

Post a Comment

Note: Only a member of this blog may post a comment.