Friday, July 23, 2010

Bad News for Devotees of Process-Oriented Quality Assurance (Part 1)

The Surgical Care Improvement Project (SCIP) and its antecedent, the Surgical Infection Prevention project, have been around for several years. In short, these consist of several rules issued by various self-appointed agencies with important-sounding names and the Centers for Medicare and Medicaid Services (CMS), a federal agency. The main rules are (1) administer the correct prophylactic antibiotic before surgery, (2) give the antibiotic within one hour before the skin is incised and (3) discontinue the antibiotic within 24 hours of the end of the operation. The stated goal of these initiatives was to reduce the rate of surgical wound infections by 25% by the year 2010. It didn’t happen.

Surgeons and administrators have been obsessively following the rules and documenting their activities with squads of internal auditors (thought police). There are numerous papers showing a remarkable increase in the levels of compliance over the years to well over 90% in many hospitals. Compliance data have even been posted on line so that patients can compare institutions. CMS is thinking of linking hospital reimbursements to SCIP compliance rates. Unfortunately, a recent large study in the Journal of the American Medical Association (JAMA) has shown that SCIP has not only not decreased the rate of wound infections by 25%, it actually has had NO IMPACT at all on the infection rate.

So what happened? Why didn’t the rules work? They were based on some sound research. There are several theories. In order to comply with the “within one hour” rule, antibiotics are being given in the operating room and on many occasions, have not been completely infused as of the incision time. Thus, they will not have arrived at the wound in time to prevent the infection from occurring. Mary Hawn, MD, MPH, a surgeon and author of an editorial that accompanied the JAMA article, suggested that perhaps prophylactic antibiotics, which had been given for many years before the advent of SCIP, have already reduced the rate of infection as much as possible, and tweaking the timing may not make that much difference. She also pointed out that there are many other variables that influence the infection rate, such as the surgeon, the condition of the patient and type and duration of the procedure.

I asked Dr Hawn if SCIP should be changed or abandoned. She said, “SCIP is likely too narrow to have a meaningful effect on surgical outcomes. One response would be to add significantly more measures, but at that added burden one wonders if we really shouldn’t collect what we all care about – outcomes.” Of course, one reason that process metrics* are so popular is that processes are much easier to define and measure than outcomes. But would you as a patient rather choose a hospital that has a high rate of compliance with SCIP or a very low wound infection rate?

What we have here is the inevitable disconnect between process (the rules) and outcome (the infection rate). It’s not the first time, nor will it be the last. There will be more to come on the topic of process vs. outcome in future blogs.

*Metric: A metric is a measure for quantitatively assessing, controlling or selecting a person, process, event, or institution, along with the procedures to carry out measurements and the procedures for the interpretation of the assessment in the light of previous or comparable assessments. (Author’ note: Even the definition of a metric is convoluted. I promise I will not use the word “metric” again.)


Anonymous said...

Thanks for the definition of metric. Unfortunately I fell asleep while reading it. But I did like the definition given by your unnamed anesthesiologist. This looks like it will be a fun blog to read, Jim. Keep it up!

Kathy Kastner said...

Oh how I relish and appreciate looking beyond the (apparent) success of the process (metrics. ok. I'll never use that word again) to the actual impact (or non) on outcomes. Thank you for putting a seemingly impressive statistic into context that includes considering complex variables.
And.. I seem unable to resist taking the opportunity of your insights to open this can of worms: drug advertising. DTC. With all the hoohah
about its impact on increased use of Rx, what about assessing the (possible?) correlative impact on improved outcomes/management of conditions. OK. I've said it. Let the scathing comments begin.

Skeptical Scalpel said...

I am planning a post on that very subject. I highly recommend the book "Our Daily Meds" by Melody Petersen. I can't insert a direct link here but you can paste this url ( in your browser.

Kathy Kastner said...

Thnx 4 ny times link which, I am guessing, also says where your post'll be coming from (and so may take up your challenge to challenge;). I, in turn, am planning a post with my views on lifestyle vs life-saving drugs (which I'll welcome being challenged). Meanwhile, will be interested (should you have time/ inclination) in your reaction to my Rx experience

IVF-MD said...

Imagine two different scenarios, one where surgeons and hospital staff are always diligently exploring innovative ways to reduce infections and to improve patient outcomes and customer satisfaction vs one where surgeons and hospital staff brainlessly scramble to comply with arbitrary checklists. The first is what you get in a system where patients control the choice of what doctors and hospitals to give their healthcare dollars to. The second is what you get when a central planning committee of bureaucrats doles out the funds depending on how compliant the medical team is. The first is what we want. The second is what we got. How about we all work together to change things for the better? By the way, great blog!

BobbyG said...

"Of course, one reason that process metrics are so popular is that processes are much easier to define and measure than outcomes."

Touche. That's going in my quotes column on my REC blog

Skeptical Scalpel said...


George Gasman said...

(grumpy old anesthesiologist here):

Full of "evidence-based" medicine - and calling a lot of it what it is: a pile of BS.

Remember when we used Betadine?

"Did your hospital, like so many, abruptly switch from povidone-iodine antiseptic solution to ChloraPrep® for cleaning a patient’s skin before surgery? If so, I’m sure the staff was told that ChloraPrep would be more effective and cheaper. No doubt, they were also warned of the extra precautions that must be taken with ChloraPrep to prevent operating room fires, since ChloraPrep contains highly flammable 70% isopropyl alcohol in addition to chlorhexidine. Even the fire risk apparently wasn’t enough to make hospitals think twice before switching antiseptics.

You (and your hospital’s staff) may not have heard this news. The US Department of Justice (DOJ) announced last month that CareFusion Corp. would pay the government a $40.1 million settlement to resolve allegations that the company violated the False Claims Act by paying kickbacks to boost sales of ChloraPrep and promoting it for uses that aren’t FDA-approved.

Who received kickbacks? According to the DOJ’s press release, the complaint alleged that “CareFusion paid $11.6 million in kickbacks to Dr. Charles Denham while Denham served as the co-chair of the Safe Practices Committee at the National Quality Forum, a non-profit organization that reviews, endorses, and recommends standardized health care performance measures and practices.” Another physician with close ties to CareFusion, Dr. Rabih Darouiche, was the lead investigator on a 2010 NEJM article which concluded (not surprisingly) that Chloraprep was “significantly more protective” than povidone-iodine against surgical site infections.

The Leapfrog Group, launched by the Business Roundtable in 2000, claims that its hospital survey is “the gold standard for comparing hospitals’ performance on the national standards of safety, quality, and efficiency.” On January 30, Leapfrog announced that it accepted the resignation of Dr. Denham, who had served as chair of Leapfrog’s Safe Practices Committee since 2006, amid concerns that Dr. Denham had failed to reveal his “potentially compromising relationship with CareFusion.” At the same time, Leapfrog said it would undertake “a thorough scientific review of its full slate of endorsed safe practices."

Skeptical Scalpel said...

Right on, George. That Denham scandal was all over Twitter. There's another one too.

Christine Cassel, former president of the American Board of Internal Medicine and now President of the National Quality Forum (at a salary of $550K/year) has been charged with conflict of interest.

Post a Comment

Note: Only a member of this blog may post a comment.