Process vs. Outcomes Part II: New Head of Medicare and Medicaid

The New York Times reports that Donald Berwick, President Obama’s new appointee as head of Medicare and Medicaid, is coming under fire for some of his views such as his professed love of the British healthcare system. Something else caught my eye in that article. It says, “He has urged hospitals to emulate Toyota’s emphasis on efficiency and quality control…”

Most people who work in hospitals are familiar with the terms “Six Sigma” and “Lean.” Lean (L) was developed by Toyota and is a program of cost and waste reduction while maintaining quality. According to the folks at Motorola University, Six Sigma is a “metric*, a methodology and a management system.” Specifically, Six Sigma (SS) refers to six standard deviations from the mean or in the case of manufacturing, an attempt to achieve fewer than 3.4 defects for every one million opportunities.

Since 1999, these two tools have been adopted by many healthcare organizations. But despite more than 10 years’ experience, evidence of improved outcomes based on re-engineering of processes using SS/L is lacking according to a recent review of the healthcare literature on the subject.

The authors looked at 177 medical articles on SS/L published in the last 10 years. Only 34 papers reported any outcome data at all and of these, two-thirds did not provide any statistical analyses to justify their results. The methods used in these reports were judged to be of low scientific quality. Only three papers focused on clinical topics.

I asked the lead author of the review, Jami DelliFraine, whether she had expected to find favorable results in the literature on SS/L. She said, “I suspected that I would find a positive association between better processes/outcomes and the use of SS/L.” Thus, an author with a predisposition in favor of SS/L reported negative results in her review of the subject. I also questioned her about the cost of training personnel and she answered, “It is very expensive to implement these programs and train people for these programs.”

The proponents of SS/L must respond to the challenge and in the era of evidence-based medicine, produce some proof that the investment in time, money and personnel in SS/L will yield significant improvements in quality.

Finally, it is hard to resist a jab at SS/L via the Lean pioneers and Dr. Berwick’s example of what medicine should aspire to, Toyota, which has recalled some 9 million vehicles in the last 10 months. Using SS and taking the position that a defective vehicle represents one opportunity, Toyota would have to manufacture 3.4 billion consecutive flawless vehicles to achieve a rate of 3.4 defects per million. Guddorakku (Good luck).

*A metric is any number that you can put in a box in a computerized nursing record, and in turn be chewed over by a second layer of bureaucrats, resulting in a another number used to beat physicians over the head.

Shortage of Primary Care Physicians Persists: Causes and Solutions

For at least the last 20 years, graduates of U.S. medical schools have resisted pleas from organized and disorganized medicine to become primary care physicians (PCPs). Since there is already a severe shortage of PCPs, pundits are wondering who is going to take care of the hordes of newly insured by 2014. Many have speculated about the possible reasons for this dilemma such as the relatively paltry earning potential of PCPs, the amount of debt incurred by graduates of medical schools, the perceived lack of prestige of a PCP career etc.

I have some theories of my own. One, primary care is boring. It has been estimated that 90% of patients appearing in PCP offices have no treatable illnesses. This leads to another issue which is that a physician assistant or nurse practitioner can treat most of these patients, often without input from a physician. PCPs function as triage officers. If an interesting case should somehow happen along, the PCP refers the patient to a specialist who deals with the problem. Since the advent of hospitalists (physicians who restrict their practices to hospitalized patients only), PCPs are never seen in hospitals which almost guarantees that they will not be involved with anything interesting.

What is the solution? Bear with me. I will make a point eventually. About 15 years ago, medical schools in the New York City area were scrambling to climb aboard the family practice bandwagon. (Grant money was available for schools to establish departments of family practice.) This was a real problem for the schools since there were about as many family practitioners in metropolitan New York as there were blacksmiths. One school managed to set up a family practice department with a chairman who practiced in a town about 50 miles north of the city. Students were offered tuition forgiveness for the fourth year of medical school if they promised to do a family practice residency after graduation. Of some 12 initial enrollees in the program, a grand total of one ended up in family practice, proving one couldn’t even bribe students to become PCPs. I recall asking a few students why they thought the program did not work. The answer was that the new rotation in family practice was too realistic. It was as boring as actually being a family practitioner.

The solution to recruiting more students into family practice is to replicate the situation that exists in specialties the medical students highly desire like emergency medicine, anesthesiology and dermatology [the most competitive residency training program in all of medicine (see page 11)]. Most schools offer very little or no exposure to these disciplines in their curricula. Medical schools should disband their family practice departments. Thus, a mystique would be created and the students would be seduced. I believe this would work. If needed, I am available to chair a task force or blue ribbon panel on this issue.

More on SCIP

I guess the process vs. outcome issue is not limited to medicine. (See original blog.) Here is a word from a friend highly placed in the world of finance.

"Like your blog. in reading the part regarding the focus on process rather than outcomes, I realized I could probably replace the surgical infection example with one from Banking/Finance and the conclusions would be pretty much the same. Administratively, there has been a huge shift in Finance toward tracking things because they are measurable, rather than meaningful. While this allows administrators to show they are being "proactive", it only serves to lower their personal risk rather than meaningfully influence outcomes. The recently passed Financial Overhaul Bill is likely to be a poster child for this."

Updates on SCIP

Here is an interesting comment on process (See original blog.) from an anesthesiologist:

“We are forced to use that metric where I work. My definition of a metric is any number that you can put in a box in a computerized nursing record, and in turn be chewed over by a second layer of bureaucrats, resulting in a another number used to beat physicians over the head.

"If an anesthesiologist fails to record the antibiotic on three occasions at (hospital name deleted for obvious reasons), he/she is required to make a video explaining him(her)self … expressing remorse. I am not making this up.”

And I thought I was being funny when I alluded to the thought police.

Two readers suggested that I should have posted links to the articles mentioned in the original blog. You must be a subscriber to JAMA to read the full text articles. But here are the links to the abstract of the research paper and an excerpt from the editorial.

Bad News for Devotees of Process-Oriented Quality Assurance (Part 1)

The Surgical Care Improvement Project (SCIP) and its antecedent, the Surgical Infection Prevention project, have been around for several years. In short, these consist of several rules issued by various self-appointed agencies with important-sounding names and the Centers for Medicare and Medicaid Services (CMS), a federal agency. The main rules are (1) administer the correct prophylactic antibiotic before surgery, (2) give the antibiotic within one hour before the skin is incised and (3) discontinue the antibiotic within 24 hours of the end of the operation. The stated goal of these initiatives was to reduce the rate of surgical wound infections by 25% by the year 2010. It didn’t happen.

Surgeons and administrators have been obsessively following the rules and documenting their activities with squads of internal auditors (thought police). There are numerous papers showing a remarkable increase in the levels of compliance over the years to well over 90% in many hospitals. Compliance data have even been posted on line so that patients can compare institutions. CMS is thinking of linking hospital reimbursements to SCIP compliance rates. Unfortunately, a recent large study in the Journal of the American Medical Association (JAMA) has shown that SCIP has not only not decreased the rate of wound infections by 25%, it actually has had NO IMPACT at all on the infection rate.

So what happened? Why didn’t the rules work? They were based on some sound research. There are several theories. In order to comply with the “within one hour” rule, antibiotics are being given in the operating room and on many occasions, have not been completely infused as of the incision time. Thus, they will not have arrived at the wound in time to prevent the infection from occurring. Mary Hawn, MD, MPH, a surgeon and author of an editorial that accompanied the JAMA article, suggested that perhaps prophylactic antibiotics, which had been given for many years before the advent of SCIP, have already reduced the rate of infection as much as possible, and tweaking the timing may not make that much difference. She also pointed out that there are many other variables that influence the infection rate, such as the surgeon, the condition of the patient and type and duration of the procedure.

I asked Dr Hawn if SCIP should be changed or abandoned. She said, “SCIP is likely too narrow to have a meaningful effect on surgical outcomes. One response would be to add significantly more measures, but at that added burden one wonders if we really shouldn’t collect what we all care about – outcomes.” Of course, one reason that process metrics* are so popular is that processes are much easier to define and measure than outcomes. But would you as a patient rather choose a hospital that has a high rate of compliance with SCIP or a very low wound infection rate?

What we have here is the inevitable disconnect between process (the rules) and outcome (the infection rate). It’s not the first time, nor will it be the last. There will be more to come on the topic of process vs. outcome in future blogs.

*Metric: A metric is a measure for quantitatively assessing, controlling or selecting a person, process, event, or institution, along with the procedures to carry out measurements and the procedures for the interpretation of the assessment in the light of previous or comparable assessments. (Author’ note: Even the definition of a metric is convoluted. I promise I will not use the word “metric” again.)