What does Jersey City have that New York City doesn’t? Two hospitals with Leapfrog Group “A” safety grades

Leapfrog just released its semi-annual hospital safety grades. Incongruities identified in my previous blog posts [here and here] appeared again.

New York City had no A grade hospitals, only four got Bs, and nationally known hospitals such as New York Presbyterian (Columbia and Cornell), New York University, and Mount Sinai received C grades.

Residents of New York City are in luck because several A-rated hospitals are located just across the Hudson River. Four of them—Jersey City’s CarePoint Health-Christ Hospital, Jersey City Medical Center, CarePoint Health-Bayonne Medical Center, and CarePoint Health-Hoboken University Medical Center—aren’t exactly household names, but they scored better on safety than their New York neighbors.

CarePoint has figured out how to achieve a top rating but can’t compare to the numbers of California Kaiser Foundation Hospitals scoring well on multiple Leapfrog evaluations. The current rankings show 19 of 26 Kaiser hospitals in California were A rated.

Lean methodology and patient safety

A recent story in U.S. News & World Report described how a Seattle hospital is taking a systems approach in improving healthcare quality and cutting costs. It said, "Virginia Mason Health System...has looked to adopt many of the much-admired and often-emulated business philosophies from Toyota."

The best-known of those philosophies is the so-called "lean methodology" which is based on eliminating waste and focusing on things that add value.

Attempts to incorporate lean into healthcare have met with varying degrees of success. I blogged about this six years ago and pointed out that a literature review done back then found "significant gaps in the [lean and six sigma] health care quality improvement literature and very weak evidence that [lean and six sigma] improve health care quality."

Randomized prospective trials of lean in medicine are lacking. A recent paper from the Journal of the American College of Radiology found only seven studies on the use of lean in radiology and they showed "high rates of systematic bias and imprecision." The authors concluded there was "a pressing need to conduct high quality studies in order to realize the true potential of these quality improvement methodologies [lean and six sigma] in healthcare and radiology."

In addition to the debatable evidence that lean actually works and the cost and time to develop and implement lean measures, the use of Toyota as a model for quality is also highly questionable.

In 2010, Toyota had recalled more than 9 million vehicles for various defects. Nothing has improved. So far this year Toyota has recalled over 11,654,000 vehicles. The problems included exploding airbags, brake failure, fuel tank defects, and minivan doors opening while cars were in motion.

Having adopted lean methodology in 2002, Virginia Mason is not really a new story. How is it doing?

About as well as Toyota.

In May of this year, the Joint Commission paid a surprise visit to Virginia Mason Medical Center and found 29 instances where the hospital was out of compliance with standards. The Seattle Times wrote that among the problems were not having an adequate infection prevention and control plan, failure to store medication safely, and failure to provide a "care, treatment, services and an environment that pose[d] no risk of an immediate threat to health or safety."

On September 17, Virginia Mason regained full Joint Commission accreditation status, and 6 weeks later the hospital announced that it received an "A" grade for patient safety from the Leapfrog Group.

A hospital that failed a Joint Commission site visit because of multiple safety issues gets an "A" for patient safety in the same year? I discussed problems with the Leapfrog patient safety rankings in a previous post.

And if lean works so well in healthcare, can anyone tell me how does a hospital that has been practicing lean methodology for 14 years achieve 29 Joint Commission citations?

Just in. My wife named a top orthopedist

A couple of weeks ago my wife received this in the mail.

Of course we are thrilled that she was finally recognized as one of the top orthopedic surgeons in our city.

Last year I blogged about a similar honor that I received from the International Association of Healthcare Professionals. That eminent organization had named me a top surgeon. I was a little skeptical because at the time, I had been retired for two years.

I am sure the vetting process for my wife’s inclusion was quite thorough. However, they may have been mistaken about some important criteria.

For one thing, she is not an orthopedic surgeon. In fact, she isn't even a doctor. She is a nurse and manages an office for a large group of orthopedists. She is pretty good at reading x-rays and is a whiz at organization, staff management, patient interactions, suture removal, and fitting crutches. Maybe that counts for something.

From the format of the letters and the mention of the "renowned publication, The Leading Physicians of the World," it looks like the International Association of Orthopedic Surgeons might be run by the same outfit as the International Association of Healthcare Professionals.

The website does not list the cost but here are the benefits of being selected: wall plaque of achievement, leading physician feature video, leading physician press release, leading physician specialist online exposure, physician feature website, and [my favorite] endorsement of credentials.

We look forward to the selection of our dog Bailey as perhaps one of the leading neurosurgeons of the world.

I think she qualifies because she has been dead for three years.

Why hospital rankings are bogus

At the end of 2015, The Leapfrog Group announced its annual list of America’s top hospitals for quality and safety; 98 hospitals receiving the honor.

Unlike some other hospital rating schemes, Leapfrog’s does not factor in reputation. You won’t find any of the usual suspects on Leapfrog’s list. Instead, Leapfrog uses surveys of hospitals and publicly available quality and safety data.

Leapfrog’s top 98 included 62 urban, 24 rural, and 12 children’s hospitals. Of the 86 urban and rural hospitals, only three were university hospitals—University of California Davis Medical Center, University of California Irvine Medical Center, and University of Tennessee Medical Center.

New York managed to place only one hospital on the Leapfrog list.

Other interesting anomalies are that for several states such as Connecticut, Indiana, and Maryland, no hospitals made the list, and of the 21 California hospitals that did, 17 are Kaiser-affiliated. Looks like Kaiser knows how to play the game.

The Surgeon Scorecard: My analysis

I've got nothing against ProPublica. If a valid way to rate surgeons is ever discovered, I would support it completely. However, ProPublica's Surgeon Scorecard is not the answer.

I keep hearing its defenders say, "Some data is better than no data at all." I disagree strongly with that. To me, bad data is worse than no data at all. People with much more statistical sophistication than I have pointed out the flaws in the scorecard.

Digression: Having written many posts about statistics, I can tell you that the mere mention of the word drives readers away about as fast as if you were to yell "Fire" in a crowded theater.

I want to focus on a different area. The scorecard has created a lot of chatter on Twitter, and just about everyone I know has blogged about it.

This reminds me of a couple of posts I wrote back in 2011. [Links here and here.] I pointed out that Twitter might not be as important as those of us who use it think it is.

While we were busy arguing about the merits of the scorecard on Twitter, I'm not so sure what the general public was doing.

For example, ProPublica says the Surgeon Scorecard has had over 1 million visitors since its launch. That sounds like a lot until you consider that the current population of the United States is estimated at 321 million. So 1 million people would be 0.3%. We do not know how many of those 1 million were unique visitors. It could be that many of them were doctors looking for their own statistics and bloggers looking for ideas.

That the public may not care was reinforced by a rather tepid response to the ProPublica AMA (Ask Me Anything) on Reddit today.

By 1:00 PM EDT, which was two hours into the AMA, there were 80 comments, 31 of which were by ProPublica staff or the spine surgeon who had consulted on the scorecard's methods.

Just to give you some perspective, an AMA last year by a guy with two penises drew 17,134 comments.

Because the demographic is skewed toward younger people, perhaps Reddit may not have been the right venue. Although Reddit boasts 169 million unique visitors per month, the most recent figures show that 33% of the Reddit users are mostly men between 18 and 49 years old. Those under 18 are not counted but represent "a substantial percentage of Reddit users."

My two favorite questions asked of ProPublica were "How can I tell if my doctor is capable of making an error?" and "Do you fix the leg which is broken completely?" [Did the question refer to a leg that was completely broken, or did it mean should the leg be completely fixed?]

What have we learned here? It's hard to say.

If you want to read a measured critique of the scorecard, go to Dr. John Mandrola's piece on Medscape.

Big data is not big enough

Today ProPublica released its “Surgeon Scorecard” touting it as the best way to pick the right surgeon.

It took me less than a minute to discover some interesting omissions from the application.

For laparoscopic cholecystectomy, the only general surgery procedure listed, the app omits approximately one-third of the hospitals in my state including two where I have practiced.

It looks like the problem is that using Medicare fee-for-service data does not yield enough surgeons performing 20 or more cases in some categories such as laparoscopic cholecystectomy for the five years included in the database.

At one of the biggest hospitals in my state, apparently only one surgeon performed 20 laparoscopic cholecystectomies on fee-for-service Medicare patients in the five years studied; 23 other surgeons were listed as having performed fewer than 20 laparoscopic cholecystectomies on patients in the target population. I don’t see how patients who want to use that hospital for their gallbladder surgery will benefit from the Surgeon Scorecard.

In general, the complication rate for laparoscopic cholecystectomy is low, but I think I understand why ProPublica chose that procedure to review. They needed to select a procedure that was done frequently enough to yield a sufficient number of cases for analysis. Unfortunately, because of the limitations of the Medicare fee-for-service data and the low complication rate of the procedure, the Surgeon Scorecard is useless for anyone looking to compare general surgeons.

Similar problems with the scorecard may be in play for prostate surgery. Again, the procedure was chosen because of its high frequency, but in quickly looking through some searches in that area, I note that a number of urologists I know also did not perform 20 cases on fee-for-service Medicare patients.

Perhaps the next iteration of the scorecard will utilize a data set that contains enough patient and surgeon records to make a meaningful comparison.

Until then, general surgeons can relax. They will not have to explain away their complications but will simply have to explain why they aren’t listed in the Surgeon Scorecard.

How to pick the leading physicians of the world

 

My "real life" self has been selected as a Top Surgeon in my city by the International Association of Healthcare Professionals (IAHCP). I will be spotlighted in the renowned publication “The Leading Physicians of the World.” Can you imagine?

As you can see, my candidacy [which I did not know was even a remote possibility] was approved on June 11.

The letter goes on to state that the IAHCP highlights and profiles the world’s Top Surgeons. The association features physicians who have demonstrated success and leadership in their profession. It even provides an opportunity to network, collaborate, and share information with other medical professionals from around the globe. [Just what I need—another social network of physicians.]

Inclusion in the book “The Leading Physicians of the World” is not only a tribute to my success, but is also a valuable resource for potential patients who are looking for a Top Surgeon in their area.

There is no charge for this honor.

I’m sure the organization did a lot of research before it approved my candidacy. I wonder how they missed that I’ve been retired from the practice of surgery for more than two years?

I thought this was remarkable, but last year New York Times writer Dr. Abigail Zuger easily topped me.

Last year, she wrote that a relative of hers had been named one of the world’s top physicians. All he had to do was to verify his biographical information, and he would be included in the same book and online reference mentioned above.

The good doctor wasn’t able to do so because unfortunately, he died 16 years ago.

What have we learned here?

As I have said on many occasions [e.g., here and here], one must take all ratings with a pinch of salt. 

Physician ratings? A truckload of salt.

How to rank surgical residency programs

In September, Doximity, a closed online community of over 300,000 physicians, released its ratings of residency programs in nearly every specialty. Many, including me, took issue with the methodology. Emergency medicine societies met with Doximity's co-founder over the issue and echoed some of the comments I had made about the lack of objectivity and emphasis on reputation.

I wonder if it is even possible to develop a set of valid criteria to rate residency programs. Every one I can think of is open to question. Let's take a look at some of them.

Reputation is an unavoidable component in any rating system. Unfortunately, it is rarely based on personal knowledge of any program because there is no way for anyone not directly involved with a program to assess its quality. Reputation is built on history, but all programs have turnover of chairs and faculty. Just as in sports, maintaining a dynasty over many years can sometimes be difficult. Deciding how much weight should be given to reputation is also problematic.

The schools that residents come from might be indicative of a program's quality, but university-based residencies tend to attract applicants from better medical schools. The other issue is who is to say which schools are the best?

Faculty and resident research is easy to measure but may be irrelevant when trying to answer the question of which programs produce the best clinical surgeons. Since professors tend to move from place to place, the current faculty may not be around for the entire 5 years of a surgery resident's training.

The number of residents who obtain subspecialty fellowships and where those fellowships are might be worthwhile, but would penalize programs that attract candidates who may be exceptional but are happy to become mere general surgeons.

Resident case loads including volume and breadth of experience would be very useful. However, these numbers have to be self-reported by programs. Self-reported data are often unreliable. Here are some examples why.

For several years, M.D. Anderson has been number one on the list of cancer hospitals as compiled by US News. It turns out that for 7 of those years, the hospital was counting all patients who were admitted through its emergency department as transfers and therefore not included in mortality figures. This resulted in the exclusion of 40% of M.D. Anderson's admissions, many of whom were likely the sickest patients.

The number and types of cases done by residents in a program have always been self-reported. The Residency Review Committee for Surgery and The American Board of Surgery have no way of independently verifying the number of cases done by residents, the level of resident participation in any specific case, or whether the minimum numbers for certain complex cases have truly been met.

So where does that leave us?

I'm not sure. I am interested in hearing what you have to say about how residency programs can be ranked.

More ratings—this time it's residency programs

Can you really decide which surgical residency program is right for you using Doximity's Residency Navigator?

I don't think so, and here's why.

The rankings of residency programs were obtained by surveying surgeon members of Doximity. They were asked name the five top programs for clinical surgery training. When the survey was announced in June, I predicted that most respondents would probably overlook the word "clinical" and focus on the usual famous academic institutions.

I also pointed out that anyone not intimately familiar with a program would be unable to judge whether it is good or not and suggested that reputation would be the main driver of results.

In fact, that is exactly what happened. Of the top 40 programs listed, all are based at university hospitals, as are 66 of the top 70. Back in June, I speculated about the top five programs and got the first two correct but in the wrong order.

A 2012 survey of surgical residents with over 4200 responders (an 80% response rate) found that community hospital trainees were significantly more satisfied with their operative experience and less likely to worry about practicing independently after graduation. Wouldn't you then expect a few community hospital programs to be among the top 40 hospitals for clinical surgery training?

Proof that the survey's findings are not reliable is that every one of the 253 surgical residency programs in the country was mentioned by one or more of those who responded. This included one program that has been terminated by the Residency Review Committee for Surgery. At least it appears near the bottom of the list.

The number of voters who cited the lower ranking programs must have been very few, meaning the difference between the 200th and 240th program ranks is probably not statistically significant.

Some programs that were rated are so new that very few or no residents have graduated yet. How could anyone know if they are turning out competent clinical surgeons?

Board passage rates for programs, which are available online, were omitted for some and were not clearly identified as the percentage of residents who passed both parts of the boards on the first attempt only.

The percentile rankings of alumni peer-reviewed articles, grants, and clinical trials are displayed prominently. What do those data have to do with the research question—which residency programs "offer the best clinical training"?

So what's the bottom line?

You can put the Doximity Resident Navigator in with the other misleading ratings of hospitals and doctors. Applicants considering surgical residencies should not rely on it for guidance.

It has warmed the hearts of faculty and residents at highly rated programs, but I wonder how the OR lounge discussions are going at places where programs ranked lower than expected.

Chance can turn a surgeon into a killer

Risk-adjusted 30- to 90-day outcome data for selected types of operations done by specific surgeons and hospitals are now being publicly posted online by England's National Health Service.

According to the site, "Any hospital or consultant [attending surgeon in the UK] identified as an outlier will be investigated and action taken to improve data quality and/or patient care."

After cardiac surgery outcomes data were made public in New York, some interesting unexpected consequences were noted.

Surgeons and hospitals resorted to "gaming the system" by declining to operate on patients who were high-risk and tinkering with patient charts to make those they did operate on seem sicker. This can be done by scouring the charts for all co-morbidities and making sure none are overlooked when they are coded. An article from New York Magazine explains it in more detail.

Interpreting outcomes data can be tricky.

In a post three years ago about a report that nine Maryland hospitals had higher-than-average complication rates, I pointed out that whenever you have averages, some hospitals are going to be worse than average unless all hospitals perform exactly the same way or, like medical students, are all above average.

A much more sophisticated way of looking at this subject appeared in a fascinating 2010 BBC News piece by Michael Blastland, who is the Nate Silver of England [or maybe Nate Silver is the Michael Blastland of the US], called "Can chance make you a killer?"

Blastland set up a statistical chance calculator for a hypothetical set of 100 hospitals or 100 surgeons performing 100 operations each. The model assumes that every patient has the same chance of dying and that every surgeon is equally competent. The standard is that a mortality rate 60% worse than the norm set by the government for any hospital or surgeon is not acceptable.

You are assigned one hospital. Using a slider, you may choose an operative mortality rate anywhere from 1% to 15%. After you do this a number of times and recalculate for each mortality rate, you will notice that the number of unacceptably performing hospitals or surgeons changes randomly for each percent mortality and your hospital may appear in the underperforming group strictly by chance alone.

The whole concept is explained in more detail on the site. I encourage you to try it for yourself. The link is here.

So it may be difficult for the NHS to separate the true outliers from the unlucky surgeons who happened to fall outside the established norms.

What do you think about this?

The best general surgery residency programs for clinical training?

I've received a couple of emails from Doximity [A closed medical "community" of > 280,000 doctors] reminding me to complete a survey which they are sponsoring jointly with U.S. News & World Report. They are asking members, possibly only surgeons, to name the best general surgery training programs in the country.

Not mentioned in the email but stated at the beginning of the survey is that they want respondents to name the 5 best programs for clinical training.

I have a feeling that not everyone will notice the part about clinical training, and we will get a list of the usual suspects just as we do every year with the U.S. News best hospitals survey.

For several reasons, the survey is fundamentally flawed.

There are 240 general surgery residency programs in the country. Unless one is personally involved with a program, it is impossible to judge the competency of its graduates. How would I or anyone else who does not work there know whether residents training at UCLA or Baylor or Lehigh Valley are clinically competent?

There are no accepted ways to judge the clinical skills of any surgeon. Video recording of procedures with judging by peers can assess technical ability, and as shown in the recent New England Journal paper from Michigan, there is some correlation with outcomes.

The American Board of Surgery publishes first attempt board passage rates for all programs, but passing the boards does not necessarily equate to clinical skill.

Most surgeons have probably encountered only one or two graduates of any of surgical residency. Even if the ones we have seen were great, they may not represent the majority of graduates.

I'll bet I can name most of the top 5 programs right now. These are not necessarily the programs that produce the best clinically trained residents.

Here are my guesses: Massachusetts General, Johns Hopkins, Mayo Clinic-Rochester MN, New York Presbyterian-Columbia, Cleveland Clinic.

In the past, some institutions on my list were rumored to be terrible places to learn to perform surgery because the residents did a lot of watching and retracting but not much operating. Whether that is true today or was so in the past, I could not tell you.

I guarantee you that no community hospital will rank in the top 20 [maybe top 50] despite the fact that such hospitals produce many fine clinical surgeons.

I have no idea which programs produce the best clinically trained surgeons. After the Doximity-U.S. News survey results are published, you won't know either.

A study says you can trust online physician ratings

This abstract comes from the Social Science Research Network:

Despite heated debate about the pros and cons of online physician ratings, very little systematic work examines the correlation between physicians’ online ratings and their actual medical performance. Using patients’ ratings of physicians at RateMDs website and the Florida Hospital Discharge data, we investigate whether online ratings reflect physicians’ medical skill by means of a two-stage model that takes into account patients’ ratings-based selection of cardiac surgeons. Estimation results suggest that five-star surgeons perform significantly better and are more likely to be selected by sicker patients than lower-rated surgeons. Our findings suggest that we can trust online physician reviews, at least of cardiac surgeons.

You won't be surprised to learn that I don't believe it. As is my custom, I decided to read the entire paper the full text of which can be found here. At 37 pages, the raw manuscript is rather lengthy. As a public service, I waded through it.

The authors, non-MD faculty from the William E. Simon Graduate School of Business Administration at the University of Rochester, in New York, combed the ratings for Florida cardiac surgeons on the website RateMDs.com and classified surgeons into three categories—five-star surgeons, non-five-star surgeons, and those with no ratings at all.

They looked at 799 quarterly opportunities for ratings over a 9-year period and found that 21% of surgeons had an average of 1.9 online ratings. The 79% of surgeons who did not have an online rating performed 79% of the total surgeries in 2012, the year that the authors analyzed for patient results.

The five-star surgeons had a mean of 1.8 reviews each, and only 10% had more than 2 reviews.

The average mortality rate for coronary artery bypass grafting (CABG) among the Florida cardiac surgeons was 1.8% in 2012. The five-star surgeons with multiple reviews had the highest mortality rates at 3.3%.

I could find no evidence that patient mortality rates were adjusted for risk. But a lot of statistical manipulations took place. It's all explained by this simple equation—one of many.

 The authors say, "For a representative patient who is severely ill, being treated by a five-star surgeon can reduce the in-hospital mortality by 55% compared with being treated by a non-five-star surgeon. [I have no idea how they determined that figure.] Moreover, the negative and significant coefficient of no-ratings suggests that patients treated by surgeons without ratings also have a lower mortality rate than those treated by non-five-star surgeons, all else being equal." Huh?

And this, "Patients with private insurance are less likely to select the surgeons without ratings than patients with Medicare. We suspect that patients with private insurance have to use search engines to figure out whether a surgeon is within the network that an insurance plan covers, while government patients enjoy a large physician network." I question that assumption. My experience is that patients with Medicare sometimes have problems finding anyone to care for them, let alone the best surgeons.

It turns out that half of the five-star surgeons had only one review. In one iteration of the study model, five-star surgeons with multiple reviews had higher mortality rates than those with only one review, but then they also say, "One surprising finding is that five-star surgeons with a single review show no statistical difference in performance from those with multiple reviews."

Are you as confused as I?

The paper makes no mention of the possibility that some of the online ratings could be fake. Recent articles [here and here] suggest that one-fifth to one-third of such reviews are phony.

You can manipulate the statistics all you want, but you won't convince me that one or two or even 20 online ratings are valid or useful in choosing a surgeon.

Reality check: Hospital safety scores

Imagine you are sick and live in New York City. Your doctor tells you that you need major surgery. Luckily, you have excellent insurance and can go anywhere in the city for that operation.

Being a good consumer, you decide to check the HospitalSafetyScore.org website, which is sponsored by the Leapfrog Group, a nationally known patient safety organization.

You pull up a handy map of upper Manhattan and the lower Bronx to check the safety scores of hospitals in that area which is near your neighborhood.

A hospital on the Manhattan side (orange arrow) has a safety score of only "C" while over in the Bronx, there is an "A" rated hospital (blue arrow).

It's a no-brainer, right?

Clearly the safer of the two is the one with the "A" rating.

But consider this. The "A" rated hospital is Lincoln Medical and Mental Health Center, one of 11 hospitals owned and run by the city of New York. It is a teaching hospital. But there is little research going on, and there are no regionally or nationally recognized experts in just about any specialty of medicine or surgery practicing there.

The "C" rated hospital is New York Presbyterian, the main teaching hospital of Columbia University's medical school.

A 2012 patient safety study by Consumer Reports rated Lincoln as the 16th worst hospital for safety in the NY metro area. Presbyterian did not make that list of 30 such hospitals.

Healthgrades rates New York Presbyterian as #5 of 203 hospitals in New York State with 15 5-Star ratings and 11 quality awards. Lincoln was ranked at #88 with 3 5-Star ratings and 1 quality award.

US News & World Report published a list of the 18 best hospitals in the country that made its "Honor Roll." That list included New York Presbyterian at #7, and it was the highest rated hospital in the New York area.

Now which hospital would you choose?