Just in. My wife named a top orthopedist

A couple of weeks ago my wife received this in the mail.

Of course we are thrilled that she was finally recognized as one of the top orthopedic surgeons in our city.

Last year I blogged about a similar honor that I received from the International Association of Healthcare Professionals. That eminent organization had named me a top surgeon. I was a little skeptical because at the time, I had been retired for two years.

I am sure the vetting process for my wife’s inclusion was quite thorough. However, they may have been mistaken about some important criteria.

For one thing, she is not an orthopedic surgeon. In fact, she isn't even a doctor. She is a nurse and manages an office for a large group of orthopedists. She is pretty good at reading x-rays and is a whiz at organization, staff management, patient interactions, suture removal, and fitting crutches. Maybe that counts for something.

From the format of the letters and the mention of the "renowned publication, The Leading Physicians of the World," it looks like the International Association of Orthopedic Surgeons might be run by the same outfit as the International Association of Healthcare Professionals.

The website does not list the cost but here are the benefits of being selected: wall plaque of achievement, leading physician feature video, leading physician press release, leading physician specialist online exposure, physician feature website, and [my favorite] endorsement of credentials.

We look forward to the selection of our dog Bailey as perhaps one of the leading neurosurgeons of the world.

I think she qualifies because she has been dead for three years.

The Surgeon Scorecard: My analysis

I've got nothing against ProPublica. If a valid way to rate surgeons is ever discovered, I would support it completely. However, ProPublica's Surgeon Scorecard is not the answer.

I keep hearing its defenders say, "Some data is better than no data at all." I disagree strongly with that. To me, bad data is worse than no data at all. People with much more statistical sophistication than I have pointed out the flaws in the scorecard.

Digression: Having written many posts about statistics, I can tell you that the mere mention of the word drives readers away about as fast as if you were to yell "Fire" in a crowded theater.

I want to focus on a different area. The scorecard has created a lot of chatter on Twitter, and just about everyone I know has blogged about it.

This reminds me of a couple of posts I wrote back in 2011. [Links here and here.] I pointed out that Twitter might not be as important as those of us who use it think it is.

While we were busy arguing about the merits of the scorecard on Twitter, I'm not so sure what the general public was doing.

For example, ProPublica says the Surgeon Scorecard has had over 1 million visitors since its launch. That sounds like a lot until you consider that the current population of the United States is estimated at 321 million. So 1 million people would be 0.3%. We do not know how many of those 1 million were unique visitors. It could be that many of them were doctors looking for their own statistics and bloggers looking for ideas.

That the public may not care was reinforced by a rather tepid response to the ProPublica AMA (Ask Me Anything) on Reddit today.

By 1:00 PM EDT, which was two hours into the AMA, there were 80 comments, 31 of which were by ProPublica staff or the spine surgeon who had consulted on the scorecard's methods.

Just to give you some perspective, an AMA last year by a guy with two penises drew 17,134 comments.

Because the demographic is skewed toward younger people, perhaps Reddit may not have been the right venue. Although Reddit boasts 169 million unique visitors per month, the most recent figures show that 33% of the Reddit users are mostly men between 18 and 49 years old. Those under 18 are not counted but represent "a substantial percentage of Reddit users."

My two favorite questions asked of ProPublica were "How can I tell if my doctor is capable of making an error?" and "Do you fix the leg which is broken completely?" [Did the question refer to a leg that was completely broken, or did it mean should the leg be completely fixed?]

What have we learned here? It's hard to say.

If you want to read a measured critique of the scorecard, go to Dr. John Mandrola's piece on Medscape.

How to pick the leading physicians of the world

 

My "real life" self has been selected as a Top Surgeon in my city by the International Association of Healthcare Professionals (IAHCP). I will be spotlighted in the renowned publication “The Leading Physicians of the World.” Can you imagine?

As you can see, my candidacy [which I did not know was even a remote possibility] was approved on June 11.

The letter goes on to state that the IAHCP highlights and profiles the world’s Top Surgeons. The association features physicians who have demonstrated success and leadership in their profession. It even provides an opportunity to network, collaborate, and share information with other medical professionals from around the globe. [Just what I need—another social network of physicians.]

Inclusion in the book “The Leading Physicians of the World” is not only a tribute to my success, but is also a valuable resource for potential patients who are looking for a Top Surgeon in their area.

There is no charge for this honor.

I’m sure the organization did a lot of research before it approved my candidacy. I wonder how they missed that I’ve been retired from the practice of surgery for more than two years?

I thought this was remarkable, but last year New York Times writer Dr. Abigail Zuger easily topped me.

Last year, she wrote that a relative of hers had been named one of the world’s top physicians. All he had to do was to verify his biographical information, and he would be included in the same book and online reference mentioned above.

The good doctor wasn’t able to do so because unfortunately, he died 16 years ago.

What have we learned here?

As I have said on many occasions [e.g., here and here], one must take all ratings with a pinch of salt. 

Physician ratings? A truckload of salt.

Chance can turn a surgeon into a killer

Risk-adjusted 30- to 90-day outcome data for selected types of operations done by specific surgeons and hospitals are now being publicly posted online by England's National Health Service.

According to the site, "Any hospital or consultant [attending surgeon in the UK] identified as an outlier will be investigated and action taken to improve data quality and/or patient care."

After cardiac surgery outcomes data were made public in New York, some interesting unexpected consequences were noted.

Surgeons and hospitals resorted to "gaming the system" by declining to operate on patients who were high-risk and tinkering with patient charts to make those they did operate on seem sicker. This can be done by scouring the charts for all co-morbidities and making sure none are overlooked when they are coded. An article from New York Magazine explains it in more detail.

Interpreting outcomes data can be tricky.

In a post three years ago about a report that nine Maryland hospitals had higher-than-average complication rates, I pointed out that whenever you have averages, some hospitals are going to be worse than average unless all hospitals perform exactly the same way or, like medical students, are all above average.

A much more sophisticated way of looking at this subject appeared in a fascinating 2010 BBC News piece by Michael Blastland, who is the Nate Silver of England [or maybe Nate Silver is the Michael Blastland of the US], called "Can chance make you a killer?"

Blastland set up a statistical chance calculator for a hypothetical set of 100 hospitals or 100 surgeons performing 100 operations each. The model assumes that every patient has the same chance of dying and that every surgeon is equally competent. The standard is that a mortality rate 60% worse than the norm set by the government for any hospital or surgeon is not acceptable.

You are assigned one hospital. Using a slider, you may choose an operative mortality rate anywhere from 1% to 15%. After you do this a number of times and recalculate for each mortality rate, you will notice that the number of unacceptably performing hospitals or surgeons changes randomly for each percent mortality and your hospital may appear in the underperforming group strictly by chance alone.

The whole concept is explained in more detail on the site. I encourage you to try it for yourself. The link is here.

So it may be difficult for the NHS to separate the true outliers from the unlucky surgeons who happened to fall outside the established norms.

What do you think about this?

The best general surgery residency programs for clinical training?

I've received a couple of emails from Doximity [A closed medical "community" of > 280,000 doctors] reminding me to complete a survey which they are sponsoring jointly with U.S. News & World Report. They are asking members, possibly only surgeons, to name the best general surgery training programs in the country.

Not mentioned in the email but stated at the beginning of the survey is that they want respondents to name the 5 best programs for clinical training.

I have a feeling that not everyone will notice the part about clinical training, and we will get a list of the usual suspects just as we do every year with the U.S. News best hospitals survey.

For several reasons, the survey is fundamentally flawed.

There are 240 general surgery residency programs in the country. Unless one is personally involved with a program, it is impossible to judge the competency of its graduates. How would I or anyone else who does not work there know whether residents training at UCLA or Baylor or Lehigh Valley are clinically competent?

There are no accepted ways to judge the clinical skills of any surgeon. Video recording of procedures with judging by peers can assess technical ability, and as shown in the recent New England Journal paper from Michigan, there is some correlation with outcomes.

The American Board of Surgery publishes first attempt board passage rates for all programs, but passing the boards does not necessarily equate to clinical skill.

Most surgeons have probably encountered only one or two graduates of any of surgical residency. Even if the ones we have seen were great, they may not represent the majority of graduates.

I'll bet I can name most of the top 5 programs right now. These are not necessarily the programs that produce the best clinically trained residents.

Here are my guesses: Massachusetts General, Johns Hopkins, Mayo Clinic-Rochester MN, New York Presbyterian-Columbia, Cleveland Clinic.

In the past, some institutions on my list were rumored to be terrible places to learn to perform surgery because the residents did a lot of watching and retracting but not much operating. Whether that is true today or was so in the past, I could not tell you.

I guarantee you that no community hospital will rank in the top 20 [maybe top 50] despite the fact that such hospitals produce many fine clinical surgeons.

I have no idea which programs produce the best clinically trained surgeons. After the Doximity-U.S. News survey results are published, you won't know either.

A study says you can trust online physician ratings

This abstract comes from the Social Science Research Network:

Despite heated debate about the pros and cons of online physician ratings, very little systematic work examines the correlation between physicians’ online ratings and their actual medical performance. Using patients’ ratings of physicians at RateMDs website and the Florida Hospital Discharge data, we investigate whether online ratings reflect physicians’ medical skill by means of a two-stage model that takes into account patients’ ratings-based selection of cardiac surgeons. Estimation results suggest that five-star surgeons perform significantly better and are more likely to be selected by sicker patients than lower-rated surgeons. Our findings suggest that we can trust online physician reviews, at least of cardiac surgeons.

You won't be surprised to learn that I don't believe it. As is my custom, I decided to read the entire paper the full text of which can be found here. At 37 pages, the raw manuscript is rather lengthy. As a public service, I waded through it.

The authors, non-MD faculty from the William E. Simon Graduate School of Business Administration at the University of Rochester, in New York, combed the ratings for Florida cardiac surgeons on the website RateMDs.com and classified surgeons into three categories—five-star surgeons, non-five-star surgeons, and those with no ratings at all.

They looked at 799 quarterly opportunities for ratings over a 9-year period and found that 21% of surgeons had an average of 1.9 online ratings. The 79% of surgeons who did not have an online rating performed 79% of the total surgeries in 2012, the year that the authors analyzed for patient results.

The five-star surgeons had a mean of 1.8 reviews each, and only 10% had more than 2 reviews.

The average mortality rate for coronary artery bypass grafting (CABG) among the Florida cardiac surgeons was 1.8% in 2012. The five-star surgeons with multiple reviews had the highest mortality rates at 3.3%.

I could find no evidence that patient mortality rates were adjusted for risk. But a lot of statistical manipulations took place. It's all explained by this simple equation—one of many.

 The authors say, "For a representative patient who is severely ill, being treated by a five-star surgeon can reduce the in-hospital mortality by 55% compared with being treated by a non-five-star surgeon. [I have no idea how they determined that figure.] Moreover, the negative and significant coefficient of no-ratings suggests that patients treated by surgeons without ratings also have a lower mortality rate than those treated by non-five-star surgeons, all else being equal." Huh?

And this, "Patients with private insurance are less likely to select the surgeons without ratings than patients with Medicare. We suspect that patients with private insurance have to use search engines to figure out whether a surgeon is within the network that an insurance plan covers, while government patients enjoy a large physician network." I question that assumption. My experience is that patients with Medicare sometimes have problems finding anyone to care for them, let alone the best surgeons.

It turns out that half of the five-star surgeons had only one review. In one iteration of the study model, five-star surgeons with multiple reviews had higher mortality rates than those with only one review, but then they also say, "One surprising finding is that five-star surgeons with a single review show no statistical difference in performance from those with multiple reviews."

Are you as confused as I?

The paper makes no mention of the possibility that some of the online ratings could be fake. Recent articles [here and here] suggest that one-fifth to one-third of such reviews are phony.

You can manipulate the statistics all you want, but you won't convince me that one or two or even 20 online ratings are valid or useful in choosing a surgeon.

Patient satisfaction and reality

Christmas came early for us skeptics this year. In a landmark study, certainly one of the most interesting and thought-provoking of the year-to-date, researchers from the University of California-Davis found that the more satisfied patients were with their physicians, the higher their hospital admission rates, prescription costs and total costs were. And patients with the highest level of satisfaction with their doctors had higher mortality rates compared to those patients least satisfied with their doctors.

The prospective cohort study included almost 52,000 patients. The full text of the paper can be viewed here. I won’t bore you with the details of how it was done except to say it looks scientifically legitimate.

The study’s authors say, “Patients typically bring expectations to medical encounters, often making specific requests of physicians, and satisfaction correlates with the extent to which physicians fulfill patient expectations. Patient requests have also been shown to have a powerful influence on physician prescribing behavior, and our findings suggest that patient satisfaction may be particularly strongly linked with prescription drug expenditures.”

Translation. What they mean is that doctors who do what their patients want receive higher satisfaction scores. This may also include admitting patients sooner rather than later, which could account for the fact that patients who were more satisfied had lower rates of emergency department visits.

So it seems patient satisfaction may not be the great “quality metric” that it is touted to be. The study concludes, “an overemphasis on patient satisfaction could have unintended adverse effects on health care utilization, expenditures, and outcomes.”

Due to something called "copyright law," I will resist the urge to quote the whole paper. I suggest you read it and judge for yourself.

Joint Commission Proves It's as Irrelevant As HealthGrades

The New York Times reports that the Joint Commission has just published a list of its 405 "Top Performing Hospitals." As is typical of these types of evaluations, most of the large, well-known teaching hospitals where knowledgeable folks [like doctors] go for care when they are really sick didn't make the list.

Similar to the HealthGrades list of "top" hospitals [which I have commented about in the past], the JC's list is dominated by small community hospitals without teaching programs. These institutions know how to play the game and have figured out that compliance with process measures results in high marks from the likes of HealthGrades and the JC. For a related post on why process measures don't mean better care, click here.

At least the Times article pointed out that no hospital in New York City made the JC list. What about Chicago? Sorry, just a children's hospital and a VA hospital. Philadelphia? No university hospital but one community hospital. Surely University of Pittsburgh Medical Center? No, not the main hospital but several of its suburban affiliates made the list. St. Louis? Nope, no good hospitals there. Pick any major city. See the list for yourself.

Guess which state had the most top performing hospitals by far? It's a state that immediately comes to mind when one thinks of quality medical care. Of course, Florida with 51, not one of which was a university hospital unless you want to count the University Hospital & Medical Center of Tamarac. A visit to its website fails to reveal the name of the university it is affiliated with. However, Don Shula, former Miami Dolphins coach, endorses the emergency department in a short video.

A survey I would like to see is what hospitals would the executives of HealthGrades and the JC choose if they or a family member had a serious illness? I'm guessing that list would be a lot shorter and would not include more than 400 of the 405 on the JC list of top performers.

Hospital and Doctor Ratings: Junk Science? No, No Science at All

Pop statistician/philosopher Malcolm Gladwell takes down the ridiculous college rating process in this week’s New Yorker magazine. While pointing out the many problems with the way U.S. News goes about rankings colleges, he mentions a 2010 study published in Archives of Internal Medicine that similarly debunks the rankings of hospitals. That study found that reputation alone is the key to a hospital receiving a high ranking and the ranking has nothing to do with quality. This is true of the college ranking system too. A key point regarding the power of reputation is that as the rankings are publicized every year, the top hospitals and colleges become even more prestigious, which of course, enhances their reputations further.

The college ranking methodology takes into account a number of factors and weighs them in an arbitrary way. Gladwell feels that cost is not given enough emphasis as expensive private universities dominate the top of the list. When it comes to hospital rankings, U.S. News surveys only 250 physicians in each specialty and asks them to name the five best hospitals in their field. It is impossible for a hospital without an established national reputation to ever be ranked highly.

This whole charade is carried to an almost comical extreme by the folks at Castle Connolly, who bring you “America’s Top Doctors” and regional offshoots of the same concept. These ratings are eagerly awaited ever year and are the subject of lengthy articles in magazines. Advertising revenue is generated as hospitals tout their MDs who have been fortunate enough to have made the Castle Connolly list.

Here is a little secret. Castle Connolly sends questionnaires to hospital department chairs and asks them to name the best doctors in not only their fields but every medical specialty. No other criteria are used. Having been a department chairman for over 23 years, I can tell you that it is impossible for me to know anything about the quality of the work of any physician at another hospital or even sometimes another specialty in my own hospital. The vote is strictly by reputation. Many fine doctors make the list because reputations are very often correct. But not always. A surgeon can be well-known for research or involvement in organizations, but she may not necessarily be the best clinician around.

So what is a prospective patient to do? I’ve already blogged about the shortcomings of Healthgrades [here and here] and the CMS [Medicare] Physician Compare website is not ready for prime time. For now, I suggest you ask friends who may have had illnesses similar to yours for recommendations. Or you’ll have to trust the doctor who refers you to a specialist and your instincts when you meet her.