How to rank surgical residency programs

In September, Doximity, a closed online community of over 300,000 physicians, released its ratings of residency programs in nearly every specialty. Many, including me, took issue with the methodology. Emergency medicine societies met with Doximity's co-founder over the issue and echoed some of the comments I had made about the lack of objectivity and emphasis on reputation.

I wonder if it is even possible to develop a set of valid criteria to rate residency programs. Every one I can think of is open to question. Let's take a look at some of them.

Reputation is an unavoidable component in any rating system. Unfortunately, it is rarely based on personal knowledge of any program because there is no way for anyone not directly involved with a program to assess its quality. Reputation is built on history, but all programs have turnover of chairs and faculty. Just as in sports, maintaining a dynasty over many years can sometimes be difficult. Deciding how much weight should be given to reputation is also problematic.

The schools that residents come from might be indicative of a program's quality, but university-based residencies tend to attract applicants from better medical schools. The other issue is who is to say which schools are the best?

Faculty and resident research is easy to measure but may be irrelevant when trying to answer the question of which programs produce the best clinical surgeons. Since professors tend to move from place to place, the current faculty may not be around for the entire 5 years of a surgery resident's training.

The number of residents who obtain subspecialty fellowships and where those fellowships are might be worthwhile, but would penalize programs that attract candidates who may be exceptional but are happy to become mere general surgeons.

Resident case loads including volume and breadth of experience would be very useful. However, these numbers have to be self-reported by programs. Self-reported data are often unreliable. Here are some examples why.

For several years, M.D. Anderson has been number one on the list of cancer hospitals as compiled by US News. It turns out that for 7 of those years, the hospital was counting all patients who were admitted through its emergency department as transfers and therefore not included in mortality figures. This resulted in the exclusion of 40% of M.D. Anderson's admissions, many of whom were likely the sickest patients.

The number and types of cases done by residents in a program have always been self-reported. The Residency Review Committee for Surgery and The American Board of Surgery have no way of independently verifying the number of cases done by residents, the level of resident participation in any specific case, or whether the minimum numbers for certain complex cases have truly been met.

So where does that leave us?

I'm not sure. I am interested in hearing what you have to say about how residency programs can be ranked.

More ratings—this time it's residency programs

Can you really decide which surgical residency program is right for you using Doximity's Residency Navigator?

I don't think so, and here's why.

The rankings of residency programs were obtained by surveying surgeon members of Doximity. They were asked name the five top programs for clinical surgery training. When the survey was announced in June, I predicted that most respondents would probably overlook the word "clinical" and focus on the usual famous academic institutions.

I also pointed out that anyone not intimately familiar with a program would be unable to judge whether it is good or not and suggested that reputation would be the main driver of results.

In fact, that is exactly what happened. Of the top 40 programs listed, all are based at university hospitals, as are 66 of the top 70. Back in June, I speculated about the top five programs and got the first two correct but in the wrong order.

A 2012 survey of surgical residents with over 4200 responders (an 80% response rate) found that community hospital trainees were significantly more satisfied with their operative experience and less likely to worry about practicing independently after graduation. Wouldn't you then expect a few community hospital programs to be among the top 40 hospitals for clinical surgery training?

Proof that the survey's findings are not reliable is that every one of the 253 surgical residency programs in the country was mentioned by one or more of those who responded. This included one program that has been terminated by the Residency Review Committee for Surgery. At least it appears near the bottom of the list.

The number of voters who cited the lower ranking programs must have been very few, meaning the difference between the 200th and 240th program ranks is probably not statistically significant.

Some programs that were rated are so new that very few or no residents have graduated yet. How could anyone know if they are turning out competent clinical surgeons?

Board passage rates for programs, which are available online, were omitted for some and were not clearly identified as the percentage of residents who passed both parts of the boards on the first attempt only.

The percentile rankings of alumni peer-reviewed articles, grants, and clinical trials are displayed prominently. What do those data have to do with the research question—which residency programs "offer the best clinical training"?

So what's the bottom line?

You can put the Doximity Resident Navigator in with the other misleading ratings of hospitals and doctors. Applicants considering surgical residencies should not rely on it for guidance.

It has warmed the hearts of faculty and residents at highly rated programs, but I wonder how the OR lounge discussions are going at places where programs ranked lower than expected.