How to select surgical residents: The evidence

On Twitter a while ago, a medical student asked me how surgical program directors select new residents. Then a discussion arose among some academic surgeons on the same topic. Someone suggested that medical school grades were the best way to tell whether an applicant would be a successful resident.

The fact is that we aren't really sure what the best way to choose residents is.

First, here's what we really do.

Study: 31% of general surgery residents need remediation. Why?

A recent paper in Annals of Surgery depicts the rate of resident remediation over a decade or so at six general surgery programs in California. The authors reviewed the records of 348 categorical general surgery residents and found that 107 (31%) required mediation with knowledge deficits the primary reason in 74%. Other issues, such as interpersonal and communication skills, patient care and professionalism, were cited far less often.

Need for remediation did not correlate with attrition. Remediated residents left programs at a rate of 20% compared to 15% of non-remediated residents, p = 0.40.

On multivariate analysis, only two factors were associated with the need for remediation. One was USMLE Step 1 scores which were lower in the remediated group. But the median difference in scores between remediated and non-remediated residents was only 7 points with wide and overlapping interquartile ranges, and both median scores were above the average for all medical students over the years of the study.

The other factor was quite remarkable. Remediated residents were significantly more likely to have received a grade of "honors" for their medical school clerkship and surgery. How can this be?

The authors speculated, "One thought is that medical students start residency under-prepared for the rigors of surgical residency." Now where I heard that before? I have previously blogged (here) about the unrealistic third-year experiences of medical students on surgical rotations. While I agree that they likely are not ready for the workload, I'm not sure what it has to do with the primary reason for their poor performance—a perceived knowledge deficit. Does hard work cause them to forget everything they’ve learned or are they taught the wrong stuff in med school?

I think not. The real reason may be found in the way medical students are graded. A group from Harvard looked at medical school grading systems and found that honors grades in third-year surgery clerkships are given to an average of about 30% of students ranging from a low of 7% to an mind-boggling high of 67%. I understand that those accepted to med school are smart, but how is it that two-thirds of the class can achieve honors in surgery? Could it be that some of the honors grades given to residents who eventually needed remediation were not warranted?

If you would like to read more about this, you may read the full text of the paper on medical school grades here or a summary on my blog.

Medical School Grading and T-Ball: Everyone Gets A Trophy

"Variation and Imprecision of Clerkship Grading in US Medical Schools” is the understated title of the paper (full text here) in the August 2012 issue of the journal Academic Medicine. The authors, from the department of medicine at Brigham and Women’s Hospital, analyzed 2009-2010 third-year clerkship grades from 119 (97%) of the 123 US medical schools. They found many different grading systems ranging from two levels (pass/fail) to 11 levels of grades.

The terminology used by the schools to describe the different grades is positively comical. To borrow an analogy I’ve used in a previous post about dean’s letters, the citizens of Lake Wobegon would be proud because no student is “average.”

Here are some examples:

High honors, honors, pass, fail (In some schools “honors” is not the highest possible grade).
Honors, satisfactory plus, satisfactory, fail.
Honors, satisfactory, low satisfactory, fail.
Honors, high satisfactory, satisfactory, low satisfactory, unsatisfactory. (Does “unsatisfactory” mean, dare I say it, “fail”?)
Honors, near honors, pass, fail.
Excellent, good, fail.
Honors, advanced, proficient, fail.
Honors, letter of commendation, fail.

The highest grade attainable was awarded to 23% of those students in schools with three-tiered systems (range 5-51%), to 34% (range 2-84%) in four-tiered systems and to 33% (7-93%) in schools with five grade levels.

It gets worse. The authors noted that 97% of all medical students were given one of the top three grades regardless of whether the schools used 4, 5, or 6 levels of grading.

From the paper, “Less than 1% of all US medical students fail required clerkships, regardless of the grading system used.” This raises the question of whether the grade “fail” is even necessary.

Focusing on surgery, an average of about 30% of all students got the highest grade possible in their surgical clerkship, but the percentage of the class receiving the top grade ranged from 7% to 67%. This may account for the paradox found in a paper on surgical resident performance: A significant predictor of the need for remediation was that the resident had received honors in his surgical third-year clerkship. It appears that a grade of honors is virtually meaningless.

This is an excellent example of what I call the “T-ball culture”: No one keeps score. All games end in a tie. Everyone gets a trophy.

The authors of the paper recommended that schools consider creating a more consistent, transparent and reliable system of grading. As a former surgical residency program director who grappled with the difficulty in interpreting the meaning of applicant grades from different schools, this seems remarkably clear to me.

An editorial in the same issue of the journal agreed that grade terminology should be standardized but cautioned that normative grading (establishing a set distribution or “curve” of grades) may not be the answer. The editorialists offered some other possibilities such as criterion-based grading or emphasizing the mastery of a subject as a goal rather than the achieving of a specific grade.

I do not have the background in educational theory to say what is right or wrong. I do know that a grading system with so many variables and such a skewed distribution is of no help whatsoever in evaluating the desirability of an individual applicant to a residency program.