Wednesday, October 17, 2012

Medical School Grading and T-Ball: Everyone Gets A Trophy

"Variation and Imprecision of Clerkship Grading in US Medical Schools” is the understated title of the paper (full text here) in the August 2012 issue of the journal Academic Medicine. The authors, from the department of medicine at Brigham and Women’s Hospital, analyzed 2009-2010 third-year clerkship grades from 119 (97%) of the 123 US medical schools. They found many different grading systems ranging from two levels (pass/fail) to 11 levels of grades.

The terminology used by the schools to describe the different grades is positively comical. To borrow an analogy I’ve used in a previous post about dean’s letters, the citizens of Lake Wobegon would be proud because no student is “average.”

Here are some examples:

High honors, honors, pass, fail (In some schools “honors” is not the highest possible grade).
Honors, satisfactory plus, satisfactory, fail.
Honors, satisfactory, low satisfactory, fail.
Honors, high satisfactory, satisfactory, low satisfactory, unsatisfactory. (Does “unsatisfactory” mean, dare I say it, “fail”?)
Honors, near honors, pass, fail.
Excellent, good, fail.
Honors, advanced, proficient, fail.
Honors, letter of commendation, fail.

The highest grade attainable was awarded to 23% of those students in schools with three-tiered systems (range 5-51%), to 34% (range 2-84%) in four-tiered systems and to 33% (7-93%) in schools with five grade levels.

It gets worse. The authors noted that 97% of all medical students were given one of the top three grades regardless of whether the schools used 4, 5, or 6 levels of grading.

From the paper, “Less than 1% of all US medical students fail required clerkships, regardless of the grading system used.” This raises the question of whether the grade “fail” is even necessary.

Focusing on surgery, an average of about 30% of all students got the highest grade possible in their surgical clerkship, but the percentage of the class receiving the top grade ranged from 7% to 67%. This may account for the paradox found in a paper on surgical resident performance: A significant predictor of the need for remediation was that the resident had received honors in his surgical third-year clerkship. It appears that a grade of honors is virtually meaningless.

This is an excellent example of what I call the “T-ball culture”: No one keeps score. All games end in a tie. Everyone gets a trophy.

The authors of the paper recommended that schools consider creating a more consistent, transparent and reliable system of grading. As a former surgical residency program director who grappled with the difficulty in interpreting the meaning of applicant grades from different schools, this seems remarkably clear to me.

An editorial in the same issue of the journal agreed that grade terminology should be standardized but cautioned that normative grading (establishing a set distribution or “curve” of grades) may not be the answer. The editorialists offered some other possibilities such as criterion-based grading or emphasizing the mastery of a subject as a goal rather than the achieving of a specific grade.

I do not have the background in educational theory to say what is right or wrong. I do know that a grading system with so many variables and such a skewed distribution is of no help whatsoever in evaluating the desirability of an individual applicant to a residency program.

16 comments:

Carolyn Thomas said...

What?!?!? How do I find out if my surgeon received highest honors in his/her surgical third-year clerkship? Should I instead be seeking out a surgeon who achieved more promisingly predictive grades (say, a "satisfactory'?)

And here I thought "highest honors" had something to do with merit and skill . . .

Like T-ball indeed. Only difference: my son's T-ball coach didn't distribute scalpels and a medical license at the end of the season!

Skeptical Scalpel said...

I think the only way to ask what grade a surgeon received in her third-year surgery rotation would be to ask. Don't panic if she got honors. the study only showed those with honors grades were more likely to have attained that grade. Not everyone who did needed remediation.

Old joke: Do you know what they call the medical student who graduated last in her class?

"Doctor"

Keeweedoc said...

Classic example of "everyones a winner" syndrome.
Its not so here, we had "excellence, pass, fail"
excellence was top 10% hard luck if your year was particularly gifted. again had max 2% fail rate. dare I say it, medical school aint that hard for the people who get in. your selecting people who have worked for years for high grades, why would we expect them to fail?

Carolyn Thomas said...

Ouch!

Skeptical Scalpel said...

Thanks for the comments. It's true that most people who are accepted to med school are probably smart enough to get through. But there are those who lack motivation or have other issues once they get in. Should there be no standards at all?

In a previous blog (http://skepticalscalpel.blogspot.com/2010/08/medical-school-and-surgery.html), I mentioned that I once received an application for my residency program from a student who had been in medical school for 10 years. He had failed various courses, taken time off, had remediation etc. Perhaps he should have simply failed and been asked to leave.

keeweedoc said...

Just to clarify that point.
2% fail rate was made up of those who mostly failed terms. med school has a policy of letting people repeat the year. if they fail again. out.
This is because most who fail usually do so due to outside pressures ect.

Skeptical Scalpel said...

Thanks for clearing that up. I agree that most who fail or drop out do so for reasons other than intelligence.

Anonymous said...

What medical schools need are "communications coaches" to help the promising (but obnoxious) students learn how to relate to patients (and nurses, respiratory therapists, social workers, etc.) Like a Dale Carnegie class. No, I'm serious.
As a nurse, I see lots of doctors either treat the RNs like we are idiots, the patients like they are ignorant dolts, or they race in and out of the patient's room so fast the patient barely knows they are there.
I also work with a lot of awesome doctors, but the "geniuses" with poor social skills do seem to be all around.

Skeptical Scalpel said...

Good points. Thanks.

Unknown said...

Midwest said...

I am digging up a dead thread, but...

The match season is gearing up soon and I have to say that my 3rd year of straight passes weighs heavily on my mind. It is great material for advisors to remind me that I wont match into surgery. This is likely true, though in my defense, I am not an idiot or unmotivated, having scored >250 on my StepI and completed numerous publications over the last 2 years.... Do you have any advice as a former decision maker on the other side of the fence?

Skeptical Scalpel said...

Excellent question.

If you apply to appropriate programs, I think you will get several interview offers. You are a long shot for the major university residencies but should get some interviews at lesser university and decent community hospital programs. You will need to wow them at the interviews.

I think you need to explain in your personal statement why you only got passes in all of your courses. I hope you have some good LORs. Your Step 1 is a real plus. I would be nice if your publications are relevant (i.e., in journals that people actually have heard of).

Program directors love applicants who already know how to write a paper. It is uncommon, especially in an applicant to a community hospital program where the need for papers is great. Feel free to email me if you have further questions.

Good luck. I would love to hear how you fare.

Unknown said...

My medical school in 1974-78 was strictly pass fail, and everybody passed. The way you got a good residency slot was the way I expect it has always worked: key people made phone calls to program directors.

Skeptical Scalpel said...

Christopher, thanks for commenting. Phone calls work for the top students. I'm not sure how they work when the mid- or low level students are involved.

Anonymous said...

In my state in Australia the medical schools have agreed on a standardised z-score system which I think works pretty well. Each school ranks its students 1 to x, and then the rankings are combined and everyone gets a spot on the curve. Hospitals are then free to use it as they prefer - some take students with the highest scores, others set a threshold ( say 60th percentile or better) and then use other factors to discriminate. We don't have any national exam like the USMLE that could otherwise be used to compare applicants from different schools.

Skeptical Scalpel said...

Thanks for the information. I wasn't aware of that. Do you think it's fair and does it work well?

Post a Comment

Note: Only a member of this blog may post a comment.