Among the issues are lack of uniformity of content, excessive use of superlatives, reliability—if the writer is not known to the recipient—and more.
Even if the writers are well-known academic surgeons, the degree of their personal knowledge of the applicants is not always clear.
During an extensive Twitter discussion over the weekend, someone mentioned that in an attempt to deal with this problem, emergency medicine had developed a standardized letter of recommendation.
A recent paper from the EM Standardized Letter of Recommendation Task Force shows that there is still work to be done. From the abstract:
For the question on "global assessment," students were scored in the top 10% in 234 of 583 of applications (40.1%), and 485 of 583 (83.2%) of the applicants were ranked above the level of their peers. Similarly, >95% of all applicants were ranked in the top third compared to peers, for all but one section under "qualifications for emergency medicine."
I've written before that dean's letters are more like public relations press releases than accurate assessments of a student's performance. You will rarely find negative comments in them. But another recent paper by a group of psychiatrists found that The presence of any negative comments in the dean's letter yielded significant correlations with future problems. Further, those applicants with future major problems had significantly more negative comments in the dean's letter than did those with future minor problems. Other factors such as USMLE scores, failed courses, letters of recommendation, and interviewer ratings and comments did not predict future problems.
These problems are not new. A 1983 New England Journal opinion piece about recommendation letters entitled "Fantasy Land" is remarkable for its validity even today. Here are a few choice quotes.
It's a land where everyone is "a pleasure to work with," has "excellent initiative," is "enthusiastic and conscientious," and possesses and "above-average fund of knowledge."
No one is ever poor, fair, or average; they are all "very good" or "excellent."
The author, Dr. Richard B. Friedman, said letters of recommendation were useless and advocated doing away with them.
A brief JAMA essay by Dr. Henry Schneiderman in 1988 called for more openness in describing students but acknowledged that negative comments were often "the kiss of death."
He proposed a new system of categorizing medical student performance. Here are just a few examples.
@AmirGharferi suggested this:
"Dr.G, do you feel comfortable writing me a strong letter?"
"No."
"Ok, I'll find someone who is."
That works if the student is aware enough to ask, and the faculty member is honest enough to say no. In my experience, even the most marginal of students can find someone—in addition to the dean, of course—to write a good letter.
I am no longer involved in the process of selecting residents. I have no suggestions.
What is your solution to this problem?
20 comments:
Patient rankings? ROFL LOL I'm sorry I couldn't resist that one.
Ha. That's the answer I was looking for.
The answer is in your blog's heading, "Mean is least himself. . ." Unfortunately, we all know the student can "waive their right" to view the letter, but in today's litigious environment, writing a negative letter which may fall into the wrong hands--or be accidentally viewed by the applicant because the file was 'left on the corner of the desk by mistake', brings a new perspective to the problem.
I now write a letter that says, "Please call me to discuss the applicant if you are, indeed, interested in him/her." It isn't foolproof. But it is better than the alternative. Give me a TRUE guarantee the applicant won't see the less-than-perfect letter, and I'll tell the truth. But, no such guarantee exists and I believe letters are truly worthless.
Anon, that's a great point and one which I overlooked. You are absolutely right. A negative letter can come back to bite the writer.
It's a double-edged sword. I know of a case where a writer of a neutral letter of recommendation was sued successfully for failing to mention that the applicant had certain difficulties that caused problems at the new job.
Ban their use. They're obviously a low sensitivity, medium low specificity test for applicants ability and they require a lot of resources.
What do we usually do with bad and expensive tests? stop ordering them.
Part of the problem is that a letter of recommendation is just too general a format. Instead, what about a series of questions that must be answered: Has this applicant mastered all of the basic procedures? What percentage of advanced procedures has he/she mastered? Is he/she routinely on time? Polite and collegial with other members of the care team? Informative and caring with patients? Are his/her patient outcomes -on a par with, -better than, -worse than what you expect? How long have you known him/her? How many surgeries of his/hers have you participated in? Would you hire this person on your own clinical staff?
And so forth. You have to define what pieces of information are important in making a recommendation of value. If everyone has to answer the same set of questions, I'm guessing you would get enough data to make comparisons.
First anon, I doubt they will ever be banned. I think second anon is on to something. A more focused set of questions instead of nebulous characterizations might be better.
Sir,
It is interesting that the question of physician (resident) quality is unanswered in this forum, where in governing bodies such as CMS and ABMS/ABIM, the matter has been settled. This is yet again a matter for physician leadership, which has sadly been absent. Physicians can't identify "quality" amongst our own trainees and in our own ranks, yet regulators, administrators, and payors propose to identify it for us. If methods of identifying excellence among medical students are substandard for own purposes, what does that suggest about the broader movement to identify "quality" in medical care in general and physicians in particular? I am quite certain that I could describe a few excellent physicians, and many good ones. Quantifying or measuring that is another matter.
I'd be curious to know if you performed a post-mortem on the residency application of poorly performing residents, in order to identify distinguishing characteristics of applications among the poor performers.
Some papers have looked at possible factors associated with poorly performing residents. No consensus has been reached. To the best of my knowledge, no trait has been identified as foolproof in predicting failure.
My medical school class was >270 students large. On the surgery rotation we were put on large impersonal teams with limited facetime with faculty members (who were also always rotating on/off service), so when I needed letters for my residency applications I was essentially asking people who had no idea who I was. The letters were terrible (I remember my PD later telling me as much). So I agree that they serve little purpose. Perhaps they should be optional? Rather than saying "Submit 3 or 4 letters" applicants should be told "A letter of recommendation from a teacher/mentor can be added as an addendum to your application" so that if the person worked in a lab or charity organization they can get recognition for that without subjecting everyone to the other requisite useless letters. When I used to interview applicants I couldn't force myself to read through all of them. I think the best you can do interviewing an applicant is trying to have a genuine face-to-face conversation with them, which is why I always tried to steer interviews towards conversing about a topic, basically to see if the lights were on upstairs.
Josh, I agree that most of the letters can be difficult to read. They all sound the same--like "Fantasy Land" as I mentioned in the post.
They really aren't much help in trying to decide whom to interview.
I don't work in the health care field, but all we are allowed to say in a "recommendation" letter is: "Yes, he worked here, and No, he didn't steal anything."
Anon, thank you. I understand that in some fields what you say is true. We in medicine haven't arrived there yet. We still write long detailed letters full of useless information.
I needed two letters of recommendations to get into the two undergrad programs & agree with you about how totally useless they are. Mine were excellent but in order to get into a masters program I need an academic recommendation and like Josh my profs/instructors in my last degree didn't really know me. So I'm not sure what I'm going to do. It's a couple of years off if I do it (I'm 56, so I'm not sure if it'll get done). Like they'd say "she's a slacker, (insert faculty) would be better off without her". Maybe the whole point is to induce stress in those who want someone who actually knows them to battle said stress.
I'm not a fan of personal references for jobs either, but that's only because I stress out over who to ask-the political figure, a teacher, lawyer or someone I worked a lot with at church.
A good in person interview can give more information than a form letter (like some of those deans don't have form letters created!)
When I was writing letters of recommendation, I kept a thesaurus handy because I always ran out of adjectives.
Great post on an impossible problem. I've always tried to be honest in letters of recommendation, which hasnt always been easy. The most interesting instance was a resident who had surgical skills and dedication to surgery below what I expect of residents asked me to write a letter of recommendation for surgical oncology fellowship. I declined, telling her I would have to mention this in the letter. She demanded it, and as Program Director I felt I had no real choice. Imagine my surprise when I was later called to the Chairman's Office (Andy Warshaw) because Murray Brennan told him it was the most bizarre letter he had ever received (hearsay--Brennan didn't actually tell ME this). I told him the position I had been put in, and that the bizarre part is that he had never seen a truthful letter of recommendation before, and pointed out that I suggested a direct telephone communication in the letter, but had never received such. I was instructed to never write such a letter again, to which I stated we would have to agree to disagree. The hospital ethicist agreed with me. Said resident has gone on to a successful career in surgical oncology, alienating most of those she works with (particularly her juniors) and with mediocre surgical results, but an impressive CV and promotions. So it goes.
charlie
Charlie, thanks. That's a very nice example of all that is wrong with the system of writing recommendation letters,
that's good post.
How about those instances when the faculty member says to the resident, "Write a draft of your own recommendation letter and give it to me."
I've heard of that happening. There is no way to know because all you see is the end product.
Post a Comment
Note: Only a member of this blog may post a comment.