On Twitter a while ago, a medical student asked me how surgical program directors select new residents. Then a discussion arose among some academic surgeons on the same topic. Someone suggested that medical school grades were the best way to tell whether an applicant would be a successful resident.
The fact is that we aren't really sure what the best
way to choose residents is.
First, here's what we really do.
USMLE Step 1, used by 37% of programs, was the most common applicant
screening criterion with USMLE Step 2 second at 24% and graduation from an LCME-accredited
US med school third at 15%. The least important criteria were previous research
experience and publications.
Final selection criteria were assessed using a Likert scale.
The number one factor was the interview followed by the USMLE Step 1 score,
letters of recommendation, and USMLE Step 2 score. The least important factor
by far was whether an applicant had done a preliminary year. Research,
publications, and a previous rotation at the institution also ranked near the
bottom. Class ranking, the dean's letter, and surprisingly, Alpha Omega Alpha
status were in the middle.
Responses consisted of 49% from university programs, 38%
from university affiliated hospital programs, and 13% from independent
community hospital programs. The average number of applicants per program was
571.
The problem is that proof of the value of the above methods
of selection is lacking. A paper from Academic Medicine in 2011 reviewed nine
studies of USMLE scores and resident performance and found no correlation
between those scores and the acquisition of clinical skills by students
residents or fellows.
A meta-analysis of 80 studies and over 41,000 participants
from the journal Medical Education in 2013 found that the USMLE Step 1 scores
and medical school grades were associated with better resident performance.
However, if you eliminate studies showing that better USMLE scores led to
better scores on in-training exams and passing licensing tests, only two
studies found that USMLE scores correlated well with subjective ratings of
residents.
What about grades?
The authors pointed
out that "These data could potentially be more useful to program directors
if grading systems across medical schools were standardized." I said the
same thing in a previous post.
A study of 348 categorical general surgery residents at six
residency programs on the West Coast looked at resident attrition or need for
remediation. The need for remediation was associated with receiving a grade of
honors in the medical school surgery clerkship and with a slightly but
statistically significantly lower USMLE Step 1 scores. For example, PGY-1
residents needing remediation averaged 225 on USMLE Step 1 vs. 232 for those
not needing remediation.
A major issue is the fact that we don't have good data on
the clinical performance of surgical residents or graduates of training
programs. I know from personal experience that a good USMLE score or a high
score on the surgery in-training exam had a "halo effect" when it
came time for faculty to evaluate overall resident performance.
According to the Wall Street Journal, some businesses are
asking job applicants of all ages "to provide SAT or ACT scores, results
from graduate-school entrance tests and grade-point averages along with their
work history."
Google, a successful company by any measure, does not care much
about grades. Here's what Laszlo Bock, senior
vice president of people operations, had to say in the NY Times about employee performance: "One
of the things we’ve seen from all our data crunching is that G.P.A.’s are
worthless as a criteria [sic] for
hiring, and test scores are worthless — no correlation at all except for
brand-new college grads, where there’s a slight correlation. Google famously
used to ask everyone for a transcript and G.P.A.’s and test scores, but we
don’t anymore, unless you’re just a few years out of school. We found that they
don’t predict anything."
But the relationship between college grades and performance
at Google isn't the same as the relationship between med school grades and performance
as a surgeon. Or maybe it is.
I suppose one could argue that applicants for residencies,
who are recent graduates of medical schools, would fall into Google's
"slight correlation" category.
I know one thing—I don't know how to
select applicants who will become good surgeons. Do you?
37 comments:
Wonderful review of the literature.
We used USMLE, transcript, letters of rec (much more weight if from people whom I knew), and then the interview. The interview can be used to see if the person had the gene.
But, ultimately it was the top group, the lower group and the 'vast number inbtwn'.
Best addition was having candidates go out with senior residents without faculty. Empowered residents, great feedback
I'm not sure I would have passed all the personality and technical exercises these papers promote. I got more confidence from my residency than anything prior.
Lastly, I am unsure if this current process is any different or better than faculty recruitment. You get the best you can, and hope it works out! Most of the time it does, sometimes it blows up
I'm in private practice, so I only saw the interview process from the standpoint of a resident. Our chairman made it clear that most of the applicants who came were academically capable, and our role in having lunch with them, showing them around the hospital, and spending most of the day with them was essentially to see how they would fit in with us if they matched. I don't know how much impact our opinions had, but the approach made sense to me.
Michael, your experience sounds very much like mine. See below for my reaction to your comment about applicants spending time with residents.
Peter, The interviews and having the applicants spend time with the residents is helpful. I used to do it that way too. The problem is that programs have a limited number of interview slots. If you get 500 applicants and only have time to interview 50 or 60, you have to screen them somehow to decide who gets an interview.
Recently, I did a research yr at a top univ hosp after graduating. I got to see the interview process and who they selected before the match. Only 2 spots, I'm talking about ENT though, not gen surg. They already had one spot reserved for a relative. They still interviewed 40 people for the one other spot. I felt sorry for the students flying down and getting hotels. When it comes down to it, from what I have observed this is a club. If you know someone in it and they make a phone call for you, it makes a big difference given the fact you have the scores, the BS recommendations, and all the extras. Lets face it everyone applying to surgery in 2014 is pretty good applicant. They have that desire to be a surgeon. We know what medicaid pays for an appendectomy, it's no secret. So if you want to do this, you got to love it. Otherwise, I know many of you senior attendings might reconsider if you were applying today.
Dr. Scalpel,
One thing I've noticed about admissions to competitive colleges and medical schools is that they look for the "odd one out". My alma mater, MIT, would frequently accept individuals (some with lower scores) if they did something unique like make a music video about MIT, or had a big personal accomplishment. I always thought of it as the "that kid" effect: admissions officers we're simply more likely to remember "that music video kid" rather than the generic high score applicant.
Is this true in your experience? Did you look for the quirky applicant or the conventional one?
More generally, I think the problem is that we have no really good way to measure the quality of a surgeon to begin with. Perhaps that Michigan bariatric surgery videotaping study showed the way forward, but until we know how to objectively tell a good surgeon from a mediocre one (both technically and with regards to decision making), how can we know if we're picking residents correctly? Especially if as you say that attending judgement of resident skills is biased by high test scores.
Respectfully,
Vamsi Aribindi
Another question I have is what is the difference btwn an exemplary resident surgeon and a bad? Forget about selection. What do I need to do to prove myself once I start intern yr.
To the Anon above: Intern year, work hard, never complain, and read everything you can.
Scalpel, the process seems similar to college football recruiting. Lots of 4 and 5 star prospects fade into obscurity when they get in the big pond, and sometimes the walk on becomes an All American. Maybe it's all a crap shoot.
Personally I think the algorithm is all wrong. While you do need people who are at least average performers on standardized tests and have excellent clinical grades (which I think are more important than test scores), it would be more helpful if there was a way of ascertaining an applicant's emotional intelligence.
In surgery, you don't really have to be smart or technically gifted. You simply have to have the drive to become a surgeon, emotional intelligence to suss out when to talk/when not to/how to get along with others, and the ability to do many things you don't like doing. I've often found that the best residents to work with are the ones who have an intuitive understanding of the system (something that can't be taught discretely), combined with a strong understanding of the fact that the world isn't fair and sometimes you just need to do as you're told.
It's much easier to teach and train someone who is malleable and intuitive and wants to work hard. People who have had other careers, who like to argue, who don't "get" social situations/cues -- they seem to have the most trouble.
Wow, you nailed it Anonymous above. I agree with you totally, the algorithm is totally wrong. As a foreign medical grad, trying to knock on the door to get my opportunity. I have done many observerships in many top teaching hospitals. I notice a huge difference in the level of thinking btwn a med student from where I went to med school in comparison to here. First attitude, the students here feel entitlement and power. Where i went to school this is common knowledge, "strong understanding of the fact that the world isn't fair and sometimes you just need to do as you're told." I am afraid this attitude carries on as a resident and as an attending. In terms of pathophysiology and basic concepts I feel american students are lacking most definitely. We place too much emphasis on boards and not a proper foundation. Yeah, memorize a review book for the boards will get you a great foundation. Another issue I have, the teaching here sucks. I have been in so many OR cases, there is no dialogue, no discussion, students just stand there holding a camera. I can get my 3 year old to do that. Then there is another extreme of an attending riding and pimping a student with questions. Stressing them out until uhh uhh I cannot think anymore. Where I went to medical school we did not have these stupid multiple choice tests, we had oral tests, you had to learn how to discuss and talk like a doctor at a very early age. I'm just surprised that the system has not changed here, I feel after observing the student, the resident, and the attending, this is a viscous cycle. I do understand the OR can be stressful, but shoot for a lap coly, there is still a lot you can discuss with the student. Often the OR is silent, then the student goes from camera holding to filling out HNP. I always joke with my colleagues back home, American students they write the best HNP's. Can we step up the medical education here please, then expect to pick out the good surgeon. Maybe we need to look at the education system. Are the medical schools doing their job or just pushing people through the system. Then you wonder why you get 4 or 5 star recruit who turns out to be a bust. I question the learning process here, and I have hard to believe a lot of people here have allowed it to not change much. Really think about it, how has medical school or teaching in and out of the OR changed since when you were in school. If you teach like crap, you get crap. That is my opinion.
Sorry for the delay in responding.
First Anon, yes, deals are occasionally made. I always felt that a personal letter or phone call from someone I knew was important. Believe it or not, I've been burned on that too.
Vamsi, I think we are looked for quirky applicants. Some were good residents and some were not.
Artiger, those are good basic recommendations, but residents have still got to pass standardized tests. Programs are penalized if fewer than 65% of their graduates don't pass both parts of the American Board of Surgery exams.
Second Anon, As I said on Twitter, It is hard to tell from an interview who will be a hard worker.
Third Anon, I agree with a lot of what you said. Here's a link to a post I wrote 3.5 years ago: http://skepticalscalpel.blogspot.com/2010/08/medical-school-and-surgery.html
It is similar to your comments.
dear anon FMG. are you FMG or FMG-A? you sound like a FMG-A who is bitter about being a prelim or having to do extra years with no guaranteed categorical postion. I find it quite contradictory when you claim that AAMC school students getting ridden/pimped but you guys getting oral exams are much better. to me pimping is a form of oral examination.
when I was interviewing for surgery, I heard one general surgery program had its 30 applicants in a room and everyone was interviewed briefly in front of everyone else by the chairman and one of the questions to one of the applicants was "why did you attend this foreign medical school?"
I find it hard to believe that an American college graduate would preferentially attend a foreign medical school. I understand your enthusiasm for your school and applaud that but its not going to sell to students applying to medical school. I believe that for most people applying to medical school they value location of school, reputation of school (brand name), tuition costs. Out of 10 kids that get accepted to New York Medical College (poor reputation + high cost) and Ross (high cost), I'm sure most kids would be happy to attend NYMC.
I agree its not fair that FMG is a huge strike against you for entering a surgical residency categorical spot. At some point I lost my pedigree and its hurt me the rest of the way as well. I went to Ivy League college, state medical school but ultimately when I didn't do that great in medical school, I was forced to go to a community type surgical training program and that hurt me when it came time to applying for surgical fellowships. its the name of the game.
Ultimately I imagine that you partied too hard in college or are just a late bloomer and ended up at Ross/St George or whatever but got the 250/260 USMLE with stellar recommendations you'd do just fine. I would love to see a study comparing Ivy League medical school grads with 210-220 usmle vs FMG-A with 250-260 and looking at their attrition rate/board pass rate/absite scores. I don't have any idea who would be better. I just know that the surgical programs I have been to would easily choose the ivy grad with lower scores.
Thanks for your comments Pamchenko. My situation is quite different. I was not born here in USA. I immigrated here at a later age after I completed medical school. My challenges of getting a residency spot have nothing to do with the fact that I'm an FMG. There are actually a lot of surgeons in this country who are surgery residents. I believe you are referring to Carribbean medical schools, so your comments would pertain to those students. As far as pimping goes and asking questions, there is usually one attending in every department who does it. Oral exams overseas are really different pamchenko, and we do this in the first two years of medical school. I was referring to talking out concepts, not taking multiple choice tests. I was pointing out the difference in teaching and learning. I appreciate next time you not assume my background. I did not attend ross/st george. Everyone has their own challenges, just because you were born here does not mean the system should be easier for you. Imagine immigrating to my country and trying to get a surgical residency spot. Put yourself in my shoes. Anyway, I don't want to get off topic. My discussion is on the difference in teaching. We concluded there is no good way to judge who will be a good surgical resident candidate, but I question the quality in teaching. I'm not saying my school or country is perfect, we have our own problems too. I just think the American students overall go into a residency with a bad foundation. Last, if I don't get a surgical spot in this country that is okay. I am a surgeon in my own country and I am happy where I am at in my life. What is sad about your comments too. There are tons of neuru surgery residents who are FMGs. They completed a residency overseas, and now to make a better life for themselves they are completing a second residency in the United States in neurosurgery. I know programs who pick residents from other countries, there is a reason for it. Thanks, I'll be a lot better off than the Caribbean students, trust me.
Dr Pamchenko, I just did some research. The students of ross/st george do rotations in USA in their last two years. My comments above talk about teaching in another country. There is a huge difference. It seems carribbean students are like a cheap/half product of the USA system. That scares me even more about our future. I was fortunate to go to medical school out of high school as that is how it is done in my country. You attend for 6 years, then you do a residency. Now when I am saying how different the teaching is, why would you assume carribbean school. Caribbean school is a cheap copy of every american medical school. They try to make it so similar to the American system so that their students have no trouble when they go through the match process. In addition, I just saw the match list of ross/st george, they all are primary care docs (Family med, Internal, psyc). Furthermore, my medical school was the IVY league of my own country. For safety and a better life style I only choose to come here. I don't think I am any different than you Dr. Pamchenko, other than my primary language is not english. I still communicate really well as I went to english teaching schools. My education is pretty equivalent to yours, and I get a good feeling I will be a colleague of yours one day.
The reality is that for American citizens, American medical schools > D.O. schools > Caribbean & Mexican med schools, and that hierarchy will follow most of them for their professional careers. True, maybe you slack off in college, and there is no difference in clinical practice 20 years down the road.
Speaking of a "halo" effect for residents, it is not really test scores or med school records; it's the perceived prestige of your med school.
I've been doing locums for a few years, and in terms of clinical competence doctors are pretty good everywhere. But, in one hospital in one of the richest counties in America, every surgeon had undergrad or med school or residency credentials at Harvard, Johns Hopkins, UCSF, MIT, etc. On the other hand, in one large med center in a Central Valley (CA) city, most of the surgeons are Iranian and Indian immigrants, plus many D.O.'s and Caribbean grads.
Right, let's stay on topic. Forget about what med school someone went to, what undergrad, what country someone is from, whether your DO or MD. Who really cares. In the end, the most important question is Anonymous above, if I get into a car accident and need trauma care, will the surgeon waiting for me at the hospital be the best possible person to keep me alive. I don't care what school you wear on your white coat, but are we choosing the right people to hold that scalpel. Are we preparing our students to be a resident. Let's talk about the real stuff. What needs to be done differently in medical school to help students grow. Above someone wrote differences in usmle scores, who cares. I really could care less if you got 99 percentile, but if you cannot manage your patient on a daily basis, forget it you're out. PD's kick him or her out! So let's take this a step further, how can we make things better education wise? We all know that statistics, the usmle score BS, and if you went to this school this is your chance for plastic, yada yada. We all know this already. I am with skeptical in his post down below, he absolutely nails it, http://skepticalscalpel.blogspot.com/2010/08/medical-school-and-surgery.html
The discussion is why isn't it clear cut selecting a surgical resident. We always look at the student and say what is deficient in them, but what about the system as a whole. Can we not be better than what we are already? Come on surgeons, am I asking for a lot, you guys are perfectionists. Do we need to restructure our curriculum. Please save me the I went to this ivy league bs, that does not say whether you are an exemplary physician.
What would you say of the prestige of one who gets a college degree, gets a masters in physiology (or whatever it is that is a 1 or 2 year masters to look good in medical school) and then gets in a low ranked school but comes out a surgeon? They seem to be a good surgeon but the rest ... well ... there's a lot of prayer there.
As I said, I find no difference in the clinical skills of surgeons no matter what their background. It is an unfortunate truth that the background will affect where they practice. It is wrong, but that is the real world.
It is not dissimilar to how some kid's college acceptance at age 17 or 18 has a huge influence on his socioeconomic status for life. It is not fair, but it is the reality.
Not picking surgical residents, but gaspassers. Still, I have 2 comments:
1) When I worked at The University (yeah, caps are intentional), I sat on the "Residency Selection Committee". All applicants were qualified, all had great recommendations, and all had great grades and experience. Yet, despite these, we still picked a bozo every now and then. I always felt inadequate sitting on that committee. I had no idea what I was doing.
1) When I worked in private practiceI would interview job prospects. All applicants were qualified, all had great recommendations, and all had great grades and experience. Yet, despite these, we still picked a bozo every now and then. I always felt inadequate doing these interviews. I had no idea what I was doing.
It's a crap shoot. Do you feel lucky, punk? Well, do you?
A large number of Fortune 500 CEOs did not graduate from Ivy League schools. In my 36 years as an attending surgeon (most of which pre-dated the Internet), I can recall being asked where I went to med school or where I did my residency fewer than 5 times.
George, as usual, your comments are on target.
here's something they taught me in MBA school: the Big Five Personality test has a measured dimension called conscientiousness. it predicts job performance across multiple professions. I found one (weak) paper about it in a brief online search. I've used conscientiousness as a factor in bringing people on board for startups and it seems to work VERY well.
http://www.sajip.co.za/index.php/sajip/article/viewFile/88/84
maybe try that?
David, thanks for commenting and the link to the paper, which unfortunately is a difficult read for me. The statistics are unfamiliar. I do agree that conscientiousness is something we seek. How do you test for it? Keep in mind that we would have to test a few hundred applicants every year.
You might be interested in this recent Atlantic piece (http://www.theatlantic.com/magazine/archive/2013/12/theyre-watching-you-at-work/354681/) which a reader emailed me.
It's interesting, because I can tell within 24 hours of working with another person - be it medical student or resident - whether I think they "get it" and would be/are an excellent resident vs someone who doesn't.
Medical students who "get it" are the ones who intuitively understand what to do and generally have already done things without being asked to do it. They've already gone into the supply room to get extra dressing supplies when you turn around at bedside to ask for an ABD pad. They've already greeted the patient in pre-an and read up on the case when you tell them which bed the patient is in. They've already written discharge summaries and asked you if there's anything else they can do before you've had a chance to run your own list. All you need to be a good surgery resident is to be hardworking, efficient, moderately intelligent, good at prioritizing, and ready to accept hierarchy/humility to learn.
Excellent residents I've worked with are ones who go see the consult and look up the information without trying to argue about who it should go to. They sign out succinctly but thoroughly, citing pertinent information and leaving out the rest. They anticipate errors in advance and double-check work to avoid them. They make checklists and efficiently work through them each day. They complain little. They are willing to help other people out, even if it's at their own inconvenience. They want to practice, to learn, and be in the OR. It doesn't matter if they're brilliant or technically gifted. A residency training program will train you to think a certain way and operate a certain way.
These are all things you discover very quickly after working with someone for a short period of time. I would choose someone who did an audition rotation or went to medical school at my institution or whom I had worked with in the past over someone with pristine scores and grades. In practice is far more valuable than on paper. And interviews are pretty much crap because they don't tell you about the work ethic, ability to compromise, ability to prioritize, or efficiency of a person.
As an aside, I'm an integrated vascular surgery resident.
Hope, thanks. You made a number of good points. The program directors apparently don't agree with you about audition electives. I've been burned on that a couple of times myself. As I said earlier, anyone can look good over a few weeks. That's a lot different than a year or five years.
Another problem with audition electives is that an applicant can only do two or three at most, and programs can only accommodate a small number of students on such electives. It's not the answer.
I think that if you're an attending it's much more difficult to get an idea of what a student is "really" like. Residents and other students have a much better idea. Even when medical students I've been in the company of have really shone in front of attendings and been viewed as the bees knees, around the residents it was obvious whether it's an act or not. So yes, I think from an attending's point of view, the audition is less telling.
for about the last five years, i've been very interested in how we select residents. i'll do a little write up about some organizational behavior stuff i learned with some citations and will link back you your blog here. wanted to do it anyway one day and this discussion got me thinking.
Hope, you are right. An attending doesn't really know a student like a resident does. Of course as a PD, I am looking for more than just a student who gets the labs on time and is available.
David, thanks. I look forward to it.
Ok here's that post. Hopefully you find the info on the Big 5 model useful.
http://www.surgicalbusinessmodelinnovation.com/decision-analysis/lets-choose-residents-differently/
Very interesting.
Since there is no metric on what makes a good surgeon, or doctor, except for the few at the very bottom, it is not surprising that there is no good criterion for picking residents.
How about motor skills? Being good with your hands is obviously useful in surgery, but also in many other specialties: anesth (airway, blocks), GI (scopes), pathology (biopsies), cardiology (caths), derm (biopsies, injections), FP (office procedures).
There is a lot of purely mechanical stuff that doctors do, so why not test for psychomotor skills at the med school or residency stage?
Here's a link to an article about a recent paper which shows that a manual dexterity test (soap carving) was not effective at predicting surgical performance. It on Medscape, which requires registration but it's free. http://www.medscape.com/viewarticle/819921
There's much more to being a surgeon than maual dexterity too.
I certainly agree there is much more than motors skills needed to be a good surgeon, or any other physician. But, since there is little value for board scores or grades or any other cognitive measure, why are psychomotor skills ignored?
Applicants for general surgery and ortho likely self-select for manual dexterity. But, from my personal observation that is less true for other procedural specialties, including surgical areas.
I hope I didn't mislead you. Psychomotor skills haven't been ignored. They just don't help predict who is going to be a good resident. I hope you read the link in my previous comment.
There is a story about a program that used to have applicants put a model airplane together. The program no longer does that.
In my med school surgery program, the faculty always said,"you can teach a monkey to operate, but you can't teach them why." This mitigates the importance we students used to ascribe to "good hands"--yes, many surgeons are elegant in their movements, but not necessarily better technically. Also, it's the decision whether/when/on whom to operate which is critical--how we teach or assess the potential for that, I still don't know.
Anon, thanks for commenting. I agree with your faculty and wrote about the monkey over two years ago. http://skepticalscalpel.blogspot.com/2011/11/i-could-teach-monkey-how-to-operate.html
by the way does anybody know what was andrew vesalius,s USMLE score ??????????????
No, but he must have been a pretty good student because "The day of his graduation he was immediately offered the chair of surgery and anatomy (explicator chirurgiae) at Padua."
Psycho-motor skills are necessary but not sufficient.
Post a Comment
Note: Only a member of this blog may post a comment.