The test was designed to assess residents' knowledge and give them an idea of where their studying should be focused. However, many general surgery program directors (PDs) use the test results in other ways. Some impose remediation programs on residents with low scores and even base resident promotion or retention on them. Some even demand that all residents in their programs maintain scores above the 50th percentile.
The Residency Review Committee (RRC) for Surgery frowns upon these practices and states in its program requirements (Section V.A.2.e) that residents' knowledge should be monitored "by use of a formal exam such as the American Board of Surgery In Training Examination (ABSITE) or other cognitive exams. Test results should not be the sole criterion of resident knowledge, and should not be used as the sole criterion for promotion to a subsequent PG [postgraduate year] level."
The problem for program directors is that the RRC also mandates (Section V.C.2.c) that "as one measure of evaluating program effectiveness" 65% of a residency program's graduates must pass both the American Board of Surgery's Qualifying Examination (written) and Certifying Examination (oral) on their first attempts. I have said before that the "65% on the first attempt rule" does not seem evidence-based.
Does performance on the ABSITE predict performance on the board's examinations?
A recent paper by the staff of the American Board of Surgery states, "Although the ABSITE does not have a direct effect on board certification, it has been shown to be predictive of ABS Qualifying Examination performance." The authors cited three references.
The best is a 2010 Archives of Surgery paper that analyzed 607 graduates of 17 programs from the western US. It found "On multivariable analysis, scoring below the 35th percentile on the ABSITE at any time during residency was associated with an increased risk of failing both examinations (odds ratio, 0.23 [95% confidence interval, 0.08-0.68] for the qualifying examination and 0.35 [0.20-0.61] for the certifying examination)."
Note: The board's paper found that ABSITE scores do not correlate with passing the certifying (oral) exam. This makes sense because the oral exam is more about judgment and situational thinking than recall of facts.
A systematic review of 26 papers, published online in the Journal of Surgical Education, showed that "Structured reading programs and setting clear expectations with mandatory remedial programs were consistently effective in improving ABSITE performance, whereas the effect of didactic teaching conferences and problem-based learning groups was mixed."
However, it's not so simple. Structured reading and mandatory remedial programs will only work if the deficient resident is committed to succeeding, an attitude that is not always present. [See "grit."]
A brilliant post of mine from two years ago pointed out that program size has a lot to do with being able to maintain a better than 65% board passage rate on the first attempt. Using a simple statistical fact, I explained why smaller programs may be much more likely to fail to meet the 65% standard.
A resident who, despite attempts at remediation, has single digit ABSITE percentile scores over two or three years creates a serious dilemma for the director of a small program. Should the PD keep the resident in the program which can ill afford to graduate a resident with a high risk of failure on the written board examination or dismiss the resident and try to find a competent replacement from a very small pool of available candidates?
Having been there, I can tell you it's not an easy decision.
11 comments:
I can see how it would be a dilemma. A couple of years ago, we had a resident held back from being promoted to chief because of her ABSITE scores. Although she received warning that she was below the 30th percentile two years in a row on the exam, I don't think it was made clear that her promotion was in jeopardy. Having worked with this resident, she was very capable and competent and was never, in a clinical situation, someone I expected would be held back.
I think that this issue can be remedied (or at least ameliorated) with early identification and support for these students. Students who perform badly twice in a row on the ABSITE should be given extra support by the department in terms of preparations/sending them to review course/assigned question sets etc.. I think that in a way it becomes the program's responsibility to help a competent resident with poor test-taking skills, as long as the resident is willing to put in the effort. I did very poorly on my in-service exam the first year...but I can honestly say I didn't study much in advance. The last couple of years I have done much better, as a result of doing questions on a regular basis for months before the test. My program helped me by reimbursing me for extra question banks. I think that if the residents with these issues are identified early on and given some attention, it will be less likely that the test will be the SOLE criterion that holds them back from being promoted.
Setting a minimum ABSITE score to be promoted into the next year of surgery (say, 32nd percentile) will inevitably cause a (32%) sacrifice of any given class of residents, creating a de facto pyramidal system - something we were supposed to have gotten rid of a long time ago.
Those falling under the mark can be remediated (especially if they have that grit) and can still become exceptional surgeons in practice; however, the stigma of being left behind may cause many to choose other fields, such as anesthesia and emergency medicine as they have in the past. In a time when we're already scarce on surgeons, we shouldn't be scaring people who are already in it away, and giving talented students yet another reason to choose a procedural Internal Medicine field over ours.
Hope, I once sent a resident to a week long review course. The other residents let me know that they were furious. They saw the slacker as getting a week off while they had to cover for him. I never did that again. It didn't work anyway. He did poorly again.
Anon, I agree that a certain threshold suchas the one you chose, 32%, is impractical because as you point out, 1/3 of the residents in the country will be below it. However, pass rates on both parts of the boards are at best 75% to 80% [http://www.absurgery.org/default.jsp?statgeneral] and have been that way for many years. It's kind of like the game "Old Maid." As a program director, you don't want to be the one holding the bag.
Only marginally related, but I was wondering if you were aware of the lawsuit being filed against the former chairman of urology at Ohio State:
https://drive.google.com/viewerng/viewer?a=v&pid=sites&srcid=d3djb2xwYS5jb218d2lsc29uLXdpbHNvbi1jby1sLXAtYXxneDo3ZGExMDBkNjY1NTNmMjlh&u=0
I noted that the American Board of Urology refused to allow the graduated resident to sit for the board exam on the basis of his chairman's negative review. While this may represent a case of retaliation, is there a similar mechanism for the ABS Qualifying Examination by which a program director could effectively shut the resident out of taking the test?
Anon, thanks for the link. That is quite a story. I urge everyone to read the document you linked to. I wonder why no news media even in Columbus have not picked up the story. The only thing I could find was what looks like a press release from the resident's law firm (http://www.digitaljournal.com/pr/2339759).
Anon, I tweeted the link that you posted in your comment. It has attracted many retweets and comments.
I forgot to answer your question. Yes, PDs have to attest that graduates of their programs are worthy. I would hope that the American Board of Surgery would handle it better if such a situation occurred in general surgery.
I just read through that whole lawsuit document. While it's important to remember that there are two (or more) sides to the story, I'd say that if the complaint is even half true, then someone needs considerable psychiatric attention. And I don't mean the residents.
I remember chilling stories about the old days, but I thought the malignant attendings from that era had either passed away or retired (out of disgust for the new rules). Present company excluded, of course.
Artiger, even if it's only 10% true, it's pretty bad.
When I scored poorly, my program director said to me, "You have to improve. I don't want everybody to think we hired a dummy."
There was no further help.
There were no organized conferences.
Our attendings, rarely, if ever, gave conferences.
There were no conferences to send residents to.
The "Selective Readings" in Surgery were photocopied and placed in our mailboxes (copyright violation).
There was no SESAP purchases or funds allotted to such purchases.
I believe my score was more reflective of the QUALITY of my surgical EDUCATION in this program, than my skill level. I found learning about breast disease, in a program where I never even saw a breast biopsy, until I was a 3rd year resident, to be exceedingly frustrating.
The question is: Why is my poor score used to evaluate and or promote me and NOT a signal to judge the adequacy of a program, its case load, its attendings teaching skills, and all those other factors that are responsible for turning out quality, competent, confident surgeons--instead of just telling the resident to "go read."
Can they fire a resident if he or she gets a one digit score for the first time.I mean is it fair legally??
To the 12/3/14 anon above, I'm sorry I never responded to your comment. You have some very legitimate issues. Unfortunately, I don't see easy solutions. I hope your ABSITE score was better this year.
To the anon from today, I am not a lawyer. First of all, the RRC says the ABSITE should not be the sole determinant for promotion or dismissal. Secondly, the human resources department of your hospital undoubtedly has rules about firing employees. There's usually a process that must be followed. You need to look into this with an HR person. Please let me know how this turns out. You can email me at SkepticalScalpel (at) hotmail.com with more details and/or questions.
Post a Comment
Note: Only a member of this blog may post a comment.