Monday, June 4, 2012

Does size matter? Surgery residency program size & board passage rates


A study in the May 2012 issue of the journal Surgery found that the larger a residency program is, the more likely are its graduates to pass the written and oral board exams of the American Board of Surgery on their first attempt. Over the years 2006-2011, 85% of residents passed the written exam and 83% passed the oral exam on the first try.

The authors show linear regression lines with positive and statistically significant correlations between increasing size of a program and its residents’ first-time passage rates. They say, “This important finding may influence the application patterns and rank lists of medical students matching into general surgery residency programs.”

They mention only one limitation of the study, which is that they did not have first-hand knowledge of how the board passage data were produced. They apparently could think of no other potential confounding factors.

I can think of two right offhand.

One, it is well-known that larger programs, which are more apt to be based at medical schools, attract smarter applicants.

From a paper about factors predicting board passage on the first try:

Significant objective predictors of successful first-attempt completion of the examinations were Alpha Omega Alpha status [the Phi Beta Kappa of med schools], ranking within the top one third of one's medical student class, National Board of Medical Examiners/United States Medical Licensing Examination Step 1 (>200, top 50%) and Step 2 (>186.5, top 3 quartiles) scores, and American Board of Surgery In-Training Examination scores >50th percentile (postgraduate years 1 and 3) and >33rd percentile (postgraduate years 4 and 5).

These are all directly related to the degree of intelligence of the resident. First-time failure to pass the board exams are much more likely to occur with graduates of small programs on the basis of the above observation alone.

The second confounder has to do with statistics. In his book, “Thinking, Fast and Slow,” Daniel Kahneman points out in Chapter 10 “The Law of Small Numbers” that “extreme outcomes are more likely to be found in small rather than large samples.” He gives an example of a large urn filled with the same number of white and red marbles. If one draws 4 marbles at a time and repeats the drawing many times, one is far more likely to have extremes of distribution, such as all marbles being the same color, than is one who draws 7 marbles at a time.

Imagine that all residents are created equal [Well, try.] and drawing balls which are all red represents a resident failing to pass the board exam. This will happen approximately 4 times more often if one draws 4 balls instead of 7. As you increase program size to say, 10 residents per year, the disparity is even greater.

The size of the program is not the issue. Large programs and small programs have been put on probation or discontinued. It’s not about the teaching either. A recent survey revealed that residents in non-university programs felt they got better teaching than those in university programs. [See my blog about this here.]

I’ll tell you a secret. It’s about the individual resident. I’ve had residents who I am certain could have passed the boards if just given a textbook and access to patients and operations. I’ve had others who no amount of teaching, prodding or remediation could salvage.

4 comments:

Brian Sabb said...

Hi,

This is a great post, very interesting information.

Here is one more potential confounder: The bigger programs have more recalls that they can study from to help them pass their written boards. They also may have more time off built into the system to study for the boards.

Thanks for another great post.

Sincerely,

Dr. Brian Sabb
www.linkedin.com/in/briansabb

Skeptical Scalpel said...

Brian, thanks for commenting. If you didn't see my post on board exams and recalls, here is the link http://is.gd/UdQST2.

John Falcone said...

Good morning. I am the first author on this paper. I appreciate the feedback on its content. I wanted to say that the initial manuscript only evaluated compliance with the RRC/ACGME program requirements. However, the peer reviewers wanted "more" than a cross-sectional study for Surgery. So, the linear regression analyses were added. The law of small numbers is an excellent point. Of note, omitting the smaller residency programs (<15 residents over the study period) did not change the results of the analyses. Finally, there are numerous limitations of the study. We thought of many. However, length restrictions of this "Brief Clinical Report" precluded a more in-depth discussion. I agree that it is ultimately about the individual resident; there are a number of manuscripts depicting USLME, ABSITE, etc as predictors of board performance. However, the "n" in this study depends on the number of residency programs. At the program level, there is really not a lot of published data on board performance.

Skeptical Scalpel said...

Dear John,

Thank you for reading my blog and commenting. I understand your explanation of why you modified your paper.

I was a PD for a small program for many years. I always thought the (then) 60% board passage rate on the first attempt discriminated against small programs. I wrote to the RRC to complain about it using the law of small numbers reasoning. They never even responded.

Post a Comment