Does size matter? Surgery residency program size & board passage rates

A study in the May 2012 issue of the journal Surgery found that the larger a residency program is, the more likely are its graduates to pass the written and oral board exams of the American Board of Surgery on their first attempt. Over the years 2006-2011, 85% of residents passed the written exam and 83% passed the oral exam on the first try.

The authors show linear regression lines with positive and statistically significant correlations between increasing size of a program and its residents’ first-time passage rates. They say, “This important finding may influence the application patterns and rank lists of medical students matching into general surgery residency programs.”

They mention only one limitation of the study, which is that they did not have first-hand knowledge of how the board passage data were produced. They apparently could think of no other potential confounding factors.

I can think of two right offhand.

One, it is well-known that larger programs, which are more apt to be based at medical schools, attract smarter applicants.

From a paper about factors predicting board passage on the first try:

Significant objective predictors of successful first-attempt completion of the examinations were Alpha Omega Alpha status [the Phi Beta Kappa of med schools], ranking within the top one third of one's medical student class, National Board of Medical Examiners/United States Medical Licensing Examination Step 1 (>200, top 50%) and Step 2 (>186.5, top 3 quartiles) scores, and American Board of Surgery In-Training Examination scores >50th percentile (postgraduate years 1 and 3) and >33rd percentile (postgraduate years 4 and 5).

These are all directly related to the degree of intelligence of the resident. First-time failure to pass the board exams are much more likely to occur with graduates of small programs on the basis of the above observation alone.

The second confounder has to do with statistics. In his book, “Thinking, Fast and Slow,” Daniel Kahneman points out in Chapter 10 “The Law of Small Numbers” that “extreme outcomes are more likely to be found in small rather than large samples.” He gives an example of a large urn filled with the same number of white and red marbles. If one draws 4 marbles at a time and repeats the drawing many times, one is far more likely to have extremes of distribution, such as all marbles being the same color, than is one who draws 7 marbles at a time.

Imagine that all residents are created equal [Well, try.] and drawing balls which are all red represents a resident failing to pass the board exam. This will happen approximately 4 times more often if one draws 4 balls instead of 7. As you increase program size to say, 10 residents per year, the disparity is even greater.

The size of the program is not the issue. Large programs and small programs have been put on probation or discontinued. It’s not about the teaching either. A recent survey revealed that residents in non-university programs felt they got better teaching than those in university programs. [See my blog about this here.]

I’ll tell you a secret. It’s about the individual resident. I’ve had residents who I am certain could have passed the boards if just given a textbook and access to patients and operations. I’ve had others who no amount of teaching, prodding or remediation could salvage.