Cheating, written board exams and “recall” questions

CNN "Exclusive: Doctors cheated on exams"

A recent dust-up about radiology residents accessing and memorizing questions from previous board examinations generated 1361 comments on CNN alone from physicians and others. Many more people took to Twitter and vented. Some said it wasn’t really cheating because the test-takers had to memorize the answers. Some said the radiology boards didn’t really sort out who was going to be a good radiologist anyway. Some said the questions were on non-clinical topics like physics. Some were highly indignant that such a thing could happen. I saw many comments suggesting the board simply write a new test every year.

The stockpiling of "recall" questions by residents and programs goes on in all specialties. I know it does in surgery.

It would be difficult to create a completely new written exam every year. For several reasons, that is not a practical solution.

My understanding of the way the American Board of Surgery handles questions is this. Questions are recycled because they must be validated by analyzing them after they are used. In surgery, questions may appear on the residents' in-training exam one year, the re-certifying exam the next year and Part I of the boards the following year. Where appropriate, questions are also recycled through the subspecialty exams such as critical care, colorectal and others. As it happens, each test contains some reused questions and some brand new questions.

The board assesses certain things about each question such as do junior residents do as well or better on a question than chief residents? An ideal question would be one in which the percentage of correct responses increased as the training level of resident increased. They also look at whether the questions are framed correctly. For example, a question which generates two or more answers that are chosen by similar numbers of test takers may be ambiguously worded. Questions that have unusual patterns of response or ambiguous answers would be reformatted or discarded, and they would not be counted for the test that resulted in the unusual answer patterns.

A completely new test would contain quite a few questions that would have to be discarded if they were not validated by prior use.

Another problem is that writing good questions is very difficult. Most educators feel that a five-answer multiple choice question should have one correct answer, one plausibly correct but not exactly correct answer, two wrong answers and one really, really wrong answer. The correct answers need to be found in commonly used textbooks. And there are just so many questions that can be asked. Try writing a few questions. You’ll see how hard it can be.

I am conflicted about whether possession and use of copyrighted material from the boards constitutes cheating. Strictly speaking, I suppose it does. But where do you draw the line? When I was a residency program director, my trainees would often ask me what I thought the correct answers to some of the questions were. If I told them, would that be cheating? If they remembered a few of the questions and discussed them among themselves, is that cheating? What if a resident remembered a question and looked up the answer herself? Is that cheating? Or is it learning?

Here’s the good news. I’m not a residency program director any more.

What do you think?