Thursday, February 2, 2012

Cheating, written board exams and “recall” questions

CNN "Exclusive: Doctors cheated on exams"

A recent dust-up about radiology residents accessing and memorizing questions from previous board examinations generated 1361 comments on CNN alone from physicians and others. Many more people took to Twitter and vented. Some said it wasn’t really cheating because the test-takers had to memorize the answers. Some said the radiology boards didn’t really sort out who was going to be a good radiologist anyway. Some said the questions were on non-clinical topics like physics. Some were highly indignant that such a thing could happen. I saw many comments suggesting the board simply write a new test every year.

The stockpiling of "recall" questions by residents and programs goes on in all specialties. I know it does in surgery.

It would be difficult to create a completely new written exam every year. For several reasons, that is not a practical solution.

My understanding of the way the American Board of Surgery handles questions is this. Questions are recycled because they must be validated by analyzing them after they are used. In surgery, questions may appear on the residents' in-training exam one year, the re-certifying exam the next year and Part I of the boards the following year. Where appropriate, questions are also recycled through the subspecialty exams such as critical care, colorectal and others. As it happens, each test contains some reused questions and some brand new questions.

The board assesses certain things about each question such as do junior residents do as well or better on a question than chief residents? An ideal question would be one in which the percentage of correct responses increased as the training level of resident increased. They also look at whether the questions are framed correctly. For example, a question which generates two or more answers that are chosen by similar numbers of test takers may be ambiguously worded. Questions that have unusual patterns of response or ambiguous answers would be reformatted or discarded, and they would not be counted for the test that resulted in the unusual answer patterns.

A completely new test would contain quite a few questions that would have to be discarded if they were not validated by prior use.

Another problem is that writing good questions is very difficult. Most educators feel that a five-answer multiple choice question should have one correct answer, one plausibly correct but not exactly correct answer, two wrong answers and one really, really wrong answer. The correct answers need to be found in commonly used textbooks. And there are just so many questions that can be asked. Try writing a few questions. You’ll see how hard it can be.

I am conflicted about whether possession and use of copyrighted material from the boards constitutes cheating. Strictly speaking, I suppose it does. But where do you draw the line? When I was a residency program director, my trainees would often ask me what I thought the correct answers to some of the questions were. If I told them, would that be cheating? If they remembered a few of the questions and discussed them among themselves, is that cheating? What if a resident remembered a question and looked up the answer herself? Is that cheating? Or is it learning?

Here’s the good news. I’m not a residency program director any more.

What do you think?

6 comments:

jstgeorge said...

I know all the traditional reasons we put stock in this method of testing. But in my career, from the MCATS to my board certification, I can't remember a time when I walked out of a standardized test and said to myself, "that was great, I'm so going to be a better doctor for this." I wouldn't care if there was no harm, but every hour I spent studying information that often was irrelevant to my daily practice, was one less hour I used to practice an important technique or read another journal article or had a discussion with a colleague about a tough case and how to manage it. Mostly I resented the governing bodies who made money off of me and gave me no educational value, only their "blessing" to be a doctor....

Skeptical Scalpel said...

jstgeorge

Thanks for the comment. Well said.

புருனோ Bruno said...

My Opinion

Looking at previous year question papers and preparing answers for them is not cheating

Getting the present question paper before exam (by some fraudulent means) and preparing answer for it is cheating !!

Skeptical Scalpel said...

Bruno,

Thank you for commenting. I'm not sure everyone would agree, which is the issue after all.

Anonymous said...

What no one in all the discussions I have seen has mentioned is the amount of money the boards, and by extension, some of it's members (not to mention an entire industry built around CME, MOC, etc.) make on credentialing and MOC. And although touted as a great good for the patients/consumers, it is in fact based on limited evidence based data as to how valuable it all is (besides the aforementioned money aspect of it) and shrouded in conflict of interest (mainly based in the aforementioned money aspect of it).

Just because in the test-taking rules someone stated that this is cheating and it is copyrighted material, and the board examiners can then call it "cheating" and "copyright violations", and whatever, and of course the media takes it at face value, runs with it, and paints the doctors as having "cheated" and possibly broken the law, does not necessarily mean it's either of those things.

Where go you draw the line? Is learning from a book considered cheating? Does that give you an unfair advantage over your peers who have not read? I posit that someone may use that argument. In fact, I just did.

Books (and journal articles) are also copyrighted material and so by extension, all modern medical knowledge has at one point or another been copyrighted in one form or another. So, aren't we all guilty of breaking copyright law when we, for example, take notes, then copy and/or share those notes with others? Aren't teachers violating copyright law when they teach from books, etc.?

Of course they're not. Because, luckily there is such a thing as "fair use". And I would argue that creating and using "recalls" falls under the rubric of "fair use", as long as you don't make a concerted effort to collect the questions verbatim ***AND*** selling them for profit.

It's interesting to see this debate unfold, but it would be nice if someone took a more broad view of the issue.

Skeptical Scalpel said...

Anon:

Thanks for the thoughtful and thought-provoking comments.

Post a Comment