More on “Publish or Perish”

Bonus: What is a receiver operating characteristic (ROC) curve and how do you interpret it?

God knows I’ve written and managed to get published some downright awful research papers in my time. I was driven by the same forces that are at work today. “Publish or Perish” was very real for me as I needed to grind out papers to keep my general surgery residency from going on probation for inadequate “scholarly activity” as the Surgery Residency Review Committee (RRC) put it. [See previous blog]

A friend alerted me to a paper just published in Academic Emergency Medicine entitled “Diagnostic characteristics of S100A8/A9 in a multicenter study of patients with acute right lower quadrant abdominal pain.” This project took a lot of effort, and I respect the authors for that. And I am not opposed to the publishing of research that produces negative findings. But since no one at present is even considering using S100A8/A9 in the diagnosis of appendicitis and other biomarkers have not proven useful, I am not certain this needed to see the light of day.

Levels of a biomarker called S100A8/A9 [alias calprotectin] have been shown to rise in the presence of acute inflammation. The paper is a prospective, randomized, double-blinded multi-center trial investigating the utility of S100A8/A9 in the diagnosis of acute appendicitis, a disease that produces inflammation. Patients with right lower abdominal pain and suspected appendicitis had blood samples drawn. They were then sent to a central lab for measurement of S100A8/A9. The time it took to do this precluded the use of the results of the test for any decision-making. Presumably if the test was deemed useful, it would become part of every hospital’s on-site clinical lab.

The S100A8/A9 test was performed on blood from 848 eligible patients. When levels were elevated, it was found to be highly sensitive [It identified 96% of patients who had appendicitis] but poorly specific at 16% [That is, 16% of patients without appendicitis had normal S100A8/A9 levels; In other words, 84% of patients without appendicitis had elevated levels of S100A8/A9.] The area under the ROC curve was 0.66.

What does that mean? It means that the test’s accuracy approached 0.50, or 50%, the accuracy of flipping a coin. A very simple explanation of ROC curves can be found at the University of Nebraska Medical Center’s website. It points out that a diagnostic test with an area under the ROC curve of 0.66 would be considered poor.

The figure below is based on one from that site. The red line is the curve from the S100A8/A9 paper. The 50% line is illustrated in green. The S100A8/A9 blood test is not likely to replace CT scanning as the diagnostic test of choice in patients with right lower quadrant abdominal pain.

Despite these dismal results, the authors are undaunted and are planning more studies on this biomarker.

There’s a “Publish or Perish” situation in emergency medicine too. Here are three excerpts from the RRC for EM regulations:

1. There must be a minimum of one core physician faculty member for every three residents in the program.
2. The definition of a core physician faculty member is a member of the program faculty who provides clinical service and teaching, devotes the majority of his or her professional efforts to the program, and has sufficient time protected from direct service responsibilities to meet the educational requirements of the program. To this end, core faculty should not average more than 28 clinical hours per week. [Emphasis added. Unfortunately, this sweet deal is NOT found in the RRC for surgery regulations.]
3. The faculty must establish and maintain an environment of inquiry and scholarship with an active research component. [Translation: “Publish or Perish”]

So now you know why this research was done. It was published because there are so many journals that need to be fed. The Thomson Reuters SCIENCE CITATION INDEX lists 19 journals devoted to the topic of human emergency medicine. [See the last two paragraphs of a previous blog of mine about this.]

At least it provided an opportunity to say something about ROC curves.