More evidence that the manuscript peer review process is broken

To the surprise of almost no one, asking authors of research papers to submit names of potential peer reviewers for their manuscripts turns out to be a bad idea.

According to a recent New England Journal of Medicine article by Dr. Charlotte J. Haug, a number of research papers have been retracted because reviews were fabricated. Email addresses of suggested peer reviewers were not legitimate. The bogus email addresses were almost all created by authors of papers who then reviewed their own work favorably using fake identities. 

More about the problem can be found on the blog Retraction Watch.

This type of fraud is simple to do because anyone can set up an email address on Gmail or Yahoo mail using any name. Unless a reviewer has an academic email address, proving legitimacy is impossible.

However even if a reviewer has an “edu” address, how would an editor know that a suggested reviewer is not the author’s sister-in-law or a former mentor?

Every medical student who applies for residency knows that you don’t ask someone for a letter of recommendation unless you are sure that it will be favorable. Why would an author take a chance on recommending someone to review a paper without knowing that the review would be a good one?

I agree with the Dr. Haug that soliciting the names of possible reviewers from authors can save editors time and bother. Having spent three years as an associate journal editor, I have experienced the frustration of trying to find high quality reviewers or even a warm body of any quality to do the job.

I also agree with her that a root cause of this problem is the pressure on faculty to publish.

Another problem is that there are too many journals. In 2014, well over 5000 journals and 760,000 papers were included in Medline. The combination of “publish or perish” and superfluous journals leads to the proliferation of marginal papers.

The problem is not simply fake reviews. Since journal reviewers are not paid and have many other responsibilities, they may not thoroughly read papers or provide useful comments about manuscripts.

Some have suggested paying peer reviewers, but who would pay them? Certainly not publishers, even though they make tons of money. And paying might attract unqualified people looking to make a little extra cash.

What about post-publication peer review? It is already happening on blogs, on sites like PubPeer, and even on PubMed. However, the volume of papers published in medicine alone certainly precludes post-publication review of all of them.

Maybe it doesn’t matter. New journals are appearing every day. Most are “open access” and the charge authors “processing fees.” For many of these publications, processing fees do not include even a cursory manuscripts peer review.

With so many journals publishing just about anything for the right price, readers will have to do their own peer reviewing. Be skeptical my friends. 

The quest for medical error transparency suffers setbacks

Two recent articles in American Medical News highlight the difficulties in moving toward a blameless culture in medicine.

First we have the results of an Agency for Healthcare Research and Quality survey of 600,000 staff from over 1100 hospitals. Half of them “believe their mistakes are held against them, and 54% said that when an adverse event is reported, ‘it feels like the person is being written up, not the problem.’" And two-thirds are concerned that records of errors are maintained in their personnel files. Despite lip service by medical thought leaders, the reported responses have not changed since the last AHRQ survey in 2007.

Much more alarming is the second amednews.com piece, which involves a family secretly recording a conversation with the chief medical officer [CMO] of an Ohio hospital.

Two days after knee surgery, a man died of cardiac arrest apparently secondary to an overlooked high serum potassium level. Before he died, the patient’s children met with the CMO, who had not personally been caring for the patient. Unaware that the conversation was being recorded, the CMO “made sympathetic and apologetic comments and admitted fault on the part of the hospital for Smith's condition, according to court records.”

The Ohio appellate court ruled that the secret recording was admissible and not protected by peer review privilege, as had been argued by the CMO’s lawyers. They had claimed that the CMO had learned of the error via the peer review process. Ohio law states that as long as one party consents, conversations can be recorded.

Thus we are left with pie-in-the-sky appeals for greater transparency and candor regarding medical errors [the so-called “Just Culture”] vs. a litigious society with stealth technical capabilites and a workplace atmosphere of fear and distrust.

Somehow I do not foresee a major change in the way doctors and hospital staffs approach this issue.

How journal articles are peer-reviewed: Part 2

Based on my experience as an associate editor of a journal, I recently blogged about the current mechanism of peer-review that most medical journals use and pointed out that it doesn't work very well.

Here is a story that illustrates some of the problems with the peer-review process.

A journal editor sent a manuscript to three peer-reviewers.

Of the three reviews returned, one recommendation was to “accept,” one said “accept with minor revision” and one said “accept with major revision.” Two of the reviews were 2-3 sentences long. One simply complimented the authors on a “nice paper.” The other pointed out two trivial areas needing revision.

The third review raised a couple of major points questioning the methods used by the authors. These were significant issues suggesting that the paper was unsalvageable.

Conflicting opinions are not that unusual. I have seen three reviews returned which had three even more diverse opinions such as, accept, accept with revision and reject. Based on that input, how does an editor decide the fate of a paper?

Perhaps you have read a paper and wondered, as I have, how did this ever see the light of day? Heck, I’ve even written a few papers like that myself. There used to be a saying, "If you have enough stamps, you can eventually get anything published." Now you don't even need the stamps.

Bottom line for me is that the peer-review process is very weak at the level of the peer-reviewers themselves. The reviews are not standardized and the quality is inconsistent.

Coincidentally, the New York Times ran a piece today mentioning some other problems with conventional journals such as “Peer review can take months, journal subscriptions can be prohibitively costly, and a handful of gatekeepers limit the flow of information.” The new wave seems to be “open science.” It will be interesting to see how this evolves.

I may be out of a job as an unpaid associate editor soon.

How are journal articles peer-reviewed?

There is a possibly some misunderstanding among science journalists regarding the process that the term “peer-review” encompasses.

I am an associate editor (AE) of a medical journal with a respectable impact factor. I also am or have been a manuscript reviewer for five different journals. I feel qualified to describe how manuscripts are reviewed and published.

In 2012, authors submit a manuscript electronically to the journal. It is assigned to an AE who screens it for appropriateness, format and, on occasion, readability in the English language. Manuscripts are not blinded. AEs and reviewers are aware of the authors’ names and their institutions.

The AE emails prospective peer-reviewers asking if they are willing to review the submission. Reviewers are chosen based on their self-reported areas of interest. They become listed as peer-reviewers by demonstrating expertise, usually having submitted papers of their own. They may also be well-known experts through society memberships or familiarity with the journal’s editorial board members. I once became a peer-reviewer for a journal after writing a letter to the editor pointing out a statistical flaw in a published paper. Although seen by many as a career-enhancing, the jobs of AE and peer-reviewer are not compensated.

If all goes well, the peer-reviewers return their recommendations in a timely way. Unfortunately, being a peer, an expert or an author in a field related to the manuscript’s topic does not necessarily mean that one can review a research paper competently. Most journals have guidelines for reviewers but no way to tell if the reviewer has read them. We often receive “two-sentence” reviews of 25-30 page (double-spaced) manuscripts.

A manuscript would have to be quite extraordinary to elicit only a two-sentence review. The reviewer may have been too busy, disinterested, incapable or not motivated to do a thorough job. But then why would he have accepted the assignment? That’s one of life’s great mysteries. The AE may have to become a peer-reviewer at times.

Assuming the AE receives two or three adequate reviews, he decides to accept, accept with revisions or reject the paper and forwards it to the editor for a final decision.

Here is what we cannot do. We cannot verify that

1. the data are not fabricated;

2. all authors deserve to have their names listed on the paper;

3. no plagiarism has occurred;

4. the paper is not an attempted duplicate publication.

Journals have no resources to investigate any of these issues. We must accept the word of those submitting. Among other causes, pressure on faculty to publish and/or greed may promote scientific misconduct.

Is this a good system? No. What are the alternatives? I don’t know. I pointed out in a previous post that many more people have read my blog than ever read my research publications. One day, every paper may be posted and critiqued by the scientific public, a movement that has already begun on websites such as Faculty of 1000.

Meanwhile, expect to see more publications retracted as internet users discover and expose fraud and duplicate publications. For more on retractions, follow the blog Retraction Watch for interesting insights into the process.