We need less research

“We need less research, better research, and research done for the right reasons. Abandoning using the number of publications as a measure of ability would be a start.” Although I have expressed similar sentiments in blog posts [here and here], I didn’t say it. It was written by Douglas Altman, a well-known statistician and researcher who died in June.

Altman made that statement in a 1994 BMJ article entitled “The scandal of poor medical research.” Here we are, 24 years later, and nothing has changed. In fact, thanks to the rise of predatory journals, things are much worse.

Altman lamented research containing flaws such as “the use of inappropriate designs, unrepresentative samples, small samples, incorrect methods of analysis, and faulty interpretation” and felt many poor studies were the result of pressure on researchers to publish.

Those who can, publish. Those who can’t, blog

What’s your view on social media and science? For example, the role of science blogs in critiquing published papers? "Those who can, publish. Those who can’t, blog," says Jingmai O'Connor.

According to Cell.com, Dr. O’Connor is a professor at the Institute of Vertebrate Paleontology and Paleoanthropology of the Chinese Academy of Sciences, and her comment was part of an interview published last month.

Dr. O'Connor says, "It often seems those who criticize or spend large amounts of time blogging are also those who don’t generate much [sic] publications themselves." She thinks comments should be peer-reviewed and published only in journals. She worries about the public who may not realize "a published paper passed rigorous review by experts, which carries more validity than the opinion of some disgruntled scientist or amateur on the internet." She adds, "criticism in social media is damaging to science, as it is to most aspects of our culture."

Apparently she isn't aware that peer review is under fire from a number of respectable sources.

"If peer review was a drug it would never be allowed onto the market," said Drummond Rennie, a contributing deputy editor of JAMA. Richard Smith, former editor of the BMJ agrees "because we have no convincing evidence of its benefits but a lot of evidence of its flaws."

In 2015, 107 scientific papers were retracted by several journals because their authors, nearly all of whom were Chinese academics, had performed fraudulent peer review by creating fictitious names and email addresses of suggested reviewers so they could write glowing reviews of their own work. Some of these charlatans are from Beijing, where Dr. O'Connor is based.

Australian bloggers found an error that had somehow been missed during "rigorous review by experts" regarding the number needed to treat in a New England Journal of Medicine paper on targeted vs. universal decolonization to prevent ICU infection. They contacted the paper's corresponding author who acknowledged the mistake within 11 days. It took five months for a correction to appear online in the journal.

Whether Dr. O'Connor likes it or not, the future will involve more immediate feedback about research papers. For example, PubMed and PubPeer already allow comments, and the BMJ also has a section for online rapid responses.

Blogger Marc Bellemare, an associate professor of economics at the University of Minnesota, cites David McKenzie, an economist/blogger at the World Bank who thinks that blogs play an important role in disseminating information to the public and "raise the profile of bloggers and their institution."

But Bellemare feels blogging might not be for every academic He quotes Tyler Cowen of George Mason University, who when asked why don't more economists blog replied, "I believe it is because they can’t, at least not without embarrassing themselves rather quickly, even if they are smart and very good economists. It’s simply a different set of skills."

Maybe Dr. O'Connor doesn't have the skill set to blog. I say, "Those who can, blog. Those who can't, insult those who can."

More evidence that the manuscript peer review process is broken

To the surprise of almost no one, asking authors of research papers to submit names of potential peer reviewers for their manuscripts turns out to be a bad idea.

According to a recent New England Journal of Medicine article by Dr. Charlotte J. Haug, a number of research papers have been retracted because reviews were fabricated. Email addresses of suggested peer reviewers were not legitimate. The bogus email addresses were almost all created by authors of papers who then reviewed their own work favorably using fake identities. 

More about the problem can be found on the blog Retraction Watch.

This type of fraud is simple to do because anyone can set up an email address on Gmail or Yahoo mail using any name. Unless a reviewer has an academic email address, proving legitimacy is impossible.

However even if a reviewer has an “edu” address, how would an editor know that a suggested reviewer is not the author’s sister-in-law or a former mentor?

Every medical student who applies for residency knows that you don’t ask someone for a letter of recommendation unless you are sure that it will be favorable. Why would an author take a chance on recommending someone to review a paper without knowing that the review would be a good one?

I agree with the Dr. Haug that soliciting the names of possible reviewers from authors can save editors time and bother. Having spent three years as an associate journal editor, I have experienced the frustration of trying to find high quality reviewers or even a warm body of any quality to do the job.

I also agree with her that a root cause of this problem is the pressure on faculty to publish.

Another problem is that there are too many journals. In 2014, well over 5000 journals and 760,000 papers were included in Medline. The combination of “publish or perish” and superfluous journals leads to the proliferation of marginal papers.

The problem is not simply fake reviews. Since journal reviewers are not paid and have many other responsibilities, they may not thoroughly read papers or provide useful comments about manuscripts.

Some have suggested paying peer reviewers, but who would pay them? Certainly not publishers, even though they make tons of money. And paying might attract unqualified people looking to make a little extra cash.

What about post-publication peer review? It is already happening on blogs, on sites like PubPeer, and even on PubMed. However, the volume of papers published in medicine alone certainly precludes post-publication review of all of them.

Maybe it doesn’t matter. New journals are appearing every day. Most are “open access” and the charge authors “processing fees.” For many of these publications, processing fees do not include even a cursory manuscripts peer review.

With so many journals publishing just about anything for the right price, readers will have to do their own peer reviewing. Be skeptical my friends. 

Effects of acupuncture on pain and inflammation in pediatric appendicitis

A paper [full text here] from The Journal of Alternative and Complementary Medicine says that "acupuncture may be a feasible and effective treatment modality for decreasing subjective pain and inflammation" in pediatric patients with appendicitis.

They studied six adolescents with appendicitis and administered acupuncture for pain control prior to surgery. Pain was assessed using three analog scales, and inflammation was measured using serial white blood cell (WBC) counts and C-reactive protein levels (CRP). CRP is a nonspecific indicator of inflammation in the body.

Figure 2 from the paper summarizes the results.

You can see that after 20 minutes of acupuncture, all three pain scores and WBCs declined. However, CRP continued to rise. Conspicuously absent from the figure and the text of the paper are any statistical analyses. This is due to the lack of a significant difference in any of these values because of the limited number of subjects studied.

How much money do journal publishers make? A lot

Many, including me, have written about who is making money in healthcare. Sure doctors do very well, but not as well as hospitals, hospital administrators, insurance companies and their corporate officers, drug companies, device manufacturers, and others.

Another lucrative area is medical journal publication, especially if you are the publisher. A researcher gets an idea, plans and carries out a study, writes a manuscript, and submits it to a journal. The research may have been funded by the government, i.e., you and me.

An associate editor or a member of the journal's editorial board looks at the manuscript, and if it is deemed worthy, it is sent out to two or more people in the same field for peer review. This process may be repeated for papers that require revision.

All of the players in the above scenario—the researchers, most of the editorial board members except maybe the editor, and the peer reviewers—are paid nothing for their work. Factor in that the cost of producing a journal has plummeted in the computer era.

How much money do journal publishers make? Here are some impressive numbers from an article that appeared on a French website called "Rue89." The figures are for the year 2011 and are in euros. They include revenue from all science publishing, not just medicine.

What is the purpose of research?

There a number of excellent answers to this question. Here are a few—exploration, description, and explanation; to learn something; to gather evidence; documentation, discovery, and interpretation; the advancement of human knowledge

Here’s a nice one from TheLancet about why medical research is done: "to advance knowledge for the good of society; to improve the health of people worldwide; or to find better ways to treat and prevent disease."

I’m sure you could come up with a few responses to the question yourself.

But I don’t think you would get it, nor do I think the lofty answers I listed above are correct either.

What's the real reason to do research?

To get published.

Follow up. How many authors does it take…?

I recently blogged about the fact that for several different specialties, the number of authors of research papers has risen exponentially over the years. The post appeared on General Surgery News.

As is my nature, I was critical of this increase and blamed it on the "publish or perish" culture that pervades medicine and other sciences and the large number of new journals, which all are looking for content so they can make money.

A Twitter follower named @terpkristin ask me what my own experience has been. This is a fair question and I applaud her for taking me to task on it.

With some trepidation, I pulled up my CV and started counting.

Over the years I have authored or co-authored 97 publications, including peer-reviewed articles and case reports, review articles, book chapters, editorials and letters to journal editors.

My first article was published in 1975 and the last in 2010. That's a span of 35 years which is similar to the times covered in many of the articles on author proliferation.

After reviewing all 97 publications, the average number of authors was 3.0 with 23 having only me as the sole author.

Excluding everything but the 52 peer-reviewed papers, the mean number of authors was 3.5.

Looking at the peer-reviewed papers another way, the 22 papers written from 1975 to 1993 had an average of 3.2 authors, while those written during 1994 to 2010 averaged 3.6 authors.

This compares favorably to the numbers of authors listed in the articles I found on the subject and wrote in my previous post.

For papers written in this century in the fields of plastic surgery, orthopedics and thoracic surgery, the average number of authors ranged from 4.0 to 7.5.

Luckily, my inquisitor didn't ask me how many of the articles were any good. That's a different question.