We need less research

“We need less research, better research, and research done for the right reasons. Abandoning using the number of publications as a measure of ability would be a start.” Although I have expressed similar sentiments in blog posts [here and here], I didn’t say it. It was written by Douglas Altman, a well-known statistician and researcher who died in June.

Altman made that statement in a 1994 BMJ article entitled “The scandal of poor medical research.” Here we are, 24 years later, and nothing has changed. In fact, thanks to the rise of predatory journals, things are much worse.

Altman lamented research containing flaws such as “the use of inappropriate designs, unrepresentative samples, small samples, incorrect methods of analysis, and faulty interpretation” and felt many poor studies were the result of pressure on researchers to publish.

The dark side of academic research

A new study found several senior academic surgeons had published papers in what used to be termed “predatory journals.” The newer, gentler term is “solicited publishing,” but it defines the same pay-to-play, low quality publications.

Surgeons from the University of California, San Diego examined 110 emails sent to the senior author from 29 publishers during a six-week period and early 2017. Nearly all were requesting manuscript submissions. The 29 publishers represented 113 different surgery journals most of which had existed for two years or less. Only 12 were indexed in PubMed, and of the 9 that mentioned a self-reported impact factor, the median was 0.24 which means they had less than one citation per article in the last two years. The median publication fee for the 88 journals posting the information was $755.

Emails from the publishers contained a mean of 9.6 grammatical errors, possibly because more than half had addresses in foreign countries, and of those with US addresses, 30% were residential.

Evidence? We don’t need no stinkin’ evidence

One of my posts requires clarification. The post "A paper of mine was published. Did anyone read it?" went live in August 2014 and has been viewed 5133 times to date.

A reader had emailed me to ask if I might know why two papers he had written did not cause much of a stir in the orthopedic world. One reason might have been that the papers appeared in an obscure orthopedic journal.

I then wrote: "A paper in Physics World claims that that 90% of published papers are never cited and 50% are never read by anyone but the authors and the journals' peer reviewers." This is simply not true.

The link in the above paragraph originally went to a nebulous Indiana University web page and eventually became a "file not found." The source of the 2007 Physics World paper remained elusive. The subject came up again about a week ago on Twitter and a follower, @TirathPatelMD, sent me a link to the full text.

Many journalists miss major flaws in research papers

Google "jet air dryers" and you will find more than 100 articles about a recent study claiming that jet air dryers dispersed 60 times more "germs" than warm air dryers and 1300 times more "germs" than paper towels.

Here are a few of the juicier headlines:

It wasn't about germs. The study involved measuring the dispersal of a benign virus from the hands of research subjects at various distances and heights from the different hand drying methods.

So what's wrong with it?

More evidence that the manuscript peer review process is broken

To the surprise of almost no one, asking authors of research papers to submit names of potential peer reviewers for their manuscripts turns out to be a bad idea.

According to a recent New England Journal of Medicine article by Dr. Charlotte J. Haug, a number of research papers have been retracted because reviews were fabricated. Email addresses of suggested peer reviewers were not legitimate. The bogus email addresses were almost all created by authors of papers who then reviewed their own work favorably using fake identities. 

More about the problem can be found on the blog Retraction Watch.

This type of fraud is simple to do because anyone can set up an email address on Gmail or Yahoo mail using any name. Unless a reviewer has an academic email address, proving legitimacy is impossible.

However even if a reviewer has an “edu” address, how would an editor know that a suggested reviewer is not the author’s sister-in-law or a former mentor?

Every medical student who applies for residency knows that you don’t ask someone for a letter of recommendation unless you are sure that it will be favorable. Why would an author take a chance on recommending someone to review a paper without knowing that the review would be a good one?

I agree with the Dr. Haug that soliciting the names of possible reviewers from authors can save editors time and bother. Having spent three years as an associate journal editor, I have experienced the frustration of trying to find high quality reviewers or even a warm body of any quality to do the job.

I also agree with her that a root cause of this problem is the pressure on faculty to publish.

Another problem is that there are too many journals. In 2014, well over 5000 journals and 760,000 papers were included in Medline. The combination of “publish or perish” and superfluous journals leads to the proliferation of marginal papers.

The problem is not simply fake reviews. Since journal reviewers are not paid and have many other responsibilities, they may not thoroughly read papers or provide useful comments about manuscripts.

Some have suggested paying peer reviewers, but who would pay them? Certainly not publishers, even though they make tons of money. And paying might attract unqualified people looking to make a little extra cash.

What about post-publication peer review? It is already happening on blogs, on sites like PubPeer, and even on PubMed. However, the volume of papers published in medicine alone certainly precludes post-publication review of all of them.

Maybe it doesn’t matter. New journals are appearing every day. Most are “open access” and the charge authors “processing fees.” For many of these publications, processing fees do not include even a cursory manuscripts peer review.

With so many journals publishing just about anything for the right price, readers will have to do their own peer reviewing. Be skeptical my friends. 

My blog cited in JAMA Surgery paper: Progress for bloggers

About a year and a half ago, I blogged that a medical student on Twitter used a blog post of mine as evidence. In January, the Canadian Journal of Anesthesia published an article I wrote under my pseudonym called “Why I blog and tweet.”

Last month, medical blogging took another step toward legitimacy. A JAMA Surgery Viewpoint formally cited my post critiquing the Finnish randomized trial of antibiotics versus surgery for the treatment of acute appendicitis.

Here is the first page with the portion of the piece discussing what I had written in the blog post.

Click on figure to enlarge.

Here is how citation appears in the JAMA Surgery article.

If you haven't read my entire post about the randomized trial, click here.

Last year I said this: “Journals may have to adapt and become more like blogs. In the future, medical information may be disseminated by blogs and comments rather than journal articles and letters to the editor.”

We have already seen prominent publications such as the New England Journal of Medicine starting online forums and the BMJ hosting blogs (at least 36 so far) and rapid responses to published papers.

The sea change in the way medical research is disseminated may be happening sooner than I thought.

Don't trust the abstract; read the whole paper

Above are the results and conclusion from an abstract of a paper called "Single-Incision Laparoscopic Cholecystectomy: Will It Succeed as the Future Leading Technique for Gallbladder Removal?" It appears online in the journal Surgical Laparoscopy, Endoscopy & Percutaneous Techniques.

It is another great example of why you need to read the entire paper and not just the abstract.

The methods section says that the study involved 875 patients with prospectively collected data. Don’t be fooled by prospectively collected data. This research is retrospective.

Here are some issues.

How much money do journal publishers make? A lot

Many, including me, have written about who is making money in healthcare. Sure doctors do very well, but not as well as hospitals, hospital administrators, insurance companies and their corporate officers, drug companies, device manufacturers, and others.

Another lucrative area is medical journal publication, especially if you are the publisher. A researcher gets an idea, plans and carries out a study, writes a manuscript, and submits it to a journal. The research may have been funded by the government, i.e., you and me.

An associate editor or a member of the journal's editorial board looks at the manuscript, and if it is deemed worthy, it is sent out to two or more people in the same field for peer review. This process may be repeated for papers that require revision.

All of the players in the above scenario—the researchers, most of the editorial board members except maybe the editor, and the peer reviewers—are paid nothing for their work. Factor in that the cost of producing a journal has plummeted in the computer era.

How much money do journal publishers make? Here are some impressive numbers from an article that appeared on a French website called "Rue89." The figures are for the year 2011 and are in euros. They include revenue from all science publishing, not just medicine.

Should social media accomplishments be recognized by academia?

In August, I posted this: "A paper of mine was published. Did anyone read it?"

A recent comment on it raised an interesting point. Dr. Christian Sinclair [@ctsinclair] said that a website he is helping to run called "Pallimed" has received almost 2 million views since 2005.

He then made the following calculation:

Two million views with an average of 1:30 minutes on a page = 3 million minutes = 50,000 hours = 2,083 days = 5.7 years of 24/7/365 informal learning on hospice and palliative care topics.

He said that this type of communication counts for nothing regarding academic advancement and added that writing another paper and having it published in a journal no one reads or a chapter in an expensive book no one will buy is considered worthwhile.

This reminded me of something I have talked about in recent presentations. The first laparoscopic cholecystectomy done in the United States took place in 1988. The procedure rapidly became popular due to its obvious benefits over traditional open surgery—smaller scars, shorter hospitalizations, quicker returns to normal activity.

A paper of mine was published. Did anyone read it?

An orthopedist asked me if I could explain why a couple of papers of his did not generate any feedback. He wasn't even sure that anyone had read them. He enclosed PDFs for me.

Not being an orthopedist, I cannot comment on their validity.

But I think I can explain why the papers have not created much interest.

Are you familiar with the term "impact factor"? If not, here is a link explaining what it is:

A journal's impact factor is an indication of how widely cited its articles are. One can also assume that it is a good indication of how popular the journal is and by inference, how many people read its papers. The impact factor has been criticized, but it is one of the few measures of a journal's influence.

The two papers in question were published in Orthopaedics & Traumatology: Surgery & Research. A list of the top 40 orthopedic journals ranked by impact factor in 2013 showed that it ranked 37th with an impact factor of 1.061. That means the average number of citations for any paper published in OTSR was about 1, and 36 orthopedic journals were more widely cited than OTSR.

A paper in Physics World claims that that 90% of published papers are never cited and 50% are never read by anyone but the authors and the journals' peer reviewers. I believe this is true of papers in medical journals too. [Update: This paragraph is untrue. See my post of 3/23/17.]

I was unable to obtain any figures regarding the number of subscribers to OTSR, but I suspect it is not large. This may also account for the lack of responses to the papers. My own experience is similar. It was very rare to receive any feedback about any of the over 90 peer-reviewed papers, editorials, or reviews that I had written.

Consider this. A blog post of mine "Appendicitis: Diagnosis, CT Scans and Reality" which I wrote 4 years ago has received over 77,000 [Updated on 3/23/17] page views and more than 100 comments. I am certain that post has been read far more than all of my published research papers combined. In fact, my 550 blog posts have recorded over 1 million page views.

What does it all mean?

Journals may have to adapt and become more like blogs. In the future, medical information may be disseminated by blogs and comments rather than journal articles and letters to the editor.

Will scientists' CVs be valued more for the number of page views their papers receive than the number of peer-reviewed papers they publish?

What you need to know about some open access journals

The other day I received this email.

Invitation to Review Manuscript: HJEHS-14-013

Dear Colleague,

We received a manuscript titled:

ARMSPAN TO THIGH-FOOT HEIGHT RATIO AMONGST ADULT MALES IN TWO SOUTH-SOUTH STATES IN NIGERIA

We will be most grateful if you could find time to review the manuscript. Please find the Abstract below: 
ABSTRACT
This investigation was carried out to determine the armspan to lower limb length ratio amongst adult males in Rivers and Bayelsa States of Nigeria. A total of 500 apparently healthy males were used which cut across several age groups ranging from 20 to 60 years. Data were collected through one on one interview after which the individuals arm span and lower limb length were taken using a flexible metre tape. Results showed that arm span correlated well with the lower limb length and significant difference exist between the arm span and lower limb length ratio of adult males in Rivers state (2.06) and Bayelsa State (2.04) with p > 0.05 [sic]. The result also showed that a significant difference exist between arm span to lower limb length in all the age groups p > 0.05 [sic]. From the results we concluded that the arm span to thigh-foot heigth [sic] ratio of adult males in Rivers and Bayelsa States varies, with that of adult males in Bayelsa state closer to the estimated normal value of 2.
Key words: Arm-span, Thigh-foot, South-south, Nigeria

Kindly send us send a mail indicating your acceptance to review this manuscript to enable us forward to you the full manuscript.

Thank you.

Faith Lily
Editorial Assistant,
Herald Journal of Environmental and Health Science

Those are some really important findings. They suggest that either the authors or the journal's editors (or both) don't know what a significant p value is. (For you non-statisticians, a significant p value should be < 0.05, not > 0.05.) At the very least, the number of typos suggests they aren't very careful.

Why send this to me? I've never dealt with this journal before, and I have no interest or expertise in "the armspan to lower limb length ratio amongst adult males in Rivers and Bayelsa States of Nigeria" or anywhere else.

From the journal's home page: "Why Publish with HJEHS - Quality Peer Review."

Some quality. Reviewers like me are apparently selected at random.

Note also that the journal's "Archive" is empty and dated 2012.

Much as I would love to see the full manuscript, I'm going to pass on the opportunity.

"Herald Journal of Environmental and Health Science" appears on Beall's List, a compilation of suspected "predatory" journals. The criteria for identifying a predatory journal are found here.

The good news is that at least it's cheap, with a stated "processing fee" of only $400. In a previous post about open access journals, I noted fees as high as $5000.

Faculty, residents, and students should be aware that any journal that promises open access and charges a fee for publishing a paper should be thoroughly vetted before submitting any work or especially any money.

Author! Author!

As journals proliferate, so do authors.

New journals are appearing almost every day. Does anyone read them? Journals keep popping up because of the need for faculty to publish. Another reason could be that publishers, particularly those who charge authors fees for publishing, are in the business of making money.

Authoring journal articles is not only enhancing to one's CV (the old "publish or perish" cliché), it is required by Residency Review Committees as evidence of "scholarly activity" in training programs. Maybe it's good for attracting referrals too.

Without too much difficulty, I have collected some interesting information about the number of authors per paper in several specialties.

First noted in 1993 by a paper in Acta Radiologica and a letter in the BMJ, the number of authors per paper has risen dramatically over the years.

A study of 12 radiology journals found the number of authors per paper doubled from 2.2 in 1966 to 4.4 in 1991. 

A review of Neurosurgery and the Journal of Neurosurgery spanned 50 years. the average went from 1.8 authors per article in 1945 to 4.6 authors in 1995.

Of note, the above two articles were each written by a single author.

Three psychiatrists from Dartmouth analyzed original scientific articles in four of the most prestigious journals in the United States—Archives of Internal Medicine, Annals of Internal Medicine, Journal of the American Medical Association, and the New England Journal of Medicine—from 1980 to 2000. They found that the mean number of authors per paper increased from 4.5 to 6.9.

The same is true for two plastic surgery journals, which saw the average number of authors go from 1.4 to 4.0 and 1.7 to 4.2 in the 50 years from 1955 to 2005. The number of single-author papers went from 78% to 3% in one journal and 51% to 8% another.

In orthopedics, a review of the American and British versions of the Journal of Bone and Joint Surgery for 60 years from 1949 to 2009 showed an increase of authors per paper from 1.6 to 5.1.

An impressive  rise in the number of authors took place in two leading thoracic surgery journals. For the Journal of Thoracic and Cardiovascular Surgery the increase was 1.4  in 1936 to 7.5 2006 and for Annals of Thoracic Surgery it was 3.1 in 1966 to 6.8 in 2006.

Where will it end?

As far as I know, the current leader in the race for the paper with the most authors is "Observation of a new particle in the search for the Standard Model Higgs boson with the ATLAS detector at the LHC" in a journal called "Physics Letters B" with 3171. The list of authors takes up 9 full pages.

What is the purpose of research?

There a number of excellent answers to this question. Here are a few—exploration, description, and explanation; to learn something; to gather evidence; documentation, discovery, and interpretation; the advancement of human knowledge

Here’s a nice one from TheLancet about why medical research is done: "to advance knowledge for the good of society; to improve the health of people worldwide; or to find better ways to treat and prevent disease."

I’m sure you could come up with a few responses to the question yourself.

But I don’t think you would get it, nor do I think the lofty answers I listed above are correct either.

What's the real reason to do research?

To get published.

Follow up. How many authors does it take…?

I recently blogged about the fact that for several different specialties, the number of authors of research papers has risen exponentially over the years. The post appeared on General Surgery News.

As is my nature, I was critical of this increase and blamed it on the "publish or perish" culture that pervades medicine and other sciences and the large number of new journals, which all are looking for content so they can make money.

A Twitter follower named @terpkristin ask me what my own experience has been. This is a fair question and I applaud her for taking me to task on it.

With some trepidation, I pulled up my CV and started counting.

Over the years I have authored or co-authored 97 publications, including peer-reviewed articles and case reports, review articles, book chapters, editorials and letters to journal editors.

My first article was published in 1975 and the last in 2010. That's a span of 35 years which is similar to the times covered in many of the articles on author proliferation.

After reviewing all 97 publications, the average number of authors was 3.0 with 23 having only me as the sole author.

Excluding everything but the 52 peer-reviewed papers, the mean number of authors was 3.5.

Looking at the peer-reviewed papers another way, the 22 papers written from 1975 to 1993 had an average of 3.2 authors, while those written during 1994 to 2010 averaged 3.6 authors.

This compares favorably to the numbers of authors listed in the articles I found on the subject and wrote in my previous post.

For papers written in this century in the fields of plastic surgery, orthopedics and thoracic surgery, the average number of authors ranged from 4.0 to 7.5.

Luckily, my inquisitor didn't ask me how many of the articles were any good. That's a different question.

Why won't the AMA stop asking me to pay dues?

Today I received another one of these in the mail.

It's a dues notice from the American Medical Association. For a mere $420.00 per year, I can become a member. 

Over 10 years ago, I dropped out of the AMA because it seemed to be run by a bunch of old guys who were out of touch with the real issues of practicing medicine in the 21st century. Other than the included subscription to a journal which was then called Archives of Surgery, I felt that I received no value from belonging.

Approximately every 3 months since then I have received notices like the one above from the organization.

By now, you would think they might have a clue that I am not interested in rejoining, but the notices keep on coming. I realize the process is probably automated and the algorithm doesn't sense the fact that I'm never going to respond. But here's a thought.

There's a scheme that some magazines run. They send you a "free" copy and if you don't read the fine print, you get one every month. Then you get a bill.

A variation is this. Figuring that your address is a business which is quite likely when someone is an MD, magazines send frequent subscription "renewal" notices when you've decided to discontinue it or even when no prior subscription exists. What they are hoping for is that an office manager or secretary will see the notice and pay the bill as a matter of routine. 

To illustrate this note that I also discontinued my subscription to the New England Journal of Medicine a few years ago, but every two months I get a renewal notice as if I had been a subscriber all along.

I'm not saying I can prove that the AMA is doing this with the same intent, but why would they keep it up for so long?

Also, I wonder how much money they waste mailing these notices.

According to recent figures, only about 17% (217,000) of all US doctors and medical students are AMA members, 25% of them are over 65 years old and enrollment is flat. 

I'm not sure why anyone belongs. Any ideas?

More on “Publish or Perish”

Bonus: What is a receiver operating characteristic (ROC) curve and how do you interpret it?

God knows I’ve written and managed to get published some downright awful research papers in my time. I was driven by the same forces that are at work today. “Publish or Perish” was very real for me as I needed to grind out papers to keep my general surgery residency from going on probation for inadequate “scholarly activity” as the Surgery Residency Review Committee (RRC) put it. [See previous blog]

A friend alerted me to a paper just published in Academic Emergency Medicine entitled “Diagnostic characteristics of S100A8/A9 in a multicenter study of patients with acute right lower quadrant abdominal pain.” This project took a lot of effort, and I respect the authors for that. And I am not opposed to the publishing of research that produces negative findings. But since no one at present is even considering using S100A8/A9 in the diagnosis of appendicitis and other biomarkers have not proven useful, I am not certain this needed to see the light of day.

Levels of a biomarker called S100A8/A9 [alias calprotectin] have been shown to rise in the presence of acute inflammation. The paper is a prospective, randomized, double-blinded multi-center trial investigating the utility of S100A8/A9 in the diagnosis of acute appendicitis, a disease that produces inflammation. Patients with right lower abdominal pain and suspected appendicitis had blood samples drawn. They were then sent to a central lab for measurement of S100A8/A9. The time it took to do this precluded the use of the results of the test for any decision-making. Presumably if the test was deemed useful, it would become part of every hospital’s on-site clinical lab.

The S100A8/A9 test was performed on blood from 848 eligible patients. When levels were elevated, it was found to be highly sensitive [It identified 96% of patients who had appendicitis] but poorly specific at 16% [That is, 16% of patients without appendicitis had normal S100A8/A9 levels; In other words, 84% of patients without appendicitis had elevated levels of S100A8/A9.] The area under the ROC curve was 0.66.

What does that mean? It means that the test’s accuracy approached 0.50, or 50%, the accuracy of flipping a coin. A very simple explanation of ROC curves can be found at the University of Nebraska Medical Center’s website. It points out that a diagnostic test with an area under the ROC curve of 0.66 would be considered poor.

The figure below is based on one from that site. The red line is the curve from the S100A8/A9 paper. The 50% line is illustrated in green. The S100A8/A9 blood test is not likely to replace CT scanning as the diagnostic test of choice in patients with right lower quadrant abdominal pain.

Despite these dismal results, the authors are undaunted and are planning more studies on this biomarker.

There’s a “Publish or Perish” situation in emergency medicine too. Here are three excerpts from the RRC for EM regulations:

1. There must be a minimum of one core physician faculty member for every three residents in the program.
2. The definition of a core physician faculty member is a member of the program faculty who provides clinical service and teaching, devotes the majority of his or her professional efforts to the program, and has sufficient time protected from direct service responsibilities to meet the educational requirements of the program. To this end, core faculty should not average more than 28 clinical hours per week. [Emphasis added. Unfortunately, this sweet deal is NOT found in the RRC for surgery regulations.]
3. The faculty must establish and maintain an environment of inquiry and scholarship with an active research component. [Translation: “Publish or Perish”]

So now you know why this research was done. It was published because there are so many journals that need to be fed. The Thomson Reuters SCIENCE CITATION INDEX lists 19 journals devoted to the topic of human emergency medicine. [See the last two paragraphs of a previous blog of mine about this.]

At least it provided an opportunity to say something about ROC curves.

How journal articles are peer-reviewed: Part 2

Based on my experience as an associate editor of a journal, I recently blogged about the current mechanism of peer-review that most medical journals use and pointed out that it doesn't work very well.

Here is a story that illustrates some of the problems with the peer-review process.

A journal editor sent a manuscript to three peer-reviewers.

Of the three reviews returned, one recommendation was to “accept,” one said “accept with minor revision” and one said “accept with major revision.” Two of the reviews were 2-3 sentences long. One simply complimented the authors on a “nice paper.” The other pointed out two trivial areas needing revision.

The third review raised a couple of major points questioning the methods used by the authors. These were significant issues suggesting that the paper was unsalvageable.

Conflicting opinions are not that unusual. I have seen three reviews returned which had three even more diverse opinions such as, accept, accept with revision and reject. Based on that input, how does an editor decide the fate of a paper?

Perhaps you have read a paper and wondered, as I have, how did this ever see the light of day? Heck, I’ve even written a few papers like that myself. There used to be a saying, "If you have enough stamps, you can eventually get anything published." Now you don't even need the stamps.

Bottom line for me is that the peer-review process is very weak at the level of the peer-reviewers themselves. The reviews are not standardized and the quality is inconsistent.

Coincidentally, the New York Times ran a piece today mentioning some other problems with conventional journals such as “Peer review can take months, journal subscriptions can be prohibitively costly, and a handful of gatekeepers limit the flow of information.” The new wave seems to be “open science.” It will be interesting to see how this evolves.

I may be out of a job as an unpaid associate editor soon.