More on Why Patient Satisfaction Surveys Are Useless

Patient satisfaction surveys are theoretically good things. After all, hospitals and doctors should know what their patients think about them. If a service is not performing well, the results of a patient satisfaction survey would be useful to help us make improvements. This would be true if patient satisfaction could be measured in a meaningful way. In a previous blog, I explained some of the statistical problems with all patient satisfaction surveys.

Here’s an example of the current state of the art and why it is useless.

A hospital posted the above graphic. As provided by a leading company, it shows the trend of patient satisfaction scores for the question “Were you informed about the causes of delays during your emergency department [ED] visit?”

You can see that the scores fluctuate around the average score for a year of the 79^th percentile, meaning that 21% of comparator hospitals fared better on this question.

This ED receives over 25,000 visits per year or over 2000 visits per month. The number of responses to the survey questionnaires sent ranged from a low of 2 in June of 2011 to a high of 42 in October of 2010. This represents a range of 0.1% to 2.1% of all ED visits, far below what even a non-statistician would recognize as a reasonable sample.

Despite efforts to improve scores for this question, no significant change has occurred. What does it all mean? If you look at the trend line, you will note that it mirrors what occurs in a feedback loop. [Digression: A thermostat is a classic example of a feedback loop in action. If you set the thermostat for 70 degrees, the temperature will fluctuate about that point. Most of the time it will not be 70 degrees in the room as the furnace will run until the temperature is 70 and when the furnace shuts off the temperature will overshoot. Likewise when the temperature falls below 70, it will take time for the furnace to warm the room back up. (Ref. GJS Wilde, Target Risk 2, PDE Publications, Toronto 2001)] This feedback loop phenomenon is a classic reaction to the posting of such scores. When the scores fall, task forces are formed and meetings are held. A slight improvement is often seen. [Hawthorne effect: performance improves when subjects are aware they are being watched.] But once the scores improve, everyone goes back to business as usual and the scores drop again.

It’s too bad that the 87.5 percentile score was achieved when the response to the questionnaire was a mere 2 patients.

If I were a betting man, I would bet the ranch that the score for this question will be lower for July.