Measuring a Hospital’s Quality

We all want to be treated in the best hospitals by the best doctors, but this is not possible so long as any difference in quality between them exists. The best hospitals and the best doctors cannot treat everybody. Moreover, it is much harder to tell which hospital and which doctor is the best than many of us suppose. League tables for doctors and hospitals are not like such tables for baseball or football teams, matters of straightforward record. They require measurements of enormous complexity, and the results are only trustworthy and valuable if the data that go into compiling them are both accurate and relevant.

A paper in a recent edition of the British Medical Journal casts doubt on the value of global judgments on hospitals as expressed in league tables.

The authors reasoned that if such judgments were of any value, a hospital’s standardised mortality ratio (the proportion of people of any particular category who died compared with the number of that category expected on average to die) ought to correlate strongly with the number of avoidable deaths that occur in that hospital. A hospital with a high standardised mortality ratio – the usual way of measuring its overall quality – ought to have a high rate of avoidable deaths, if that ratio is a true measure of the quality of medical care in that hospital compared with other hospitals.

The authors then examined the statistics for 34 hospitals in England, 10 of them in 2009 and 24 others in 2012-13. They correlated their SMRs with the proportion of deaths that were avoidable, calculated by the proportion of 100 deaths that occurred in each of the hospitals, chosen at random and examined by experts to determine whether they occurred because of any act of commission or omission by the hospital. Of course, whether a death is avoidable is usually a matter of judgment; it is rarely that incompetence or negligence is so great that death is indubitably its consequence. For the purposes of this study, a death was deemed to have been avoidable if the experts assessing the case thought there was more than a fifty percent chance that it was.

The correlation between hospitals’ SMRs and their rate of avoidable deaths was so slight as to be negligible: indeed it was not statistically significant. Overall the rate of avoidable death was low: 5.2 percent in 2009 and 3.6 percent in 2012-13, 115 cases in 3400 examined. This difference was statistically significant, but one cannot rush to the conclusion that hospitals had improved in the intervening period, for various factors had changed also that could have affected the rate (for example, the wider use and compliance with requests not to resuscitate).

There were limitations to the study: for example, agreement between experts as to what was an avoidable death was far from unanimous. Moreover, the experts were not blinded to the hospitals from which the cases they examined came. They might therefore have been influenced by biases, for or against, conscious or unconscious, in their judgment as to which death was avoidable. Further, a hospital’s global Standardised Mortality Ratio might have disguised exceptionally good and exceptionally bad departments within it that balanced each other overall.

Nevertheless, the lesson seems clear: the global SMR as a measure of a hospital’s quality is invalid. This is not to say that there are no good and bad hospitals, only that the SMR is not the way to assess them, perhaps because the SMR itself is far from a watertight measure and is subject to a large number of confounding factors. We should be as accurate as possible, but not believe ourselves to be more accurate than were actually are.

First published in PJ Media.