A new study from the Johns Hopkins Armstrong Institute for Patient Safety and Quality, published in the journal Medical Care, performed a systematic review and meta-analysis of two sets of safety measures used for pay-for-performance and public reporting The measures evaluated in the study are used by several public rating systems, including U.S. News and World Report’s Best Hospitals, Leapfrog’s Hospital Safety Score, and the Center for Medicare and Medicaid Services’ (CMS’) Star Ratings.
The two sets of measures evaluated are:
- the 26 Agency for Health Care Research and Quality (AHRQ) Patient Safety Indicators (PSI)
- the 14 Centers for Medicare and Medicaid (CMS) Hospital-acquired Conditions (HACs)
The investigators first performed a systematic review of all published medical research since 1990, looking for studies that addressed the validity of the HAC and PSI measures. They identified only 5 of these 40 safety measures with enough data in these prior studies to permit a pooled meta-analysis:
- A. Iatrogenic Pneumothorax (PSI 6/HAC 17)
- B. Central Line-associated Bloodstream Infections (PSI 7)
- C. Postoperative hemorrhage/hematoma (PSI 9)
- D. Postoperative deep vein thrombosis/pulmonary embolus (PSI 12)
- E. Accidental Puncture/Laceration (PSI 15)
The investigators then performed a meta-analysis, pooling the results of all studies about the validity of each of these measures. Their findings in the figure below show that in pooled studies (the diamond at the bottom of each lettered rectangle) only one measure–Measure E, PSI 15 (Accidental Puncture and Laceration)–met the investigators’ criteria for validity: a positive predictive value of at least 80% (indicating that at least 80% of the patients determined by the measure to have an accidental puncture or laceration truly had an accidental puncture or laceration.) Actual occurrence (reference standard) of each reported safety event was determined, in each individual study, by medical chart review.
Measure D, PSI 9 (Postoperative hemorrhage or hematoma) came close to the 80% PPV threshold, with a pooled PPV of 78.6%.
Based on these findings, the investigators conclude that these measures, widely used for public reporting and pay-for-performance, should not be used for either purpose:
CMS and others have created payment incentives based on hospitals’ performance for a variety of hospital-acquired complications, which are measured with the respective PSIs and HAC measures. Policy makers and payers have argued that the PSIs and HAC measures are good enough for reporting and pay-for-performance, whereas many providers believe they are not. Our results suggest that the PSIs and HAC measures may not be valid enough and/or have insufficient data to support their use for these purposes. This is especially true given the potential financial impact these pay-for-performance approaches may have on the narrow financial margins on which most hospitals function.
Marion R. Sills, MD, MPH 