Consumer Reports shoots itself in the foot

Just after I referred to hospital rankings as a charade, Consumer Reports jumped into the fray with ratings of 2,463 hospitals and specific ratings for 5 common procedures: knee replacement, hip replacement, back surgery, coronary angioplasty and carotid artery surgery (Consumer Reports, September 2013, p. 31-41).  The ratings are based on billing claims for Medicare patients, they are risk-adjusted, and use death of a patient undergoing elective surgery and extended length of stay as quality measures for 27 different procedures.

I am deeply disappointed.  Here’s why: In September 2010, Consumer Reports took a giant step forward by making public the ratings for Coronary Artery Bypass Grafting (CABG) developed by the Society of Thoracic Surgeons (STS).  This was correctly referred to as a watershed event because a medical society, not a third party, had developed the rating.  The STS had come up with a three-star index for service/hospital combinations using clinical, not administrative, data.  The index was based largely on outcomes – risk-adjusted mortality and morbidity – with a component that takes account of conformance to best practice.   By contrast, Consumer Reports now takes a step back by (1) rating hospitals rather than a service at a hospital; (2) using administrative data rather than clinical data; and (3) coming up with ratings not created by clinicians themselves.  Why would any intelligent patient requiring a particular procedure rely on the Consumer Reports rating when making an important decision?

To give Consumer Reports its due, the article includes some excellent points:

• They’re open with their objectives: to shed light on hospital quality and to push for greater transparency.
• They observe that current bad incentives provide little reason for hospitals to improve.
• They indirectly criticize NSQIP for not allowing its ratings to be made public.
• They acknowledge that hospitals can excel on one procedure but not so well on others.  For example, they mention that the Massachusetts General Hospital excels at CABG, but performs less well on hip replacement.
• They commend STS for making its ratings public.

Many readers who examine the Consumer reports ratings will question the methodology behind the ratings because many hospitals with excellent reputations, e.g., Memorial Sloan-Kettering Cancer Center, Johns Hopkins Hospital, the Cleveland Clinic Foundation, Mayo Clinic – St. Mary’s Hospital, and Virginia Mason Medical Center, receive an “average” rating.  However, Consumer Reports does not even report whether it tried to calibrate its ratings for procedures by comparing its scores for CABG with those created by the STS.

In summary, I am saddened that Consumer Reports, which is highly regarded for its product ratings, could have – in time – developed credibility in health care ratings.  Instead, it jumped the gun and shot itself in the foot.