Survey of the U.S. News and World Report Hospital Survey

The U.S. News and World Report Hospital Survey Report has made it to the top ten hot topics of the day in "Google Trends". Although it's a good thing to have such openness about hospital mistakes that kill 200,000 people every year, we need to evaluate the criteria by which the good people of USNWR do their assessments. Therefore I have decided to respond to some of the sections of their methodology description.



USNWR: "The mission of the annual U.S. News Best Hospitals rankings has remained unchanged in 21 years: to help guide patients who need an unusually high level of hospital expertise. Other "best" lists factor in routine procedures such as hernia repair and unthreatening conditions such as mild heart failure. Best Hospitals judges medical centers on competence in complex, demanding situations, often with patients whose age or other health conditions pose their own risks. Replacing a heart valve in a man in his 90s, diagnosing and treating a brain tumor, and managing inflammatory bowel disease are a few of many examples."



Although judging hospitals as to how well they perform in critical risky situations seems like a good idea, there are some problems here. For one thing, there are too many confounding variables to make comparisons between hospitals regarding the death rates of patients with high risk conditions. For example if the researchers compare the survival rate of heart valve replacements of patients in their 90's the sample size would be too small to be of any statistical significance and the comparative analysis would have to take differences in pre-existing conditions into account.



USNWR: "For 2010-11, we ranked hospitals in 16 specialties, from cancer and kidney disorders to orthopedics and urology. A total of 4,852 hospitals were put through our statistical wringer. This year, only 152 of the 4,852 hospitals evaluated performed well enough to rank in any specialty. And of the 152, just 14 qualified for a spot in the Honor Roll by ranking at or near the top in six or more specialties."



This is a telling statement about the quality of care in the U.S. provided that the surveyors looked at things like nurse-to-patient ratios and unexpected deaths and complications.



USNWR: "In 12 of the 16 specialties, hard data such as death rates, procedure volume, and balance of nurses and patients largely determined rank. In the four remaining specialties-ophthalmology, psychiatry, rehabilitation, and rheumatology-hospitals were ranked on reputation alone; it makes no sense to take mortality data into account in specialties in which few patients die."



Excluding mortality data was a mistake since most of the deaths would be considered unexpected, requiring further investigation. The public would need to know which of those hospitals in which "few patients die" has the highest death rate.



USNWR: "Reputation (32.5 percent). Each year, 200 physicians per specialty are randomly selected and asked to list five hospitals they consider to be the best in their specialty for complex or difficult cases. The figure shown in the rankings is the total percentage of specialists in 2008, 2009, and 2010 who named the hospital."



Asking physicians to rate their hospitals is like asking chefs to rate the quality of food in their restaurants. The hospital performance is only as good as the physicians, nurses and allied professionals who provide the services. However, if you ask an off-duty nurse, you will most probably get an earful of complaints, and if you are doing a survey, it would be worth your while to listen.



USNWR: "Mortality index. A hospital's success at keeping patients alive was judged by comparing the number of Medicare inpatients with certain conditions who died within 30 days of admission in 2006, 2007, and 2008 with the number expected given the severity of their illness. An index number above 1.00 means the hospital did worse than expected and below 1.00 better than expected. Software used by many hospitals and researchers (3M Health Information Systems MS-DRG Grouper) made the severity adjustments."



The method of relying on the 3M MS-DRG Grouper is inaccurate at best. The severity scores in this software determine the amount of revenue that hospitals receive from Medicare and other third party payers. The higher the severity, the more the reimbursement; therefore, hospital billing personnel tend to exaggerate the severity data as much as possible to enhance the institution's revenue. Consequently, since the actual severity would likely be less than what is in the 3M software reports, the "better-than-expected" death rate could be actually hiding a number of deaths due to negligence. Those hospitals that did worse than expected might only be guilty of having more integrity when entering diagnostic severity codes for billing.



In conclusion, it appears that the hospital ranking methodology is grossly unreliable. There is also the possibility that there may be some biases at work in naming the top hospitals. What's seems to be missing from the entire survey is an in-depth look at which hospitals are experiencing the largest numbers never events, like death from narcotic overdose, choking, wrong surgery, surgical instruments and sponges left behind, traumatic injuries, bedsores, the percentage of hospital-acquired infections and the like. If the people in charge of the USNWR hospital survey want to get a reality check on which are the best hospitals, they should survey the nurses after guaranteeing anonymity.



Thomas A. Sharon, R.N., M.P.H.Internationally Known Nursing & Patient Safety Expert.Tom Sharon has been helping litigation lawyers turn medical law suits into settlements with incredible consistency and success over more than 25 years. His results are unparalleled in the industry: * 1,250 cases reviewed; 1,215 cases settled* * 35 court appearances; 32 successful verdicts