Five Star Rating Techincal Manual: Staffing Domain Accuracy

Like many of you, I have been pouring over the CMS Nursing Home Compare Five Star Ratings today.  The results have been . . . unexpected.  I operate a facility in Illinois, a state somewhat notorious in the Long Term Care industry, and when I looked at the ratings of facilities in my geographic region, I was truly surprised.  The variance is what got me.  Homes I know are pleasant, and seem to be well-run, scored much worse than those with bad reputations.  Many had perfect marks in the Quality Measures domain, but only one star in the Health Inspections domain.    

To try to get a handle on this new metric, I have also been pouring over the Technical Manual for the Five Star Rating System.  Entitled, “Design for Nursing Home Compare Five-Star Quality Rating System: Technical User’s Guide”, the 23 page report does  a fair to good job of laying out hte technical details of how the different domain scores are calculated.  My objection to this manual is that it does not give a very good rationale for why the score algorithms were selected over others.  I found myself wondering, ‘how did the Techincal Expert Panel (TEP) decide to score a domain this way instead of another.’  There are vague justifications: “distribution is based on CMS experience and input from the Project’s TEP” (page 5), but no real explanation or rationale.  

This Technical Manual may be (somewhat) easy to use then, but difficult to critique.  However, I would like to discuss one issue that bothered me.  First, I found it difficult to locate the study cited to on page five, “The Relationship Between Nurse Staffing Levels and the Quality of Nursing Home Care.”  (executive summary located here).  But, in the process of searching for this 2001 CMS staffing survey and study, I discovered the following, interesting article, first published in the scholarly journal, The Gerontologist: “Comparing Staffing Levels in the Online Survey Certification and Reporting (OSCAR) System With the Medicaid Cost Report Data: Are Differences Systematic?,” Bita A. Kash, PhD, Catherine Hawed, PhD, and Charles D. Phillips, PhD.  (abstract located here).  This article discusses some interesting results from a study of the OSCAR system.

The Techincal Manual cites the Online Survey Certification and Reporting (OSCAR) System as the source data for the staffing measures/Staffing Domain.  This is the same data used now for Nursing Home Compare, although the staffing data was not compressed into a quintile metric before the Five-Star Rating.  The OSCAR database includes essentially every piece of information a surveyor gathers during certification/licensure surveys and complaint surveys.  However, information on nursing home operational characteristics (like staffing ratios and Case-mix) are reported by the nursing homes themselves.  Inspectors review the information, but the data are not formally audited to ensure accuracy.  The study Kash, Hawes and Phillips study cited above arrives at a critical result to this operation.  The study sought to assess the validity of the OSCAR staffing data by comparing them to staffing measures from audited Medicaid Cost Reports.  The results of this study were that “[A]verage staffing levels were higher in the OSCAR than in the Medicaid Cost Report data.”  Meaning that “[C]ertain types of facilities consistently over-report staffing levels.”  This 2007 study went on to say that the implications of these findings are that “reporting errors will affect the validity of consumer information systems, regulatory activities, and health services research.”  

In other words, OSCAR data is inaccurate in reporting staffing levels and some facilities will have a lower staff-to-patient ratio than the data reflects.

What does this mean with respect to the Five-Star System?  Because the staffing ratios/RUG-III modifiers are relatively scored (i.e. a score based on how your facility matches up to other facilities in the same category), if a facility is over-reporting its staff ratios (as the above study suggests OSCAR does), then every facilities score in the Staffing Domain will be inaccurate.  If your facility happens to have a lower-ratio and higher complexity modifier than one of the innaccurate reporting facilities, your relative position will be even lower than it otherwise would have been if all facilities staffing ratios were accurately reported.  

The Staffing Domain can skew the entire star system.

Of course, it would be wrong to discard CMS’s new system based on the strength of one adverse study, but Administrators and academics need to give the whole Five-Star System a hard look.  The Staffing Domain is an important indicator of quality of care.  It should be in any comparison metric, provided the data source is accurate.  There have been a lot of criticisms leveled at the Five-Star System, but drilling down into the technical manual, I think, can provides the most salient.  

I encourage everyone reading this to do just that.  Keep this conversation going.