The Secret of NIMS

March 28, 2009

This past week I attended the Life Services Network annual meeting in Chicago.  Honestly, it was a great conference, and many of the things I learned, whether in the sessions themselves or the hallways, receptions, drinks with colleagues, etc., will likely find there way, in one form or another, into this blog.  One notion that particularly caught my attention was an idea that has apparently been coming down the pike for quite some time, but with the stimulus money, finally might have the chance for real government backing and implementation.  In its simplest terms, state departments of public health are considering including nursing homes in regional emergency response teams, much like hospitals are now. 

What interested me about this development is really two things: 1) I know many of my colleagues, especially those that work in skilled nursing facilities, already work hand in glove with their local emergency management agencies.  So, of course, I was left to wonder what shape this involvement would take.  And 2) the plan to include nursing homes in a fundamental way into the emergency response plans will require training in the government’s emergency response system, the  National Incident Management System or NIMS.

The official handbook to NIMS describes its origin and usefulness in this way (link here): Since the September 11, 2001, attacks on the World Trade Center and the Pentagon, much has been done to improve prevention, preparedness, response, recovery, and mitigation capabilities and coordination processes across the country. A comprehensive national approach to incident management, applicable at all jurisdictional levels and across functional disciplines, would further improve the effectiveness of emergency response providers1 and incident management organizations across a full spectrum of potential incidents and hazard scenarios. Such an approach would also improve coordination and cooperation between public and private entities in a variety of domestic incident management activities.

The actual training in the NIMS system consists of some on-line, work at your own pace training with some short term (one hour) classroom training, and, at higher levels, or for executive management, full day, multi-day training.  Although, NHAs, as a profession, suffer from an ever expanding job description, I think it is a good idea to include nursing homes into the emergency response mix.  The simple fact is that aside from hospitals (and in someways superior to hospitals), nursing homes are the go-to infrastructural asset in the face of a large emergency.  No other type of facility has the concentration of health care technology, personnel and space to serve the public in time of great need.  Therefore, I like the idea that may be coming to a state near you; NHAs trained in the decision making process the government has created, assisting their communities through the careful application of their expansive expertise.

Like many of you, I have been pouring over the CMS Nursing Home Compare Five Star Ratings today.  The results have been . . . unexpected.  I operate a facility in Illinois, a state somewhat notorious in the Long Term Care industry, and when I looked at the ratings of facilities in my geographic region, I was truly surprised.  The variance is what got me.  Homes I know are pleasant, and seem to be well-run, scored much worse than those with bad reputations.  Many had perfect marks in the Quality Measures domain, but only one star in the Health Inspections domain.    

To try to get a handle on this new metric, I have also been pouring over the Technical Manual for the Five Star Rating System.  Entitled, “Design for Nursing Home Compare Five-Star Quality Rating System: Technical User’s Guide”, the 23 page report does  a fair to good job of laying out hte technical details of how the different domain scores are calculated.  My objection to this manual is that it does not give a very good rationale for why the score algorithms were selected over others.  I found myself wondering, ‘how did the Techincal Expert Panel (TEP) decide to score a domain this way instead of another.’  There are vague justifications: “distribution is based on CMS experience and input from the Project’s TEP” (page 5), but no real explanation or rationale.  

This Technical Manual may be (somewhat) easy to use then, but difficult to critique.  However, I would like to discuss one issue that bothered me.  First, I found it difficult to locate the study cited to on page five, “The Relationship Between Nurse Staffing Levels and the Quality of Nursing Home Care.”  (executive summary located here).  But, in the process of searching for this 2001 CMS staffing survey and study, I discovered the following, interesting article, first published in the scholarly journal, The Gerontologist: “Comparing Staffing Levels in the Online Survey Certification and Reporting (OSCAR) System With the Medicaid Cost Report Data: Are Differences Systematic?,” Bita A. Kash, PhD, Catherine Hawed, PhD, and Charles D. Phillips, PhD.  (abstract located here).  This article discusses some interesting results from a study of the OSCAR system.

The Techincal Manual cites the Online Survey Certification and Reporting (OSCAR) System as the source data for the staffing measures/Staffing Domain.  This is the same data used now for Nursing Home Compare, although the staffing data was not compressed into a quintile metric before the Five-Star Rating.  The OSCAR database includes essentially every piece of information a surveyor gathers during certification/licensure surveys and complaint surveys.  However, information on nursing home operational characteristics (like staffing ratios and Case-mix) are reported by the nursing homes themselves.  Inspectors review the information, but the data are not formally audited to ensure accuracy.  The study Kash, Hawes and Phillips study cited above arrives at a critical result to this operation.  The study sought to assess the validity of the OSCAR staffing data by comparing them to staffing measures from audited Medicaid Cost Reports.  The results of this study were that “[A]verage staffing levels were higher in the OSCAR than in the Medicaid Cost Report data.”  Meaning that “[C]ertain types of facilities consistently over-report staffing levels.”  This 2007 study went on to say that the implications of these findings are that “reporting errors will affect the validity of consumer information systems, regulatory activities, and health services research.”  

In other words, OSCAR data is inaccurate in reporting staffing levels and some facilities will have a lower staff-to-patient ratio than the data reflects.

What does this mean with respect to the Five-Star System?  Because the staffing ratios/RUG-III modifiers are relatively scored (i.e. a score based on how your facility matches up to other facilities in the same category), if a facility is over-reporting its staff ratios (as the above study suggests OSCAR does), then every facilities score in the Staffing Domain will be inaccurate.  If your facility happens to have a lower-ratio and higher complexity modifier than one of the innaccurate reporting facilities, your relative position will be even lower than it otherwise would have been if all facilities staffing ratios were accurately reported.  

The Staffing Domain can skew the entire star system.

Of course, it would be wrong to discard CMS’s new system based on the strength of one adverse study, but Administrators and academics need to give the whole Five-Star System a hard look.  The Staffing Domain is an important indicator of quality of care.  It should be in any comparison metric, provided the data source is accurate.  There have been a lot of criticisms leveled at the Five-Star System, but drilling down into the technical manual, I think, can provides the most salient.  

I encourage everyone reading this to do just that.  Keep this conversation going.