home email us! sindicaci;ón

Evaluating Savings from Employee Health Management: Side-by-Side vs. Before/After Comparisons

by Scott MacStravic

The two most common approaches to evaluating EHM interventions, which are becoming increasingly popular among employers in general, and healthcare organizations (HCOs), since they are both employers and providers of EHM services in many cases, are:

  • Side-by-side comparisons of employees (plus dependents, retirees, when applicable) who are participating in the EHM intervention compared to those not participating
  •  Comparisons of participants in a period during which they participated in an EHM intervention, compared to a similar period prior to such participation

While common, neither is, by itself, an adequate approach to discovering both what kinds of changes have occurred with the implementation of an EHM program, and whether or not such changes can logically and credibly be attributed to the program.  Unfortunately, when one of these is used alone, or even when both are used, but not completely, the findings of evaluations can either exaggerate or understate EHM results, or give far too much credit to EHM for changes that occur due to other factors.

Side-by-Side Comparisons 

While it may seem the most logical thing in the world to do, comparing participants in EHM to non-participants is not enough to detect changes, or to ascribe them to the EHM intervention.  This is because people who choose to participate in an EHM program are, by definition, different from those who chose not to participate.  They may, for example, be younger, more or less healthy, more heavily female than male, or vice versa, etc.

Whenever there are significant differences between participants and non-participants, side-by-side comparisons may reflect no more than differences between these two groups that have nothing to do with their participation.  Moreover, when the comparisons cover the same period for both participants and non-participants, they cannot indicate any change in healthcare costs, absences, productivity, performance or value to the organization – only differences between groups.

Before/After Comparisons

If participants in an EHM intervention turn out to have significantly lower healthcare costs, absences, turnover, error rates, etc. compared to these dimensions before they enrolled, there is at least a demonstrated change.  But is that change because of the intervention, or was it due to something(s) entirely unrelated to the intervention, even to employee health at all?

The best way to have even an idea as to causality for the change is to make before/after comparisons for participants, and then look at whether these changes were different from what happened among non-participants.  By comparing two different groups, participants vs. non-participants over two different periods, before vs. after the EHM intervention, there should be at least enough information to make a logical decision as to whether any differences and changes were “real”, i.e. related directly to the intervention.

I recently read an analysis of the results of a vendor’s program, unfortunately limited to a focus on changes in healthcare costs, rather than the full array of value that employers can obtain from EHM.  It reported on side-by-side differences, and concluded that the EHM investment had saved the employer money, and realized a positive ROI after taking into account the costs of the intervention.  It used the results to calculate the “breakeven” level of participation, i.e. how many employees would have to participate in order to at least achieve an ROI of $1:1, a dollar saved for every dollar spent.

It added the step of checking on “self-selection bias”, the chance that participants were inherently different from non-participants in a significant way even before the EHM intervention.  This involved merely looking at the healthcare costs of participants before the EHM intervention compared to after.  And it was found that participants had significantly higher healthcare costs before the EHM program began, hence there really was a change in such costs.

But the analysis lacked the “rest of the story” – it did not compare the before vs. after healthcare costs of non-participants.  It is entirely possible that the before/after costs for non-participants dropped as much as such costs did among participants, perhaps even more.  If there was a significant drop in the healthcare costs of non-participants, then clearly there was something at work besides the EHM intervention.  If so, then the EHM intervention can hardly be given full credit for any apparent change realized among participants.

The use of a complete set of side-by-side comparisons of both participants and non-participants, for periods both before and after the initiation of an EHM program is essential to even basis validity for evaluations. It is by no means the “gold standard” for rigorous scientific evaluation – random assignment of individuals to participation vs. not, and control against external factors would both be needed to achieve such a standard.  But with both side-by-side and before/after comparisons –for both participants and non-participants — there is at least some security in avoiding the two most common and serious hazards in evaluation of results.

No comments yet »

Your comment

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>