There are a large number of health management challenges that can probably best be handled via national efforts, rather than by individual insurers, employers, or even local governments.  Charitable organizations, associations, business alliances, and other non-governmental bodies may work with or without government support on such challenges.  But there is likely to be a significant difficulty when evaluating such efforts, whenever there are more than one “solutions” being pursued simultaneously.

One classic example is that of efforts to eradicate, or at least dramatically reduce the HIV/AIDS epidemic that has been endemic in Thailand.  The “solution” chosen as most practical and effective was a national effort to persuade men to use condoms when engaging in sexual activities with Thailand’s sex workers.  But there were at least two different approaches taken to achieving this change in behavior.

One was aimed explicitly and directly at men, and sponsored by the national Population and Community Development Association (PCDA), a not-for-profit organization founded in the 1970a to promote family planning.  Sexual activities logically fit into its larger mission, and a major national effort to encourage condom use resulted.  The PDCA used a playful tone in “social marketing” to men, and engaged allies such as the “Cabbages and Condoms Restaurant” in Bangkok to help in the effort.

It included a statue made of a wide variety of colored condoms to indicate that condom use could be discussed publicly.  The restaurant, which is owned by the PDCA also uses its profits to fund micro-loans to people with HIV who want to start their own businesses.  Thailand has been extraordinarily successful in its HIV/AIDs efforts, with a 87% reduction in new infections. [J. Goldstein “Cabbages, Condoms and HIV in Thailand” Wall Street Journal Online Health Blog Apr 11, 2008 (blogs.wsj.com)]

Meanwhile, a recent book cites Thailand’s efforts to engage sex workers directly in the HIV/AIDs prevention effort.  The workers were gathered together in groups for discussion of how they could be key agents in the effort if they demanded that their male clients used a condom whenever engaging in sex with them.

This was a major behavior change for the sex workers, who normally took orders from their clients, rather than giving orders to them.  Many were worried about losing clients and thereby reducing their earnings.  Only by engaging the vast majority of them in the effort, so that men would get the same message and refusal of services from everyone in the same market, could the effort work.  And it did, with the same almost 90% reduction in HIV/AIDs incidence cited as proof. [K. Patterson, et al. Influencers: The Power to Change Anything McGraw-Hill 2008]

Of course, it is mathematically impossible for both efforts to be responsible for the same reduction in new cases.  Moreover, unless there were portions of Thailand that were subjects of one of the interventions but not the other, there is no scientific way to determine how the credit should be shared.   It may be possible to separate out men who have changed their behavior without having had any contact with sex workers, but visiting sex workers is far more common in Thailand than in the US, for example, and if only such men as had no such contact are counted as “successes” by the direct social marketing evaluation, it may be thought to have made only a small difference.

There is a pure mathematics approach to sharing the credit, which also happens to be the “common sense” approach.  If both approaches can logically be accepted as having some effect, and that effect could be anywhere from minimal to all, the best guess for the actual credit due to both would be 50%, an example of  “Bayesian” logic, or common sense.  If, for example, the estimated savings to the country are a total of $100 million, for illustration purposes, and the costs of both programs were only $10 million each, then as long as either can be credited with at least making 10% of the difference, each could be deemed successful.

Using a “split-test” approach, where half the population, randomly assigned to one or the other, but not both, of the interventions, or better one where there were four groups — 1) getting no intervention, 2) getting the direct social marketing, 3) relying on sex workers as influencers; and 4) using both approaches – might have been tried.  But if only one of these approaches actually works, it would mean denying the benefits of HIV/AIDS reduction to three quarters of the population.

There are times, particularly when the problem to be addressed is quite serious, and there is support enough to try multiple approaches, to simply adopt more than one in hopes that there will be more widespread and effective results than would otherwise have been the case.  It would normally be better, however, to include at least a modest comparison trial, which may only require a few hundred or thousand people in each, to test which one works best separately, as well as compare results of use of both or all together, since if only one is necessary or effective, it would be wasteful to invest in more than that.

Since the HIV/AIDS example is one involving a “binomial” outcome, i.e. getting the condition or not, the sample size needed to compare the success rates of the different methods can be determined in advance.  In the Normal distribution approximation of binomial distributions, the largest standard error for a sample would be that produced by an equal split of yes and no results, when it is 0.50 x 0.50 = O.25 divided by the square root of the sample size.

For a sample of 100, for example, the standard error would be 0.25 divided by 10 (the square root of 100) = 0.025.  Such a sample would have over 95% confidence in its result plus or minus two standard errors, or 0.050. As long as the results of different samples of 100 people, where different interventions have been applied, differ by at least 10%, i.e. there is no overlap between the 95% confidence range distribution of both, then the results of even such a small sample may suffice to select the best option.

To achieve 99.5% confidence, samples of 10,000 people with each of the tested interventions would be needed, but even that would not be outrageously difficult or expensive in a large population such as that of Thailand.  And even if some might argue for “leaving well enough alone”, it usually works better to know more, rather than choosing deliberately to know less, about how we invest in the health of populations.