This post was updated on March 23rd to fix a few minor typos (in blue) and link to a few thoughtful tweets we received in response. In addition to acknowledging Amanda Glassman’s incredible efficiency at expressing complex thoughts in 140 characters (seriously!), we should have done a better job of emphasizing that our goal was mainly to highlight how related strategy goals can create tensions amongst one another when it comes time to implement and evaluate global health programs in general, rather than pick any bones with SM2015. Here are links to our twitter responses.
Inspired Foreign Aid Reform, or Just Another Race to the Bottom?
The Center for Global Development recently blogged a new public-private partnership known as Salud Mesoamerica 2015. As a collaboration funded on the donor side by Carlos Slim, Bill Gates, and the Government of Spain, this exciting initiative incorporates several popular policy ideas in global health.
We’re excited to see this program get off the ground and believe it may hold significant promise, but it also seemed like a timely opportunity to spotlight some of the ways conflicting policy objectives in global health can undermine strategy execution.
Our goal here is not to criticize this particular program or its design per se, but rather to point out how seemingly related programmatic goals can conflict with one another when put into practice.
Targeting the Poorest (Goal #1) and a Cost-effective Package (Goal #2)
The poorest populations (goal #1) are almost never the most cost-effective to target–and for the same reason, most cost-effectiveness data pertaining to services (goal #2) does not come from high-quality evaluations of programs targeting services to the poorest populations in developing countries.
CGD smartly points out that delivering interventions cost-effectively (goal #5) can be learned through iterative evaluations. But the results of cost-effectiveness analyses and impact evaluations could still be quite misleading–and potentially for quite some time–if:
- the effects of an intervention outlasts the time horizon for program evaluation, but are not captured or estimated with uncertainty–as you might expect when only 18 months elapse between baseline and endline evaluations
- the marginal cost of the intervention itself is tiny relative to the costs of delivering, managing, or evaluating interventions in general, and especially when rolling out an intervention quickly—as is arguably the case with The Global Fund, which only spends ~37% of its budget on actual commodities, including the second-line treatments that are “100 times or more as expensive as first-line regimens” as mentioned by CGD
- an intervention is expensive to deliver because its coverage is low, whereas economies of scale can “bend the cost curve” in ways that may not be obvious from cost-effectiveness analyses, or are impossible to observe over an 18-month time horizon
Why are these trade-offs important? The poorest populations are usually also the most expensive to target because they fundamentally lack access to resources. Without first building or strengthening a functional “delivery channel” for interventions in general–which inevitably implies significant start-up costs and risks–many high-impact services will not be cost-effective to deliver, at least as long as priority setting exercises assume the status quo. And yet by focusing on the most cost-effective interventions (and by extension, avoiding investments in health systems whose “results” or “impact” may be much more difficult to predict or measure in a specific context), establishing a track record of “rapid impact” further undermines the investment case for longer-term, structural reforms to health systems because all of these decisions are made on the margin, based on overlaps between donor priorities, government priorities, and patient priorities. What’s cost effective given this “sweet spot” of overlap is rarely the most cost effective for governments in the long term.
Cost-effectiveness analyses allow analysts to generate cost effectiveness estimates for diarrhea interventions, for example, but the sensitivity of these estimates to context is often ignored, unknown, or underestimated. Another issue is the role of underlying secular trends in the dynamics of cost effectiveness estimates over the lifecycle of its deployment vs. the individual investment decisions made upfront, which are typically based on data that are out of date by several years. By improving coverage of oral rehydration salts to reduce diarrhea mortality, for example, the investment case for sanitation improvements may look weaker in comparison based on analyses of survey data from 2005 or GBD estimates from 2010. By investing in rotavirus vaccines–one of ~70 known etiological causes of diarrhea–the investment case for sanitation also becomes weaker in comparison. What PPPs relying on cost-effectiveness analysis often forget, however, is that improving coverage of oral rehydration salts also undermines the investment case for rotavirus vaccines over time, and not just the investment case for sanitation.
In other words, the interventions deemed cost-effective based on data from 2005 may not still be the most cost effective in 2013, and 18 months between baseline and endline is still plenty of time for the poorest to experience real effects of vaccine roll-out or even plain old economic growth, which can hit 7-10% annually in some developing countries and whose sub-national effects on the purchasing power of the poorest families might be even larger. These hyper-local economic effects and the geographical prioritization of the scale-up of other interventions mean that even randomized evaluation–and especially cluster randomized evaluations–may still be significantly biased and/or have limited external validity…potentially even within a country.
The effects of underlying secular trends on the cost, coverage, and effectiveness of competing interventions often means that what appears to be cost-effective when an investment decision is made based on data from 2005 may be much less so by the time a program’s baseline data are collected in 2013, which in turn may be further diluted by secular trends and competing interventions inside and outside the health sector between 2013 and 2015.
Targeting the Poorest (Goal #1) and Money Attached to Results (Goal #4)
A similar tension exists between targeting the poorest (goal #1) and attaching money to results (#4). Performance-based financing is a powerful and potentially promising concept, but implementers (both before and after transition) will always be incentivized to prioritize the most cost effective way to deliver the most cost-effective interventions, which (again) almost never involves reaching the poorest except to the extent that it’s mandated. This leads us to another set of conflicting goals.
Incentives for Governments to Take Over the Job (Goal #3) and Targeting the Poorest (Goal #1)
Incentives for governments to take over the job (goal #3) and target the poorest (goal #1) imply what might be a politically implausible “theory of change” in the long-run because it assumes that governments will experience sustained incentives to spend public funds on their poorest citizens, enabling donors to “get out of Dodge”.
Are the poorest not being reached by health interventions because the government doesn’t know which interventions are cost-effective, because the government doesn’t have enough money to provide them, or simply because the poorest aren’t an important political constituency? Even if the answer is “all three,” who will hold the government’s feet to the fire once donors have “gotten out of Dodge?”
The poorest are often politically disenfranchised, and getting half of a country contribution back for a successful program may not be a strong incentive when the counterfactual is the government spending almost nothing on the poorest to begin with. A government accepting half-free or mostly-free aid money is not the same thing as creating and sustaining political pressure from its poorest citizens (or even non-poor citizens) to ensure that programs deemed a success continue. Many foreign aid programs emphasize the accountability of developing country governments to donors in the short-run, which is not the same thing as those governments becoming and remaining directly accountable to their poorest citizens in the long-run.
Cost-Effective Package (Goal #2) and Money Directly Attached to Results (Goal #4)
The goals of focusing on cost effective intervention packages (goal #2) and attaching money directly to results (goal #4) also conflict. Will the intervention packages selected (ignoring the non-additive effects of bundled interventions for now) be determined by cost-effectiveness and what can be delivered cost-effectively as described, or will governments and/or donors instead opt for the interventions or intervention packages whose results are easiest to measure over 18 months? Will performance-based financing favor “soft” metrics like survey-reported behavioral observations on, say, bednet use—or will these programs gravitate towards “hard” outcomes like medical autopsies documenting reductions in mortality attributable to malaria (instead of, say, verbal autopsy)?
We certainly know which evaluations are cheaper, but do the Hawthorne effect and social desirability bias disappear from household survey evaluations if they’re independently conducted, especially if beneficiaries know that continued funding depends on their survey responses? Is it even possible to hire an independent evaluator whose business model fundamentally depends on having additional programs to evaluate? We wonder.