The evidence on evidence
Michelle Rao on evidence use, how evaluations shape funding, and a key evidence gap in development economics.
You can listen to this podcast on Spotify, Apple Podcasts, or wherever else you get your podcasts. You can also watch this conversation on YouTube.
Development economics has spent the past couple of decades building a large body of empirical evidence on a range of topics. Billions of dollars and thousands of careers have gone into attempts to causally estimate the impacts of cash transfers, deworming, microfinance, teacher incentives, nudges, conditional school grants, etc.
But we know very little about whether any of this evidence is used. There is an evidence gap on evidence use – the where, when, why, who, and how remain largely unclear.
That is the motivation behind our new Ideas in Development series on evidence, and for the first episode, I’m joined by Michelle Rao, a fellow at the Center for Global Development and one of a small but growing group of researchers trying to study evidence use empirically.
Her own research looks at 128 evaluations of conditional cash transfers in 17 Latin American and Caribbean countries between 2000 and 2015. Michelle finds no overall relationship between what evaluations find and governments’ subsequent spending decisions. We discuss that finding, the other research in this field, and more.
What development economists tend to mean by evidence
The word evidence means different things to different people. In the policy world, it has a broader meaning, encompassing descriptive data, case studies, and academic papers. Typically though, when a development economist is talking evidence, they are referring more narrowly to causal evidence – that is, the kind of evidence that tries to tell us what would have happened in a counterfactual world where a particular policy or programme had not existed.
A great deal of modern empirical economics is a set of techniques – RCTs, instrumental variables, differences-in-differences, regression discontinuity – designed to construct some form of believable counterfactual, so that we can credibly claim causality. This is the body of work that has grown dramatically in development economics over the past two decades, and it is the body of work which Michelle’s work focuses on.
Of course, other forms of evidence remain hugely important for both academia and policy. Raj Chetty’s descriptive work on mobility, Claudia Goldin’s on the labour market – none of it is causal in this technical sense, and all of it has reshaped how economists and policymakers see the world. But causal evidence has, in principle, a more direct route into policy. If you can credibly say that a given programme produced a certain effect that, in theory, maps more cleanly onto decisions – i.e. adopt it, redesign it, spend more, spend less.
From evaluation to spending
Conditional cash transfers (CCTs) are one of the most evaluated programmes in development. More than half of the 128 evaluations in Michelle’s sample were done in collaboration with the governments running the programmes, which means policymakers cannot claim not to have known about the results. If evidence-based policy is happening anywhere, it should be happening here.
To test this, Michelle matches every evaluation to subsequent spending on the same programme and explores whether resources seem to flow towards what works. The answer is no. There is no relationship between individual evaluation findings, whether you look at the framing, the size of the effect, or the direction, and what governments do with their budgets in the years that follow. Allowing for the possibility that policymakers are more sophisticated than that, and learn from the evidence base as a whole rather than individual studies, she fits a Bayesian hierarchical model to estimate the cumulative evidence base in each country, and still finds zero effect.
While in these settings, credibility and generalisability are unrelated to spending, Michelle does find an important variation in impacts based on the timeliness of evaluations. When evaluations are completed quickly and the same political party is in power for both the evaluation period and the publication of results, evaluations are predictive of spending.
Why nothing happens
The natural temptation is to read the null result as a failure of policymaker rationality or attention – they are not reading the papers, they don’t understand the methods, or they are captured by ideology. Michelle’s reading is different. The policymakers in her sample are not poorly trained. More than half of the finance ministers in her data have PhDs, mostly in economics. So the plausible problem is not a lack of understanding, but that even a policymaker who has read and understood an evaluation faces real political costs to acting on it.
Cash transfers are a particularly politically challenging setting for clean evidence-based adjustment. It’s unpopular to withdraw spending, and hard to increase it – so evidence has to clear a high political bar to translate into either expansion or contraction.
However, evaluations that arrive while the responsible party is still in office means they have ownership of what to do next, and can take credit for what worked. Along similar lines, Alix Bonargent finds that International Growth Centre research that is co-created with governments is more likely to translate into implementation. So an early insight from this evidence base on evidence use is the importance of the political conditions surrounding the production and arrival of research.
What development economics is missing
Development economics has invested enormously in generating evidence and almost nothing in studying what happens to it next.
The empirical literature on evidence use is genuinely small - I summarised what I could find here.
What is particularly striking, the more I’ve looked at this area, is that we lack even the basic descriptives we need. We don’t have good data on what kinds of evidence policymakers actually consult, how often, at what points in the policy cycle, or through which channels. While here is a great deal of institutional knowledge and useful anecdotes, on the importance of relationships, the importance of being there at the right moment, the importance of trust, very little of it is in a form that lets us test claims or design interventions. As Michelle puts it, “we need to know what we don’t know.”
This is the gap this series is trying to map out. The episodes ahead will cover both sides of the supply-and-demand problem: how research is funded, conducted, framed and communicated on one side, and what policymakers actually want, how organisations learn, and what would be needed to build genuine capacity for evidence use.
With aid budgets contracting and the political space for development cooperation narrowing, the cost of producing evidence that doesn’t have an impact is increasingly unacceptable to funders. Every dollar that goes towards funding a paper is a dollar that could have gone into a programme itself, and the case for donors funding research has always rested implicitly on the claim that the resulting evidence improves how the remaining dollars are spent. If Michelle’s data is pointing in the right direction, that claim needs more scrutiny than it has received.