Medir impacto e avaliar projetos sociais
Experiments in development: Time to rethink RCTs?
A community health volunteer gathers health data. While most senior executives who responded to a recent Devex survey aren’t aware of the debate on randomized controlled trials or have no opinion, 70 percent of those who do believe the tool is overhyped. Photo by: Nyaya Health / CC BY
Randomized controlled trials, an impact evaluation method, have been a contentious development issue of late, with as many advocates as critics.
A Devex survey of senior development executives across six continents and varied organizations revealed that nearly half aren’t familiar with the debate or have no opinion at all. This indicates that while RCTs remain a hot topic among academics and practitioners, the issue is not yet relevant to many senior managers. Another sizable group — 37 percent of executives surveyed — responded that RCTs are “somewhat overhyped.”
As billions of dollars are channeled each year toward meeting development goals like poverty reduction and basic education for all, tools to measure impact like RCTs are increasingly being used to inform policymaking.
The World Bank currently has at least 150 RCTs across its regions and sectors. The nonprofit Innovations for Poverty Action has done 400 RCTs in 51 countries.
A randomized trial IPA conducted in Kenya found that school-based deworming reduces absenteeism by 25 percent. With total cost per child amounting to less than 50 cents per year, deworming was deemed one of the most cost-effective solutions to improve school attendance. As a result of this research and the deworming efforts of IPA’s partner Evidence Action, more than 35 million children are dewormed every year.
“Deworming turned out to be very effective in bringing children to school. It is an example of how RCTs can come up with a result nobody expected,” said Rachel Glennerster, executive director of the Abdul Latif Jameel Poverty Action Lab, or J-PAL, a research center at the Economics Department at the Massachusetts Institute of Technology that uses randomized evaluations to answer critical policy questions in the fight against poverty.
But while RCTs may prove successful in measuring the impact of specific interventions, critics point out the tool has limited value when it comes to broader issues.
“They are not useful for the biggest questions in development, such as whether foreign aid undermines the contract between governments and their people or what sort of macroeconomic policy is best,” said Angus Deaton, professor of economics and international affairs at Princeton University. “Poverty reduction is a macro and political question, about which RCTs can say very little.”
Nearly half of executives are not familiar or have no opinion on RCTs.
Of the executives who do have an opinion on RCTs, 70 percent believe the tool is overhyped and should only be used in specific cases. William Savedoff, senior fellow at the Center for Global Development, noted that this could be the result of executives thinking “the pendulum has swung too far in the direction of experimental studies,” making them less inclined to favor expanding RCTs.
Seventy percent of executives who have an opinion on RCTs believe the tool is overhyped.
He added that development executives shouldn’t be too quick to dismiss the merits of RCTs and should find alternatives for evidence-based programming.
“My preference is to recognize how little real research is being done on the thousands of development interventions that are going on and to expand the overall research envelope, rather than fighting against RCTs in favor of some other method,” Savedoff said.
Devex’s survey also found that among executives who have an opinion on RCTs, there is more pessimism from executives in donor countries than in developing countries. Eighty-two percent of executives in the United States, Europe and Australia think the tool is somewhat over-hyped, while only 61 percent of executives in developing countries share that opinion.
While executives in developing countries are generally less pessimistic about RCTs, there are broad differences among regions. For instance, 48 percent of executives in Asia believe RCTs are an important innovation, while only 30 percent of executives in Africa think this.
“RCTs are indeed a welcome improvement for development practitioners,” said Hyun Hwa Son, principal evaluation specialist at the Asian Development Bank.
A higher proportion of development executives in Asia believe RCTs are an important innovation.
“At the end of the day, evaluations are however not so much about methods as they are about asking the right questions,” she added. “The more urgent and relevant point is for evaluations to consider that each development project is made unique by the political, economic and social contexts where it is implemented.”
While RCTs’ application is limited due to the trial’s context and specificity, World Bank Senior Economist David Evans argued this evidence-based tool is valuable. “Trials, like all evidence, have their place in time and space. This is true for every kind of evidence,” he said. “But I believe that a policymaker improves her odds of success by trying something that has been proven to work somewhere, ideally somewhere similar.”
Future policymakers’ application of the tool may revolve around whether RCTs can answer some of the biggest questions in development.
“Maybe no one RCT can answer a big question, but a series of RCTs can build up an evidence base to answer this question,” J-PAL’s Glennerster said, noting that because there has been a lot of work done on primary education, there now is greater understanding on how to improve school attendance and test scores.
Do you think RCTs are overhyped and or are they an important innovation? Join the conversation by leaving a comment below or tweet us at #futuredev.
What do nearly 1,000 senior-level executives from NGOs, donor agencies, corporations and the public sector think about the future of global development? View our complete series — featuring exclusive insights and interviews with top executives — to find out.
About the author
Mario Villamor
Mario is a development analyst at Devex's surveys and advisory services team. Prior to joining Devex, Mario was a researcher for the Boston Consulting Group in Boston, Massachusetts, where he supported client engagements in a variety of sectors, including public sector, global health and the social impact space, among others. Before joining BCG, Mario earned his master’s degree in global affairs from New York University.
