A randomized controlled trial (RCT) assigns people, villages or schools to receive a program or not by chance, like a coin flip. Because chance decides, the groups are expected to be alike in everything except the program. The difference in their outcomes then estimates the program’s effect. A trial is a way of seeing the counterfactual: what would have happened without the program.
In development economics
In 2019 the Royal Swedish Academy of Sciences awarded the Prize in Economic Sciences in Memory of Alfred Nobel to Abhijit Banerjee and Esther Duflo of MIT and Michael Kremer of Harvard “for their experimental approach to alleviating global poverty.” The Academy said they split poverty into smaller, more precise questions, often best answered by carefully designed experiments among the people most affected. It credited one of their studies with leading to remedial tutoring programs that have reached more than five million children in India. J-PAL, the research center Banerjee and Duflo founded at MIT in 2003 with Sendhil Mullainathan, says its network has led more than 2,700 randomized evaluations.
What trials can show
Trials can confirm that something works and show how much. A Cochrane review of five trials with about 200,000 participants found that insecticide-treated nets cut child deaths from all causes by about 17 percent, and rated that evidence high-certainty.
They can also show that something does harm. A Cochrane review of nine trials of “Scared Straight” programs found that they increased offending compared with doing nothing.
The critique
Angus Deaton and Nancy Cartwright argue that the public, and sometimes researchers, put too much trust in RCTs compared with other methods. Randomization does not guarantee that the groups in any one trial are equal in everything but the treatment. At best a trial gives an unbiased estimate, which they say is of limited practical value on its own. Its result applies to the people in the trial, and carrying it to a new place or population needs a further argument about why the program worked. They see trials as useful inside a cumulative program of research that combines them with other methods.
Where trials stop
Many of the questions Charitology cares most about cannot be randomized. Some national laws and cultural changes cannot feasibly be assigned at random, though policy rollout or components can sometimes be randomized. Charitology hypothesizes that limits on trial evidence can affect funding for structure and source work. Finding other ways to measure it is Open Problem 3. How often trial findings hold up when repeated is Open Problem 10.