What can randomized trials tell us?

Trials are the best test of what works

Random assignment makes groups comparable, so differences in outcomes can be credited to the program rather than to who chose to take part. Without trials, givers fall back on stories and correlations that often mislead.

Trials answer narrower questions than they seem

A trial’s estimate is unbiased only on average over imagined repeats of the trial, and it describes the people and place studied. Using it anywhere else requires knowing why the program worked and whether the conditions it needs are present, which the trial itself can’t show.

Results change at scale

Programs run by researchers and nonprofits tend to show bigger effects than programs run by governments. A trial of a pilot is a weak guide to the national program that follows it.

The deepest questions can’t be randomized

National laws, policies and social movements can rarely be assigned at random. If funders back only what can be trialed, money drifts away from some of the deepest work.

The debate

J-PAL, a research center at MIT, sets out the case for trials. Random assignment makes the groups comparable, so differences in outcomes can be attributed to the program, and chance differences shrink as samples grow. J-PAL also lists the limits: randomization is hard when a policy applies nationally, when spillovers affect the comparison group, or when the thing being studied, such as gender, can’t be assigned.

Angus Deaton and Nancy Cartwright (2018) push further. They argue that randomization makes an estimate unbiased only across hypothetical repeats of a trial, so any single trial can still be thrown off by chance. Trial samples are often convenience samples, so applying a result elsewhere needs its own justification. Whether a program works in a new place depends on its mechanisms and on whether the conditions it needs are present. In their view trials work best combined with theory, prior knowledge and other methods.

The evidence on scaling supports some caution. Eva Vivalt (2020) built a dataset of impact evaluation results and found a large amount of variation in effects. Programs implemented by governments showed smaller effects than those implemented by academics or nonprofits, even after accounting for sample size. Teaching at the Right Level shows the pattern. Early attempts to run it through government schools in two Indian states failed, later versions worked, and a trial of Zambia’s government-run program found real but smaller gains than the strongest Indian results.

Trials can also be contested after the fact. An independent reanalysis of the data from a Kenyan deworming study found coding errors and weakened some of its findings, while the original authors held that their conclusions stood. That dispute is in the Disagreement Register.

Where Charitology stands

Charitology treats randomized trials as a strong tool with known limits, and its evidence grades lean on them. Both sides largely agree that a well-run trial is good evidence for the question it tests. The real dispute is how far results travel, and the scaling evidence suggests results often shrink when programs grow. Charitology adds a concern of its own, which is a hypothesis, not a finding: methods shape funding. If only trial-ready work counts as proven, money may tilt toward the relief and prevention levels of the depth scale, where trials are easiest to run. See randomized controlled trials, the replication rate and measuring the source.

The cruxes

What would have to be true for each side to change its mind:

How well a result from one place predicts the result in another.
Whether theory and studies of mechanisms can make trial results travel.
How much weight other kinds of evidence deserve when a trial is impossible.
Whether what a trial teaches is worth what it costs.