“If a program sounds logical, it works”

Why people believe it

Most everyday decisions run on plausible reasoning, and most of the time that works well enough. A program’s logic is visible the moment you hear it. Its results take years, and a comparison with people who didn’t take part, to see. So the logic does the persuading.

What’s true

Logic matters. A good program starts with a theory of change, a chain of reasons from what it does to the change it wants, and writing that chain down is a good way to find its weak links. And evidence can tell sound logic from unsound. In a 1994 review of rigorous studies, interactive programs that built social skills did better at preventing drug use than D.A.R.E.’s original lessons.

What’s false

That a convincing theory is enough. Three cases:

• Scared Straight. The logic was deterrence: show teenagers what prison is like and they will avoid it. A 1978 documentary reported a 94 percent success rate with no comparison group. When researchers ran randomized trials, the programs increased the odds of later offending by a factor of about 1.6 to 1.7. On average, the teenagers did better without the program.
• D.A.R.E. Police officers teaching children to resist drugs became the most widely used school drug prevention program in the United States. Ten years after sixth grade, young people who had gone through it were doing no better on drug use than those who had standard drug education.
• One Laptop per Child. Giving every child a laptop seemed a direct route to better learning. A randomized evaluation in rural Peru found children used computers far more, but their math and language scores did not improve.

In each case the logic was clear, and only measurement against a comparison group showed what was really happening. See the case studies on Scared Straight, D.A.R.E. and One Laptop per Child.

What to ask instead

Compared with what? Has anyone checked what happens to similar people who don’t get the program? That comparison is the idea of counterfactual impact, and a randomized controlled trial is one strong way to test it. How often promising findings hold up when repeated is Open Problem 10, the Replication Rate.