Failure and learning

Learning from unsuccessful results

An unsuccessful result can help later teams learn. Charitology hypothesizes that concern about losing support can discourage disclosure in charity; the sources here do not establish its frequency or funding effect. Selective reporting also occurs in medicine. That is the Silence About Failure, and it costs the field the data it most needs: what doesn’t work, and why.

The hypothesis concerns incentives, not character. It should be tested without assuming what any individual thinks or why they disclose a result.

When good intentions do harm

The cost of silence is not only wasted money. Some programs do harm, and only careful comparison reveals it. “Scared Straight” programs take young people at risk of offending on organized visits to prisons, to deter them from crime. The idea sounds sensible. A Cochrane review of nine trials found that the programs increased offending compared with doing nothing. See the case study, and the Historical Case Studies for other cases worth learning from, such as PlayPumps and D.A.R.E.

When does charity cause harm, and through what mechanisms? That is Open Problem 20, When Charity Harms.

When the evidence itself fails

Failure is not only something programs do. Studies fail too.

In 2015 the Open Science Collaboration repeated 100 published psychology studies. Ninety-seven percent of the originals had reported statistically significant results, but only 36 percent of the replications did, and the effects found were on average about half as large. Some findings that shape how charities raise money are now being retested. A pre-registered replication published in 2024 found no support for the identifiable victim effect, one of the best-known findings about giving.

Estimates fail as well. In 2011 GiveWell found five errors in the spreadsheet behind a standard global health reference’s cost-effectiveness estimate for deworming. Together they overstated it by a factor of about 100. Charitology has not established a sector-wide rate for how often charity impact findings hold up when tested again. That is Open Problem 10, the Replication Rate, and the Replication Program is meant to work on it.

Organizations that publish their failures

A few organizations have shown that admitting failure is possible. Engineers Without Borders Canada has published failure reports since 2008, and a how-to guide based on its process, written by Fail Forward, helps other organizations do the same. GiveWell keeps a public page listing its own mistakes in research, grantmaking, strategy and operations, and the lessons it drew from them. These examples do not establish whether disclosure costs or earns trust and funding. That is Open Problem 9, the Failure Commons. Charitology’s Failure Archive is the commons in seed form. See also the Failure Lab.

Practices that turn failure into learning

Science has developed habits that make failure useful. Each has a place in charity.

Pre-registration. State in advance what you will measure and what would count as success, so a disappointing result can’t be quietly redefined. The 2024 replication above was pre-registered.
Red teams. Ask people whose job is to find the flaws to attack a claim before it is published. Charitology publishes no ratings of named charities until its method has passed public red-team review.
Scored forecasts. Before funding a program, write down how likely you think it is to work, then check. A Brier score measures how close probability forecasts come to what happened, from 0 for perfect to 1 for the worst possible. The average Brier score combines calibration and the ability to distinguish likely from unlikely outcomes. Calibration should also be checked directly across many forecasts; a score alone is not a complete calibration test. See the Forecasting Platform.
Failure reports. Publish what didn’t work, what was learned and what changed, on a regular schedule, so that disclosure becomes routine rather than an admission.

Charitology’s own failures

A field that asks others to admit failure has to do it first. Every Charitology page links to a prefilled issue for corrections and red-team challenges. Mistakes are fixed in the open, and significant ones are noted when they are fixed. Its own founding claims are labeled hypotheses, and some of them may turn out to be wrong. When the Depth Census or any other test contradicts a condition in the Diagnosis, the condition should come off the list, in public. That is what would make Charitology a science rather than a set of opinions.

Reading notebook

Private notes and reading progress stay in this browser. This is a reading marker, separate from course assessment.

Mark this chapter readQuestions, objections and connections