Evidence and its grades

Computer narration. This is a reading of the open textbook, not a recorded guest interview. The text remains an editorial draft awaiting independent review. No voice was cloned. Published October 25, 2026.

Download MP3 · 5:42 · Podcast RSS feed

Transcript

The audio opens with the show name, episode title and narration disclosure, then reads the chapter below. Spoken source names link to the references in this transcript. It closes by inviting corrections and noting that every depth level is needed.

Unknown stays unknown

A science is defined as much by what it admits it doesn’t know as by what it knows. Charitology’s rule is simple: claims should make their support and uncertainty clear, and a claim that nobody has tested is called a hypothesis, including Charitology’s own. A gap in the evidence is not hidden. It is published as a finding and, where it fits, linked to an Open Problem.

Charitology uses two label systems, because it makes two different kinds of claim. Claims about the charity system, the conditions in the Diagnosis, get evidence labels. Claims about whether a particular intervention works, in the Intervention Database, get evidence grades. Both are summarized in the Method.

Labels for claims about the system

Each condition in the Diagnosis carries one of four labels:

• Established. Strong published evidence from several independent sources. The Overhead Myth is labeled established.
• Documented. Reported by credible sources, though how big the effect is remains less certain. The Missing Feedback Loop is documented.
• Debated. Credible people disagree, and both sides are shown. Fragmentation is debated: the number of charities is a fact, but whether it is too many is not.
• Hypothesis. Charitology’s working claim. It has not been tested, and testing it is the work. The Symptom Trap is a hypothesis.

Grades for interventions

Each intervention carries one of five grades:

• Strong. Several rigorous studies agree.
• Moderate. Some rigorous studies, or strong studies of a related program.
• Mixed. Rigorous studies disagree.
• Weak. Plausible, but rigorous studies are scarce.
• Unknown. The Record has not established what it does.

Insecticide-treated bed nets are graded strong. A 2018 Cochrane review pooled five randomized trials with about 200,000 participants and found that nets cut child deaths from all causes by about 17 percent. The reviewers rated that evidence high-certainty.

A grade says how sure we are that something works. It does not say how much good it does per dollar or where it sits on the depth scale. A relief program can have strong evidence and a source-level campaign can have none, and both can be worth funding.

What makes evidence rigorous

The core of rigor is a fair comparison: what happened to people who got the program, compared with similar people who did not. Random assignment can make that comparison more credible by deciding who receives the program by chance, as in a randomized controlled trial. In 2019 the Royal Swedish Academy of Sciences gave the economics prize to Abhijit Banerjee, Esther Duflo and Michael Kremer for bringing that approach to the study of global poverty. Systematic reviews then pool many trials, so that no single study decides the answer.

Rigor has limits. Angus Deaton and Nancy Cartwright argue that a trial’s result applies to the people in it, and that carrying it to a new place needs an argument about why the program worked there. Many important questions, especially about structure and source, can’t be randomized at all. Charitology asks for evidence in proportion to the decision being made, and for methods that fit the question.

Evidence can overturn intuition

Sometimes the most valuable evidence shows that a popular program does harm. “Scared Straight” programs organize prison visits for young people at risk of offending, in the hope of deterring them. A Cochrane review of nine trials found that they increased offending compared with doing nothing at all. Without a credible comparison, apparent success would not establish the effect. See the case study.

Evidence can be wrong

Labels can be lost. In 2015 the Open Science Collaboration repeated 100 published psychology studies. Of the original studies, 97 percent had reported statistically significant results. Only 36 percent of the replications did, and the replicated effects were on average about half the size. Charitology has not established an equivalent sector-wide replication rate for charity impact research. That is Open Problem 10, the Replication Rate.

The check reaches findings that matter to Charitology directly. The identifiable victim effect, one of the findings behind the Story Bias, failed a pre-registered replication published in 2024. Estimates can also contain plain errors: in 2011 GiveWell found mistakes behind the deworming estimate in Disease Control Priorities in Developing Countries, a standard reference, that overstated its cost-effectiveness by a factor of about 100. A label is a judgment about the evidence today. When the evidence moves, the label should move too.

How labels change

Anyone can challenge a label. Every page links to a prefilled issue for corrections and red-team challenges, and corrections are made in the open, with sources. Charitology publishes no ratings of named charities until its method has passed public red-team review. The Evidence Reviews and the Replication Program are meant to keep the grades honest, and the Evidence Library is meant to collect the studies behind them.

For a giver, the practical question is the one in the Giver’s Checklist: how do we know it works? Look for outcomes, not activity, and ask what the strongest study found. See outputs and outcomes.

Original chapter, reading notebook and correction links