A counterfactual is what would have happened otherwise. The counterfactual impact of a gift is the difference between the world with the gift and the world without it. A program’s results and a gift’s results are not the same thing. If the program would have been funded anyway, the gift’s real effect lands somewhere else.
Three ways the counterfactual changes the picture
Funding gaps get filled. Benjamin Todd of 80,000 Hours warns against treating a charity’s funding gap as if you were the only donor. If other donors would have filled it, your gift does less than it seems. The opposite mistake is also possible: assuming your gift does nothing because others would give. Filling the gap frees those donors for other work and saves the charity fundraising time, so the honest answer depends on what everyone else does.
Other funders respond. GiveWell uses two words for this. Leverage is a charity’s influence on how others, such as governments, allocate their money. Funging is when spending on a program crowds out money that would have come from somewhere else. Since late 2017 GiveWell has estimated both for each charity it recommends, and it says the estimates involve substantial guesswork. Its test of “room for more funding” asks the same question from the charity’s side: what would extra money achieve, beyond what the charity would raise anyway?
The comparison group decides. A program can look like a success until you see what happened to similar people without it. A Cochrane review of nine trials of “Scared Straight” programs, which organize prison visits for young people at risk of offending, found that they increased offending compared with doing nothing at all. See the case study.
Shared credit
Most outcomes have many causes. Todd points out that when two groups are each necessary for a result, both can be counterfactually responsible for all of it, so claimed impacts can add up to more than 100 percent. That is not a logical error. The practical risk is that funders, each counting the whole result, collectively pay more than it is worth.
Why it is hard
The counterfactual is never observed. Randomized trials estimate it for the people in a study. Ordinary gifts get no estimate at all, which is part of why the feedback loop is missing. Producing counterfactual estimates cheaply and at scale is Open Problem 4.