EA - A dozen doubts about GiveWell’s numbers by JoelMcGuire

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A dozen doubts about GiveWell’s numbers, published by JoelMcGuire on November 1, 2022 on The Effective Altruism Forum.An entry to GiveWell’s Change Our Mind ContestJoel McGuire, Michael Plant, and Samuel Dupret[Note: We co-authored a post on deworming that has been considered for the contest.]SummaryWe raise twelve critiques of GiveWell’s cost-effectiveness analyses. Ten apply to specific inputs for malaria prevention, cash transfers, deworming, and two are relevant for more than one intervention. Our calculations are available in this modified spreadsheet of GiveWell’s cost-effectiveness analyses. We provide a summary of these first, then set them out in greater detail. We encountered these as part of our work to replicate GiveWell’s analysis. They do not constitute an exhaustive reviewMalaria prevention1. GiveWell assumes that a large fraction (34%) of the total effects of anti-malaria bed nets come from the extra income children will later earn due to reduced malaria exposure. However, GiveWell provides little explanation or justification for these numbers, which are strikingly, indeed suspiciously, large: taken at face value, they imply that bednets are 4x more cost-effective at increasing income than cash transfers.We were unsure what the income-increasing effect of bednet should be, so we investigated a number of factors, which we enumerate as the next six critiques.2. The evidence GiveWell uses to generate its income-improving figure for bednets is from contexts that seem very different from that of the charities it recommends. While GiveWell does discount this evidence it uses for ‘generalisability’, we suggest a larger discount would be consistent with GiveWell’s reasoning regarding deworming and would reduce the total cost-effectiveness by at least 10%.3. We present possible mechanisms for malaria prevention to income benefits, but they do not explain most of the effects. It’s unclear how this would change the cost-effectiveness results.4. We demonstrate how GiveWell’s analysis does not sufficiently adjust for differences in context by looking at malaria prevalence. Malaria prevalence was higher in the context of the evidence than in AMF’s context. When we adjust for this, we expect this will lead to around a 17% decrease in AMF’s total cost-effectiveness, with low confidence.5. GiveWell has missed some evidence for the effect of malaria prevention on income. We expect this decreases the total cost-effectiveness of AMF by 14%, with moderate confidence.6. GiveWell may underestimate the differences in benefits between women and men. We expect with moderate confidence that incorporating more evidence would decrease the total cost-effectiveness of AMF by 16%.7. GiveWell assumes, without clear justification, that the income effects of malaria prevention will endure in just the same way as they suppose the effects of deworming do. We’ve previously called deworming’s benefits into question (McGuire et al., 2022), incorporating the adjustment proposed there would decrease the total cost-effectiveness by AMF by 3.4% to 10.2% (low confidence)We’ve presented these changes in cost-effectiveness for changing only one variable at a time. If we implement all of them at once (excluding critiques 3 and 7 for which we are more uncertain how they’ll change the results), then the effects decline by 29% (see row 202 of “AMF total” tab).Cash transfers8. GiveWell’s analysis of the effectiveness of cash transfers only uses one study to estimate GiveDirectly’s immediate increases in consumption (Haushofer & Shapiro, 2013) when there are many more available. Using more data we show that recipients of GiveDirectly cash transfers are actually 70% richer than assumed in GiveWell’s cost-effectiveness analysis. Hence, the relative increase in consumption from GiveDirectl...