Pappus' plane

Separate to the question of whether the Millennium Village people are doing a good job at analysing how well the MVP is working is the question of how well the MVP is indeed working. What follows is basically the derivation of my current Bayesian prior on the MVP.

I believe with fairly high probability that the post-hoc comparisons and so forth in Pronyk et al. are of little use, but I think there is useful information to be found in the decline in under-5 mortality. I believe, with a probability somewhere around 60%^*, that the Lancet study actually understates the true gains at the MVP sites.

^*In general I think that non-experts should have a confidence at or only a smidgen above 50% on binary choices like this one. There is an enormous tension in my brain between what I feel are moderately strong beliefs, and another strong belief telling me that I shouldn't be so sure of myself. Still, if I imagine God coming down to earth and offering me a free bet with his bookmaker, my breakeven point on "Does the Lancet study underestimate the decline in under-5 mortality at the MVP sites?" would be at an implied probability of about 60%. Perhaps I'd lose money if God ever offers me these bets.

Regardless of what academics should do to demonstrate it, I think it's quite plausible that sinking a lot of money into a village, including a sizeable fraction going to healthcare, will pretty much immediately start improving health outcomes. I don't know what probability I'd put on it, but my best guess at modelling mortality rates in the MVP would involve a step function when the project starts. Below I sketch out the maths that I started edging towards in the previous post.

Let the under-5 mortality rate in the baseline period be m₁(t) = m₁(0)*exp(-k₁t). The baseline period lasted 5 years, so we'll consider t in [0, 5]. The average mortality rate over the period is the integral of m(t) over [0, 5], divided by the length of the period (i.e., 5). Set this equal to the measurement M₁ that we have of the mortality rate over the baseline period, namely what is in the paper (113.3 per 1000 births). Then for a given decay constant k₁, we can calculate the constant initial mortality rate m₁(0) and hence the mortality rate at the start of the project period m₁(5). (For those playing at home, m₁(0) = M₁²k₁/(1 - exp(-5k₁)).)

For example, if in the baseline period the annual rate of decline was 2%, then the decay constant is -ln(1-2%) = 0.0202, and we find that we get an average mortality rate of 113.3 by having an initial rate of 119.1 decaying to 107.7 after 5 years.

The follow-up period (of 3 years, though of course the maths is more general) requires a numerical solution. From the earlier work we "know", given a baseline decay constant k₁, what the initial mortality rate in the project years is. Let the mortality rate be given by m₂(t) = m₁(5)*exp(-k₂t), with (a reset) t in [0, 3]. Then the average modelled mortality rate over the follow-up period is the integral of m₂(t) over [0, 3], divided by 3. We now have to solve for the decay constant k₂, which appears both in the exponent and in the 1/k₂ that comes out the front when you take the integral. So you can't solve it by hand, but it converges in a few iterations of Newton-Raphson. Once we have a numerical value for k₂, we can convert it into an annual rate of decline in by 1 - exp(-k₂).

Now some numbers. The lower the baseline decline, the greater the decline in the follow-up period. So if we wanted to be very generous to the MVP, we would set the initial decline to be zero (or we could even posit that mortality was increasing before the MVP came along). With a baseline annual decline of zero, you end up with a follow-up annual rate of decline of 15.7%. Being more realistic, a baseline rate of decline of 2.6% gives a follow-up 11.5% (note: well above what is claimed in the Lancet). Being more pessimistic, at a baseline rate of 5.9%, the MVP's measurements would imply no change to the rate of annual mortality decline.

Any baseline rate of decline below 4.8% would imply a greater rate of decline that what is presented in the Lancet study. For the first half of the last decade, I believe with probability about 65% that this is indeed the case, and my best guess is that the rate of decline in under-5 mortality at the MVP sites is something like 10%.

There are many ways in which the model could go wrong, of course, and so I dial back my confidence that the Lancet figure is an underestimate to 60%.

So now I think that the MVP is having some success at reducing child mortality rates, and I'll guess that they're falling at about 10% per year. The recent national trends (seen in the table in this post) average to 6.4%; for rural areas we can probably add on another percentage point. The MVP would then be improving about 2.5 percentage points faster than the rest of the countries they're in.

Ballpark calculation time. The Mortality rate is 100, so 2.5 percentage points means 2.5 extra deaths prevented per 1000 births. The birth rate is about 40 per 1000 population annually, so you'd get 1000 births a year from a population of 25000. The MVP idea is to spend about $100 annually per person. So 25000 * $100 / 2.5 = $1mn per extra child death averted.

Well. Obviously the MVP have far broader goals than just reducing child mortality, but that is not the sort of cost-effectiveness that would make me consider donating to MVP. I was initially surprised at how high that number is, given that Sachs is a huge proponent of free bednet distributions – if only 1% of the budget had gone to this, then the cost per life saved should be an order of magnitude lower. But the estimation here is comparing what happened at the MVP sites to what happened in the rest of the countries, and there has been an enormous increase in the number of people in malaria-prone areas sleeping under bednets, independently of the MVP (perhaps not so independently of Jeffrey Sachs).

For completeness (so that this post is a decent record of "what I think"), I'll make one last comment, slightly more positive about the MVP and more pessimistic about the world. It is not clear what is driving the fairly consistent economic growth in sub-Saharan Africa that's occurred over the last decade. If it is largely because of a fragile commodities boom, then perhaps after the bust, the MVP sites will move well ahead of national trends. With fairly high probability, I don't think that the Millennium Villages will succeed at getting people sustainably out of poverty at anything like the rates that were originally hyped (relevant link). But perhaps the MVP suffers from the misfortune of being started just as things were getting better anyway.

In today's instalment of "rising above my station", I disagree with Demombynes and Prydz at the World Bank blog. They criticise the recent Lancet study (Pronyk et al.) on the MVP; I will focus on one part of the post, the calculation of the decline in under-5 mortality between the baseline and follow-up periods.

The authors of the Lancet study write:

The average annual rate of reduction in mortality in children younger than 5 years of age was three-times faster in Millennium Village sites than the most recent 10-year national rural trends (7.8% vs 2.6%).

I will assume that Demombynes and Prydz are correct when they write:

Child mortality is inherently a retrospective measure, as it is derived from the survival probabilities for some period before a given survey. As the paper’s appendix explains,

For the purposes of the analysis, the “baseline” period is defined as the 5 years before the intervention started; the “follow‐up” period is the first 3 years of implementation.

Thus the “year 0” or “baseline” mortality estimates in Pronyk et al. correspond to the 5-year period preceding the start of the intervention at “year 0.” The “year 3” or “follow-up” mortality estimates correspond to the 3-year period after the start of the intervention.

So we can check how the annualised rate of 7.8% was derived – Table 2 says that the deaths per 1000 births decreased from 113.3 to 88.7, a reduction of 21.7%. The average annualised rate over three years is therefore 1 - (1 - 21.7%)^(1/3) = 7.8%.

Demombynes and Prydz say that it is not correct to annualise the decline in under-5 mortality over three years:

The time elapsed between these two periods should be calculated from the midpoints of those two periods and is thus 4 years.... Pronyk et al. mistakenly treats this elapsed time as 3 years, yielding an average annual rate of decline of 7.8%. Using the correct elapsed time of 4 years, the true average rate of decline across the MV sites is 5.9%.

This somewhat subtle point may be clearer if one considers a reduction ad absurdum case. What would the correct elapsed time be for the calculation if the “follow-up” period were 3 years but the “baseline” period had been 30 years? Clearly, the elapsed time would not be 3 years, because the “baseline” period would cover the period of children’s lives and mortality risk from decades in the past, on average 15 years before the start of the intervention. (In this case, the elapsed time would be 16.5 years.) By the same logic, the mortality risk experience described by the 5-year “baseline” period took place at a point in time on average 2.5 years before the start of the intervention. The mortality risk experience described by the 3-year “follow-up” period took place at a point in time on average 1.5 years after the start of the intervention. Thus the correct elapsed time is 4 years, not 3.

There may be good arguments for using the mid-points of the intervals (Michael Clemens says "The midpoint thing is clear", and he'd probably know), but this alleged reductio ad absurdum is not one of them. It is easy to construct a reductio ad absurdum to "disprove" using the midpoints.

Suppose that under-5 mortality had been constant before the MVP, so that a baseline of any length will give the same "year 0" mortality rate. Using the mid-points method and a very long baseline, the 21.7% decline in mortality would be annualised over a long period, and the average annual rate calculated at something close to zero, even though all of the decline occurred during the three project years.

Using the mid-points gives approximately correct results if the true annual decline is constant over the whole time under study, i.e., if there is no change in the annual rate when going from baseline to follow-up periods. And in this case of constant percentage decline, annualising over 3 years would give erroneously high measured declines for baseline periods longer than 3 years (and erroneously low declines if the baseline is only 1 or 2 years).

But if there is a change in the true rate of decline as you go from baseline to follow-up period, then it is not clear what you should do. For example, suppose that the true annual rate of decline was 2% in the five-year baseline period, and 8% in the three-year follow-up. Then each year's mortality would look like this (starting from 100; I've bracketed off the baseline and follow-up periods): {100, 98.0, 96.0, 94.1, 92.2}, {84.9, 78.1, 71.8}. The average mortality in the baseline is measured at 96.1 and in the follow-up at 78.3. That's a total decline of 18.6%. Annualising over 4 years (i.e., using the mid-points) gives 5%; annualising over 3 years gives 6.6%. The method used in the Lancet paper would get closer to the true value of 8%.

In the extreme example of no decline in the baseline period, you actually get the approximately right answer by annualising over 2 years.

Demombynes and Prydz:

Pronyk et al. present an estimated annual rate of decline in rural areas of the 9 countries as 2.6% over 2001-2010. However, this estimate is based largely on trends from the first half of the decade, before the MVP started in 2006...

If we take the average annual decline during the baseline period as 2.6%, and assume that the true rate of decline during the follow-up period was 9.2%, then the mid-point method would give a rate of decline during the follow-up of 5.9%, and the Lancet method would say 7.8%. Obviously I chose the figure of 9.2% so that I end up with what the Lancet study actually reported; my point is that in this plausible scenario, the Lancet figure would be an underestimate, but closer to the truth than what Demombynes and Prydz correct it to.

Update 12 May: While it doesn't change the substance of my post, I got some of the numbers wrong. e.g., when I made the sequence {100, 98.0, 96.0, 94.1, 92.2}, {84.9, 78.1, 71.8}, I really made end-of-year instantaneous mortality rates, not what you'd measure over the course of a year – for the latter I should have calculated something close to the average of last year's end-of-year rate and this year's end-of-year rate.

The question of whether or not we should discount future utility is an important one when considering which sort of charities to donate to. If you treat all present and future human lives equally, then donating to medical research might be as cost-effective (in terms of dollars per life saved) as health interventions in the developing world today. By contrast, applying a discount rate of 3% would make you much more likely to try to save lives today, rather than those who might only be cured in 50 years' time. Furthermore, even if you are determined to donate only to some area of medical research, a discount rate will make it more attractive to fund research into a disease that is likely to be cured sooner rather than later.

Writing in the context of climate change policy, John Quiggin made a bit of a splash with this recent paper (free access), which argues against discounting future utility. Much of the paper is technical and I admit to not having read it – I believe that excessive mathematical formalism obscures the core issues on topics like these. So I will instead focus on the excerpt he posted on his blog. I've bolded the important sentence.

Much of the debate on the question of whether a pure rate of time preference can be justified is concerned with determining the appropriate way to balance the interests of “current” and “future” generations. The central question, in this framing of the problem, is whether, and to what extent, members of the current generation have the right to allocate resources in their own favour, at the expense of unborn future generations.

The central point of this note is to observe that this way of posing the problem is invalid, because members of different generations are alive at the same time. Any policy that discounts future utility must discriminate not merely against generations yet unborn but against the current younger generation. Assuming that members of any given generation are concerned about their own lifetime utility, rather than myopically concerned with current utility alone, a social allocation rule that incorporates pure time preference gives higher weight to the lifetime utility of earlier born generations than to their later born contemporaries. Assuming a 3% pure rate of time preference, as above, and 25 years between generations, the lifetime welfare of those aged 50 or more is valued twice as highly as the welfare of their children, and four times as highly as the welfare of their grandchildren, all of whom may be alive at the same time. This is obviously inconsistent with any form of utilitarianism in which all those currently alive are valued equally.

Furthermore, by the nature of overlapping generations, there is no point at which a coherent distinction between current and future generations can be drawn. In the absence of some general catastrophe, many children alive today will still be alive in 2100, at which time people already alive will reasonably be able to anticipate the possibility of survival well into the 22nd century.

I find the bolded sentence puzzling. When I consider the impact of a donation or a social policy, I give a weight of zero to past utility. "Lifetime utility" is not a concept that I would ever think of. Rather I would look at "rest-of-life utility". If those 50-year-olds have 25 years left to live, then at a discount rate of 3%, those years count as about 18. Their 25-year-old children, who have 50 years left to live, have those 50 years counted as about 27 years.

Perhaps that is wrongly myopic, but at least younger generations alive today are given more weight than older generations – there is no immediate contradiction arising from using the discount rate. Future generations who do get assigned lower lifetime utilities really are those who aren't born yet.

I am at least sometimes comfortable with this sort of myopia. GiveWell's current top two charities (Against Malaria Foundation, Schistosomiasis Control Initiative) essentially give us a choice between saving a young child's life, or reducing the non-fatal disease burden of a much larger number of people (mostly school-aged children) temporarily. There are several reasons why people with different moral preferences might prefer one charity over the other. But ignoring the many complications, in terms of numbers the choice is something like: a) $2000 to prevent a young child from dying of malaria, in countries where life expectancy is typically in the low 50's; b) $2000 to avert about 25 disability-adjusted life-years (the uncertainty in the latter figure is large). I prefer the former option, but not by much: I usually use a conversion factor of about 30 to go from a full life of about 50 years to units of DALY's. I'm implicitly using a discount rate of about 2.4%, and I'm OK with that.

But here's Quiggin again, this time from the second paragraph of his paper:

The inclusion or exclusion of a pure rate of time preference in the evaluation of social policy is a matter of great practical significance. With the most common choice of pure time preference rate (3%), the discount factor for a period of 25 years is below 0.5 and for a period of 100 years the discount factor is just over 0.05. In particular, assuming that the utility of life as opposed to death does not change over time, this implies that a policy that saved one life today would be justified even if it costs, with certainty, 19 lives a century in the future.

This is not something I worry about with donations to anti-malaria charities, but given the possibility of an increased rate of natural disasters with much higher average global temperatures, it is the sort of thing that would come into calculation when studying the effects of climate change policies. It seems to me absurd to discount lives in the early 22nd century so heavily.

So you can call my intuition confused when it comes to discounting.

On Boxing Day last year I bought the DVD set of 30 for 30, a series of sports documentaries that Twitter had told me were excellent, and which I'd caught a couple of episodes on ESPN. I've now finished watching them all, and for the interest of approximately one of you, I now give a rating and very brief review for each. Scores are out of 10, and I am scoring them as I remember them, so the earlier ones are based on two-month-old memories.

( Read more... )

That was great. Let's all say a prayer of thanks to Bill Simmons, whose idea this series was.

I am not an expert on charities. But as an interested layman, I have spent hundreds of hours thinking about the topic over the last two and a half years. In the following essay, I try to systematically describe my thoughts and feelings about giving to charity. Despite its length (5700 words), it is not comprehensive. In some areas I could go into more detail, and at least one topic has been omitted because it is of low importance and outside the main flow of reasoning.

I don't really have a target audience in mind, except that I wrote this in part for my future self – I would like a record of my thoughts and feelings on charity (I've already forgotten some of the development of my thinking), and in 2015 I'll be able to look back at this essay and remember how I felt in 2012. I am occasionally asked for my opinions on where or how to donate; it is probably useless directing curious people to an essay this long, but perhaps some excerpts will be useful for me or others to copy-paste in discussions on the topic. I also know that I occasionally do a blog search for GiveWell, and temporarily lurk in forum threads and blogs of like-minded people; maybe there are other people with this strange habit, who might enjoy the following tour of my mind. Lastly, I fondly hope that the essay will be of general interest.

TL;DR: Donate to the Against Malaria Foundation because GiveWell recommends them, and give 10% of your salary because Giving What We Can says so.

( Read more... )