fatvegan

Everyone is upset about the The Human Cost of War, research published in the Lancent on the number of deaths in post-invasion Iraq. People figure the number is way too high and assume there must be a problem with the statistics. In the sample they covered, 289 people died violently, which was 2.26% of their sample. They extrapolated that number to get their figure of something around 601,027 for the whole of Iraq. The 95% confidence interval was between 426,368 and 793,663. Some people initially complained that the range was so high that the whole study was meaningless. I think enough statisticians have written about why that isn't the case. But a few people still think the representativeness of the sample must be bad. Iraq Body Count is one group who have said this although I think they're unconvincing and verging on misleading.

People have said that for the researchers to get a number so high, they must have sampled from an area of the country where violence was higher than normal. While it is possible that the sample isn't representative, this criticism is kind of missing the point of the confidence interval. The confidence interval tells you about the likelihood of getting a sample so unusual that the true number is outside that interval. If you knew the sample was representative, you could just measure the deaths in that sample, and work out the total number by multiplying. You wouldn't have a range of values at all. Unless there is systematic bias in the sample, you can have a lot of confidence in the confidence interval.

I've read the paper, and I don't really have doubts about the randomness of the sample. I don't think there was systematic bias in the sample selection, and probably not in the surveying process either. Their methods were very rigorous, and self-selection bias does not appear to be an issue. Virtually everyone they approached was willing to participate. The vast majority (92%) of reported deaths were backed up by death certificates so people aren't just making deaths up. The researchers chose to omit one of the areas they surveyed from the final results because the death rate there was so high. So obviously haven't got a totally crazy sample. There are places where the death rate is much higher.

When you start to think about it, the number doesn't seem so extreme. The paper pointed at that if three violent deaths per day occurred in each of Iraq's major urban centres outside Baghdad that would equal about 270,000 deaths plus however many died in Baghdad.

The majority of deaths counted by Iraq Body Count occurred in Baghdad. 43 of the 50 most violent events occurred in Baghdad. Those 43 Baghdad reports account for 50% of the total for the whole war for the whole of Iraq. Virtually all (maybe 95%) of the very violent events recently reported occurred in Baghdad. Baghdad represents a fifth of the Iraqi population, but far more than half the total number of deaths reported by Iraq Body Count. Maybe people are only dying in Baghdad, but that definitely isn't what the Lancent survey found. Also Iraq Body Count doesn't count military casualties, while the Lancet study counted all deaths, and that could cause huge distortion.

The same research team did a similar survey in 2004. The results of the first survey were confirmed by the second survey, despite a completely different sample. The results of the first survey were also confirmed by a separate UN survey done at the same time. Both surveys found a similar number of war-related deaths.

I think the article is very credible. The problem with the number of of 601,027 isn't that it's unreasonable. The problem is that it's too awful to accept without flinching and looking for reasons why it is wrong. If you don't believe that people can die this fast, read about Sudan.

Update: One issue I haven't heard anyone else talk about is the problem of population migration. If the survey uses population densities from 10 years earlier then the house selection is not biased against violent areas. However, it should be biased against violent areas because there is bound to be some migration out of violent areas and into peaceful areas. So you are more likely to pick a house in a violent area, than you should be based on current population distributions. The sample will be systematically biased towards violent areas, but it's probably hard to know how significant the bias is without knowing the extent of the displacement.