When interpreting the statistics is a matter of life and death

Were the Iranian elections stolen? This blog is no place for a full political analysis; as pointed out by our colleagues at FreshNetworks, other social media platforms are already doing this job on an unprecedented scale (and here’s why).

What we can do, however, is show just how important the role of proper research methodology is in such a controversial issue. For, while emotive pictures, videos and first-hand accounts abound, no-one has yet produced that indisputable piece of evidence that could persuade the disinterested.

One attempt to do so was this graph, showing votes for Ahmedinejad and Mousavi in the six ‘waves’ of votes that were announced:

The social-media chronology of the graph is as follows:

1. The graph appears on TehranBureau on Saturday, alongside the claim that “statistically and mathematically, it is impossible to maintain such perfect linear relations between the votes of any two candidates in any election — and at all stages of vote counting. This is particularly true about Iran, a large country with a variety of ethnic groups who usually vote for a candidate who is ethnically one of their own.”
2. It is republished by Andrew Sullivan at the DailyDish, one of the world’s most influential blogs, from where it passes to other such respected political blogs as GlobalDashboard.
3. Nate Silver, who runs what was the go-to site for the 2008 US Presidential election statistics, disputes these claims with an equivalent Obama-McCain graph that shows remarkably similar output. He concludes: “I am not suggesting that any and all statistical analysis purporting to show tampering in Iran’s election results will turn out to be fruitless. I am merely suggesting that this particular analysis is dubious; it is not a smoking gun.” Both the DailyDish and GlobalDashboard post this revised opinion.
4. Nate Silver’s commenters see a flaw in his comparison. (He had split the American votes alphabetically when one might assume that the splits in Iran would not be so random.) However, a DailyDish reader’s more appropriate comparison serves to support Silver’s objection.
5. Nate Silver finally receives the data broken down by province (though he cannot fully verify the accuracy of his source). At this level, the variation is much higher, with Ahmedinejad winning 78% in Semnan province and 46% in Sistan/Balouchestan. The breakdown also highlights some oddities: Karroubi won just 5% of the vote in his home province, Lorestan, and Mousavi won just one province (also his home) by a mere 3%.

In conclusion, what appeared incontrovertible at first glance looks very different in context; and more detailed data put to bed one set of questions while raising another. As such, this post is partly a plea to all experts in the field of electoral statistics to get stuck in. If ever proof were needed that our job is to help clear the fog so that abstract and difficult decisions can be premised upon accurate foundations, this is it.

This (ongoing) episode also serves to remind us of some age-old research principles:

• Benchmarking: on its own, the graph looked somewhat unreliable; not so when compared to other elections (though there is far more that could be done, such as looking at past Iranian elections)
• Granularity: what looked uniform, was, on closer inspection, merely an agglomeration of non-uniform constituent parts (assuming the figures supplied to Nate Silver are accurate)
• The wider context: a 50%-47% victory for Mousavi in his home province might seem reasonable, but it is entirely possible that those who know the area would laugh at such a suggestion. Until such evidence is properly collected, the statistics are meaningless on their own.

Ignore them at your peril: in this case, the stakes could hardly be higher.

Read more here

• Stealing the Iranian election
• Iran’s election results: the “inevitable”?
• Twittering the Iran election - and the coverage

About the author

Dave Bevan is an Interim Analyst working mainly in the Education Team at FreshMinds Research. He previously worked for the G77 (group of developing countries) at the Rome Chapter of the United Nations, and before that was a dessert chef, a tour guide on London’s open-top buses and an inconsistent stand-up comic. Dave’s interests include this, this and this.