10/18-19/2006 - Election day

I have a number of thoughts about the mid-term elections, what they mean, what they foretell and why I should care, but the post that I really want write is about forecasting elections. As a professional (insurance) and amateur (sports) forecaster, I always get swept away in the excitement of predicting the results of big elections. It goes beyond politics, it's just and exciting thing to forecast. Much more exciting then, say, the weather.

Usually, when I do my modeling, I have the luxury of using fairly objective data: money, hits, yards, minutes, etc. This is not the case in modeling elections, where data includes such things as approval and intensity. And even so, it could be objective, but approval and intensity are measured by asking people questions and writing down their answers. People may lie, overestimate, change their minds, all of which goes to making these political forecasts that much more suspect.

Going back to 1996, it seems to me that polls have tended to understate Republican results. I distinctly remember that Dole came closer than the polls would have guessed, and that Gore was actually polling ahead of Bush (by more than the popular vote eventually suggested). Whenever the results would deviate from the polls, various pollsters would talk about Republican turnout and how they were the more motivated electorate. Well, it turns out that polls measure the likelihood of "likely voters" to actually vote. The term is called "intensity." A typical question might go something like this:

Q: What party are you likely to vote for next Tuesday?
A: Democratic
Q: How strongly do you feel about voting for the Democratic party?
A: Very strongly.

So you can see, it's not entirely objective. But polling groups make up for the lack of objectivity by maintaining consistency. If the wording stays the same from election to election, then changes in the results mean something, although it's not always clear what those changes mean.

For those who followed the most recent election (including, I believe, 100% of my readers) there will be no surprise when I say that there was a huge swing in "intensity" in 2006. Republican intensity decreased. Democratic intensity increased. Why? Who knows. The pundits have offered

• Iraq war
• Backlash against social conservatism
• Decrease in fiscal conservatism among Republican candidates
• Scandals

But we don't really know (although some polls certainly tried to isolate the factors). When I created my own forecasts of the election results (of course I created my own forecasts of election results) I essentially discounted intensity. Given that it, in my experience, it has always been an explanation for the same bias -- underprediction of Republican results -- I gave it little credibility. I was wrong. This year's elections will bring about a host of changes. One of them will be my increased confidence in measuring the intensity of opinion by polling groups.

So now would probably be an opportune time to roll-out another of my favorite bloggers, one who hasn't yet been added to my links. The Mystery Pollster is my favorite polling blogger. He's not the only one, but he does a good job explaining some of the technical stuff. In the run-up to the elections I was reading him daily. Among the stuff he does well, I would include consolidating lots of polls, defining industry terms, focusing on important issues. He does occasionally wander off into his own hobbies (1,000 posts on "push-polls") but he's good at what he does, so I bear with him.

26,500 steps today. Standard error = 2.5%