Understanding more about Bayesian analysis

Since I finished reading Nate Silver's book "The Signal and the Noise" (you can read my review of it here) I've been trying to find a way to describe the difference between Bayesian and standard statistics.

As I understand it standard, or frequentist statistics, the kind we were taught in school, asks the question: given a set of data, what is the frequency that a particular phenomenon will occur? According to Silver, this way of looking at a question means that we are thinking hard about the accuracy of our measurement (but assuming that we are measuring what we want to measure).

Bayesian statistics, on the other hand, asks the question: given a certain outcome or set of data, what is the most likely cause (or causal chain) for that outcome? Again according to Silver, Bayesian statistics allow us to think about how certain we are we know something.

Here's a relatively simple explanation of the math. 

 
And here is a more complex one:

The power comes from the ability to vary the different scenarios. Using a Monte Carlo simulation the analyst builds a model but substitutes a range of values for any factor that is uncertain. That's what Nate Silver does in the fivethirtyeight.com analysis for his blog, as you can see when you read his methodology. (You can read Jim Manzi's book 'Uncontrolled' for a look at the same thing using big data.) Acknowledging and accounting for uncertainty means that you get better results in the long run - as in the submarine search example in the first video above.  

Why weren't we taught it? Two reasons. First, running these simulations (Silver talks about running 10,000 a day, and that was in 2008) takes a lot of computing power, power that has only recently become available. Second, because Bayesian analysis starts with what we think we know, with a greater or lesser degree of certainty, some philosophers of science have argued, for various reasons, that Bayesian analysis failed to take account of the problem of induction: ie, that the only true knowledge comes from deduction. (I admit I am way oversimplifying here.) (Silver has a very interesting chapter on his discussions with Donald Rumsfeld about unknown unknowns). This view is now being rebutted. If you are interested, there's a good and reasonably accessible paper, "Philosophy and the practice of Bayesian statistics" written by Andrew Gelman and Cosma Shalizi available here.

The Signal and the Noise, by Nate Silver

I've been a fan of Nate Silver's work since the 2008 election when I, like perhaps many of you, obsessively checked his blog. I've always thought that his writing is clear and that he is transparent - to a point - about his methodology. So I was eager to read his very interesting book, "The Signal and the Noise."

What Silver sets out to do in this book is explore our ability to make predictions based on big data. Silver's main thesis is that we should be using Bayesian statistics to make and judge our predictions about the world. As Silver puts it,

The argument mad by Bayes and Price is not that the world is intrinsically probabilistic or uncertain . . . It is, rather, a statement . . . about how we learn about the universe: that we learn about it through approximation, getting closer and closer to the truth as we gather more evidence. [Italics in original.]

As Silver acknowledges, this approach is not the one we are taught in school (or in classes in the history and philosophy of science. For a review of that approach, read the first third or so of Jim Manzi's book "Uncontrolled." My review of "Uncontrolled" is here.) Instead, Silver argues, we use statistics that focus on our ability to measure events. We ask, given cause X, how likely is effect Y to occur? This approach raises lots of issues, such as separating cause from effect - we get mixed up a lot about the difference between correlation and causality. We mistake the approximation for reality. We forget we have prior beliefs, so allow our conclusions to be biased.

In contrast, Silver explains, the Bayesian approach is to regard events in a probabilistic way. We are limited in our ability to measure the universe, and Pierre-Simon Laplace, the mathematician who developed Bayes' theorem into a mathematical expression, found an equation to express this uncertainty. We state what we know, then make a prediction based on it. After we collect information about whether or not our prediction is correct, we revise the hypothesis. Probability, prediction,  scientific progress - Silver describes them as intimately connected. And then he makes a broader claim:

Science may have stumbled later when a different statistical paradigm, which de-emphasized the role of prediction and tried to recast uncertainty as resulting from the errors of our measurements rather than the imperfections in our judgments, came to dominate in the twentieth century.

Silver describes the use of Bayesian statistics (to greater or lesser rigor) in many contexts, including sports betting, politics, the stock market, earthquakes, the weather, chess, and terrorism. We are better at predictions in some of these contexts than we are in others, and he uses the chapters to illustrate various corollaries to his main theme. In his first chapter, on the 2008 financial meltdown, he identifies characteristics of failed predictions: the predictor focused on stories that describe the world we want, we ignore risks that are hard to measure, and our estimates are often cruder than we think they are. On the other hand, in a chapter about sports data, he makes a compelling case for the premise that a competent forecaster gets better with more information. Throughout, he urges us to remember that data are not abstractions but need to be understood in context.

This is not a how-to book, and it certainly left me with many questions. How do you test social programs using Bayesian analysis? But it is a very good starting point.

Image via amazon.com

Election predictions

In case you might have missed it, once again Nate Silver correctly predicted the outcome of the election. You can follow his thoughtful description of his model and regular updates on his blog.

I am reading Silver's book, "The Signal and the Noise," now, and will post a review sometime next week. In the meantime, here is a post describing Silver's outcome (and response) in various media. (That's a screenshot of graphic artist Christoph Niemann's take on it.) Basically,

Not a bad night for the math nerds. However, the truth—which Silver would readily admit—is that he didn't really "predict" anything. The math did ... and the math was based on polls, which are also based on math. He pulled them all together and came back with a number, which was very useful (and comforting to Democrats), but not magic. 

 And here is an Atlantic.com post showing how various pundits' predictions succeeded (or, in most cases, did not).

Skewed political polls?

Update, October 17: Here's a link to Nate Silver's more serious NY Times column on the temptations, and risks, of cherry picking poll results.

Here's a link to Nate Silver's take on the skewed polls claim. The bottom line? Over the last few decades of political polling, the polls have been pretty balanced, making mistakes in one direction one year, and in the other in other years. Four years ago, they got the presidential election exactly right. Silver points out that the "oversampling" criticism is "largely unsound." He goes on:

But pollsters, at least if they are following the industry’s standard guidelines, do not choose how many Democrats, Republicans or independent voters to put into their samples — any more than they choose the number of voters for Mr. Obama or Mitt Romney. Instead, this is determined by the responses of the voters that they reach after calling random numbers from telephone directories or registered voter lists.

Pollsters will re-weight their numbers if the demographics of their sample diverge from Census Bureau data. For instance, it is typically more challenging to get younger voters on the phone, so most pollsters weight their samples by age to remedy this problem.

But party identification is not a hard-and-fast demographic characteristic like race, age or gender. Instead, it can change in reaction to news and political events from the party conventions to the Sept. 11 attacks. Since changes in public opinion are precisely what polls are trying to measure, it would defeat the purpose of conducting a survey if pollsters insisted that they knew what it was ahead of time.

For another take on the issue, see this Stephen Colbert segment (I've linked it as well as embedded it).

The Colbert Report	Mon - Thurs 11:30pm / 10:30c
"Skewed" Presidential Polls
www.colbertnation.com
Colbert Report Full Episodes Political Humor & Satire Blog Video Archive

I'm coming late to this, I realize, but if you haven't already read it, take a look at Nate Silver's article about weather prediction in the NY Times Magazine of September 9. It's a good explanation of the uncertainty that goes into making predictions based on statistical models, and the assumptions and corrections that are required.

Silver demonstrates how dramatically weather predictions have improved over the last few decades. For an example, you can see my post from last year about the predicted path of Hurricane Irene here, and a follow up post here.

The article is an excerpt from Silver's forthcoming book "The Signal and the Noise: Why So Many Predictions Fail - but Some Don't."

Nate Silver on Romney's VP pick

As you know, in election years I like to check in on Nate Silver's blog FiveThirtyEight, (eg here, here, and here) which is consistent, clear, interesting and well written. He's doing a particularly interesting job relating the impact of Romney's pick of Paul Ryan as a running mate. There have been a lot of theories for the pick floating around: Ryan is a Washington insider, he'll bring out the base, he understands the budget issues. (And don't forget to read Samuel Popkin's take on James Fallows' blog, here.)

I'm a numbers person, so I would have gone for someone who I thought might help me in a swing state, like Florida or Ohio. That means someone from a swing state who is popular in the state. Here's Silver's analysis of the likely VP choices as of August 8:

According to Silver, Florida, Ohio and Virginia are in play (as of now); Wisconsin and New Mexico are leaning Democratic. Silver says:

 Five candidates stood out as having especially strong positive ratings with their home-state voters. These were Mr. McDonnell of Virginia, along with Govs. Susana Martinez of New Mexico, Bobby Jindal of Louisiana and Brian Sandoval of Nevada, and Senator John Thune of South Dakota. The ratings for Mr. Christie and Mr. Rubio were also fairly strong.

Another of Mr. Romney’s potential choices, Representative Paul Ryan of Wisconsin, drew more mixed reactions. Although Mr. Ryan should win his home district, pollsters who tested his numbers throughout Wisconsin found more tenuous results, with 38 percent of voters giving him a positive rating and 33 percent a negative one.

Silver's model predicts (remember, this is as of August 8) that the pick of Ryan would increase Romney's chances of winning the Electoral College by 0.1%. That's not much.

You can read Silver's analysis of the bounce Ryan gave the ticket last week here. His bottom line? The bounce was "below-average." But, as Silver points out, the bounce now is not what's important. "If Mr. Romney makes a one-point advancement in the polls and holds it permanently from his selection of Mr. Ryan, then this would count as a successful vice presidential selection: a one-point shift in the polls is actually fairly meaningful given how close this race is."

And today, after giving it some thought, Silver argues that the selection, and the outcome, will give us some insight into the country's longer term political shifts. Read the post - the bottom line in particular is pretty interesting in the context of the full discussion.

Recall vote in Wisconsin as harbinger for the 2012 general election? Not so fast

Governor Scott Walker of Wisconsin survived yesterday's recall vote despite the passion, hard work, and money poured into the recall efforts. Many news organizations are using the vote to discuss the future of organized labor. Others wonder whether the vote sends a signal about improving Republican chances in the November presidential elections.

Nate Silver posted a very good column yesterday analyzing his suggestion that races for governor can sometimes give contrary indicators for presidential elections. Here's what he says:

But one thing that the recall is unlikely to do is tell us much about how the presidential contest in Wisconsin is likely to evolve in November. The politics for a governor’s campaign are often subject to different currents than presidential ones, and historically the party identification of a state’s governor has said little about how presidential candidates will fare there.

Over the past 40 years, in fact, the relationship has run in the reverse direction than you might expect. The Democratic presidential candidate has typically done a little better when the state’s governor is a Republican, and vice versa.

Why is this so? As usual, Silver is clear in his explanations, providing two tables, one showing presidential vote margins by party of state governor, and the other, slightly more refined, showing presidential vote shift by party of state governor. But correlation is not causation, and Silver is always careful to remind readers of that fact. He offers two hypotheses and one caution. One hypothesis is that voters like balance in their elected officials; the other is that some voters tend to vote for the incumbent. The caution is that the aggregated data may hide some factors. This counterintuitive suggestion, backed up by numbers, is an interesting addition to the discussion. Do you agree?

Consider the complications

One of the many things I like about Nate Silver's approach to interpreting numbers is that he takes the time to explain his methodology clearly, supports his decisions, and sets out the advantages and disadvantages of his choices. One example is last week's column titled "Why is Santorum Overperforming his Poll Numbers?"

As Silver puts it:

The FiveThirtyEight forecast model is based on statewide polls and statewide polls alone in presidential primaries and caucuses. It doesn’t do anything especially fancy. This approach has advantages and disadvantages.
The disadvantage is that if the polls are wrong, the model will be wrong too. . . 

The advantage is that the model gives us a relatively clean and objective benchmark to evaluate the candidates’ actual performance against the polls.

He then compares his model against Santorum's performance in the states with robust polling, and in states with limited polling, and concludes that Santorum has outperformed the polls by a small amount - 2.3 percentage points in the smaller set, and 2.1 percentage points in the larger set, enough to be statistically significant. Silver offers fours hypothesis (higher Santorum turnout, neglect of cellphone-only voters, unwillingness to reveal choices to poll takers, and tactical voting) that might explain the difference, and considers each of them in turn.

It's a good piece, clearly written. It enlarges our understanding of what's happening.  (And in case you're wondering, Silver thinks Santorum is unlikely to get the nomination.) Because it continues the conversation as events develop, this kind of iterative approach is a good model to follow for anyone who has to explain numbers for a living.

A couple of interesting articles from the weekend

The NY Times Magazine carried not one but two interesting articles yesterday. The first, by Charles Duhigg, about how companies learn from and exploit your shopping habits, is here. It's pretty interesting, because whether or not you are curious about the numbers, everyone, here in the US at least, has to shop.

The second article is Nate Silver's "Why Obama Will Embrace the 99 Percent," part of a series he's working on for the Magazine in conjunction with his blog. It's a fascinating comparison of the comparative electability, as they look now, of Republican Presidential hopefuls Mitt Romney and Rick Santorum. It's clearly written, with good graphics, including an interactive chart looking at chances of winning the popular vote in varying economic (and presidential popularity) circumstances, and well worth reading.

It's an election year! Time to start obsessing about polling

Let's start with Presidential approval ratings.

According to a CBS News poll released yesterday, President Obama's are up, reaching 50% for the first time since May, 2011:

CNN's polls say the same thing. So do Rasmussen Reports, which says that 49% approve of the President, and that 49% disapprove. They display the data differently:

What's going on here? I suspect it's in the way the questions are asked and the data tabulated. If CBS and CNN subtracted the lukewarm approvals from the strong approvals, you might get the 27% strong approvals that Rasmussen reports. But it appears that right now there are more people expressing strong and lukewarm approval than there are expressing disapproval. The takeaway? If you can, look at more than source for information. And think about how each organization is reporting its data. And right now, in a head-to-head competition, even Rasmussen reports that Obama beats Santorum and Romney.

Which is what Nate Silver, of FiveThirtyEight (now a part of the New York Times) reports as well. I should say that's what Silver concludes, as he is using an early stage model taking account of the economy, each candidate's ideology, and the approval ratings. He also has some projections about upcoming Republican primaries (in Arizona, Michigan, Georgia, and Ohio). FiveThirtyEight is a great site, and I'll be keeping an eye on it in the coming months. Even though the election is still nine months away.