Bayes's Theorem is not optional

Abstract: I present a probability puzzle, the Rain in Seattle Problem, and use it to explain differences between the Bayesian and frequentist interpretations of probability, and between Bayesian and frequentist statistical methods.  Since I am trying to clear up confusion, I try to describe the alternatives without commenting on their pros and cons.

Introduction

Conversations about Bayesian statistics sometimes get bogged down in confusion about two separate questions:

1) The Bayesian interpretation of probability, as opposed to the frequentist interpretation.  

2) The Bayesian approach to statistical inference, as opposed to frequentist approach.

The first is a philosophical position about what probability means; the second is more like a practical recommendation about how to make inferences from data.  They are almost entirely separate questions; for example, you might prefer the Bayesian interpretation of probability by philosophical criteria, and then use frequentist statistics because of practical requirements; or the other way around.

Under the frequentist interpretation of probability, we can only talk about the probability of an event if we can model it as a subset of a sample space.  For example, we can talk about the probability of drawing a straight in poker because a straight is a well-defined subset of the sample space that contains all poker hands.  But by this interpretation, we could not assign a probability to the proposition that Hillary Clinton will win the election, unless we could model this event as a subset of all elections, somehow.

Under the Bayesian interpretation, a probability represents a degree of belief, so it is permissible to assign probabilities to events even if they are unique.  It is also permissible to use probability to represent uncertainty about non-random events.  For example, if you are uncertain about whether there is life on Mars, you could assign a probability to that proposition under the Bayesian interpretation.  Under the frequentist interpretation, there either is life on Mars or not; it is not a random event, so we can't assign a probability to it.

(I avoid saying things like "a Bayesian believes this" or "a Frequentist believes that".  These are philosophical positions, and we can discuss their consequences regardless of who believes what.)

In problems where the frequentist interpretation of probability applies, the Bayesian and frequentist interpretations yield the same answers.  The difference is that for some problems we get an answer under Bayesianism and no answer under frequentism.

Now, before I get into Bayesian and frequentist inference, let's look at an example.

The Rain in Seattle problem

Suppose you are interviewing for a data science job and you are asked this question (from glassdoor.com):

You're about to get on a plane to Seattle. You want to know if you should bring an umbrella. You call 3 random friends of yours who live there and ask each independently if it's raining. Each of your friends has a 2/3 chance of telling you the truth and a 1/3 chance of messing with you by lying. All 3 friends tell you that "Yes" it is raining. What is the probability that it's actually raining in Seattle?

Take a minute to think about it before you go on.  Then take a look at the responses on glassdoor.com. The top response, which uses Bayes's Theorem, is correct.  I'll explain the correct solution first; then I want to comment on some of the other responses.

The question asks you to compute the probability of rain conditioned on three yesses, which I'll write P(rain|YYY).

Now, here's an important point: you can't give a meaningful answer to this question unless you know P(rain), the probability of rain unconditioned on what your friends say.   To see why, consider two extreme cases:

1. If P(rain) is 1, it always rains in Seattle.  If your friends all tell you it's raining, you know that they are telling the truth, and that P(rain|YYY) is 1.

2. If P(rain) is 0, it never rains in Seattle, so you know your friends are lying and P(rain|YYY) = 0.

For values of P(rain) between 0 and 1, the answer could be any value between 0 and 1.  So if you see any response to this question that does not take into account P(rain), you can be sure that it is wrong (or coincidentally right based on an invalid argument). 

But if we are given the base rate, we can solve the problem easily using Bayes's Rule,   According to the Western Regional Climate Center, from 1965-99 there was measurable rain in Seattle during 822 hours per year, which is about 10% of the time.

A base rate of 10% corresponds to prior odds of 1:9.  Each friend is twice as likely to tell the truth as to lie, so each friend contributes evidence in favor of rain with a likelihood ratio, or Bayes factor, of 2.  Multiplying the prior odds by the likelihood ratios yields posterior odds 8:9, which corresponds to probability 8/17, or 0.47.

And that is the unique correct answer to the question (provided that you accept the modeling assumptions).  More generally, if P(rain) = p, the conditional probability P(rain|YYY) is

Probability(8 Odds(p))

assuming that Odds() converts probabilities to odds and Probability() does the opposite.

What about the frequentist answer?

Several of the responses on glassdoor.com provide what they call a frequentist or non-Bayes perspective:

Answer from a frequentist perspective:  Suppose there was one person. P(Y|rain) is twice (2/3 / 1/3) as likely as P(Y|no rain), so the P(rain) is 2/3.  If instead n people all say YES, then they are either all telling the truth, or all lying. The outcome that they are all telling the truth is (2/3)^n / (1/3)^n = 2^n as likely as the outcome that they are not. Thus P(YYY | rain) = 2^n / (2^n + 1) = 8/9 for n=3.  Notice that this corresponds exactly to the Bayesian answer when prior(raining) = 1/2.

And here's another:

I thought about this a little differently from a non-Bayes perspective.  It's raining if any ONE of the friends is telling the truth, because if they are telling the truth then it is raining. If all of them are lying, then it isn't raining because they told you that it was raining.  So what you want is the probability that any one person is telling the truth.  Which is simply 1-Pr(all lie) = 26/27.  Anyone let me know if I'm wrong here!

These are not actually frequentist responses.  For this problem, we get the same answer under Bayesianism and frequentism because:

1) Everything in this problem can be well-modeled by random processes.  There is a well-defined long-run probability of rain in Seattle, and we can model the friends' responses as independent random variables (at least according to the statement of the problem).

AND

2) There is nothing especially Bayesian about Bayes's Theorem!  Bayes's Theorem is an uncontroversial law of probability that is true under any interpretation of probability, and can be used for any kind of statistical inference.

The "non-Bayes" responses are not actually other perspectives; they are just incorrect.  Under frequentism, we would either accept the solution based on Bayes's Theorem or, under a strict interpretation, we might say that it is either raining in Seattle or not, and refuse to assign a probability.

But what about frequentist inference?

Statistical inference is the process of inferring the properties of a population based on a sample.  For example, if you want to know the fraction of U.S. voters who intend to vote for Donald Trump, you could poll a sample of the population.  Then,

1) Using frequentist inference, you could compute an estimate of the fraction of the population that intends to vote for Trump (call it x), you could compute a confidence interval for the estimate, and you could compute a p-value based on a null-hypothesis like "x is 50%".  But if anyone asked "what's the probability that x is greater than 50%", you would not be able to answer that question.

2) Using Bayesian inference, you would start with some prior belief about x, use the polling data to update your belief, and produce a posterior distribution for x, which represents all possible values and their probabilities.  You could use the posterior distribution to compute estimates and intervals similar to the results of frequentist inference.  But if someone asked "what's the probability that x is greater than 50%", you could compute the answer easily.

So, how does this apply to the Rain in Seattle Problem?  It doesn't, because the Rain in Seattle problem has nothing to do with statistical inference.  It is a question about probability, not statistics. It has one correct answer under any interpretation of probability, regardless of your preferences for statistical inference.

Summary

1) Conversations about Bayesian methods will be improved if we distinguish two almost unrelated questions: the meaning of probability and the choice of inferential methods.  

2) You don't have to be a Bayesian to use Bayes's Theorem.  Most probability problems, including the Rain in Seattle problem, have a single solution considered correct under any interpretation of probability and statistics.


UPDATE October 13, 2015:  A few people asked me how to solve this problem using Bayes's Theorem (based on probabilities) rather than Bayes's Rule (based on odds).  To apply Bayes's Theorem, you might find it helpful to use my "Bayesian update worksheet".  Here's one that shows how to solve this problem: