Lecture 37—Monday, April 16, 2007

What was covered?

• The Bayesian alternative to statistical analysis
• Contrasting the Bayesian and frequentist perspectives

The Bayesian and frequentist approaches to statistical inference

• Bayesian inference is as old as statistics. The frequentist approach is a more recent innovation and derives from the uncomfortable marriage of methodologies developed by the Neyman-Pearson-Wald school (hypothesis testing) and the Fisherian school (everything else). While there are fundamental differences in the Bayesian and frequentist approaches that are difficult to reconcile, most statisticians treat them as just two options on a menu and are willing to use the tools they offer without necessarily embracing the accompanying baggage.
• While there has been considerable rancor in the literature (particularly among scientists) with regard to these two approaches, I think it is fair to say that Bayesian inference has always been recognized as being more legitimate than the frequentist approach. Because Bayesian inference was difficult to carry out except in a few fairly artificial cases, the frequentist approach has held sway. Except for a few universities that have built there reputation on being Bayesian hotbeds, most statistics departments tended to include only token representatives of the Bayesian viewpoint among their faculty.
• This has dramatically changed. Making use of methodologies originally developed by physicists, Bayesian inference is now easy to do. Furthermore Bayesian methods are able to solve problems that are currently impossible for the frequentist to solve. This has generated a great interest in Bayesian statistics among scientists in general.
• I introduce Bayesian statistics here not as an alternative to frequentist statistics but as a collection of tools that can be useful for specific problems. First I outline the differences between the two approaches to understand what the fuss is all about.

A critique of the frequentist perspective

• For the frequentist everything hinges on the notion of repeated sampling.
  • In the informal definition of probability the probability of an event represents the long range relative frequency of occurrence of that event. To say that the probability of heads is one half when a fair coin is tossed once means that if we were to flip a fair coin repeatedly the long run relative frequency of heads is one half. (Note: another way to interpret probability in this example is to treat the possible outcomes of a coin flip as equally likely.)
  • Statistical inference derives from the sampling distribution of a statistic. The sampling distribution is the set of estimates obtained when all possible samples are selected from a target population. Properties of a statistic (unbiasedness), standard errors, etc. derive from the sampling distribution of that statistic.
• Implicit in the frequentist approach to estimation is that there is a fixed quantity in nature, the parameter θ, that we wish to measure. So, for example, the bias of an estimator is the difference between mean of the sampling distribution and the the true value that is being estimated.
• The notion that the parameter is a fixed quantity in nature causes problems in interpretation for the frequentist.
  • When we construct a confidence interval for a parameter we like to treat the interval as a probability statement for the parameter—a set of likely values. But in truth if the parameter is fixed then it is either in the interval we've constructed or it's not. There's no probability associated with it.
  • The probability instead derives from the sampling distribution. We call it a 95% confidence interval because we're guaranteed that 95% of the intervals we might have constructed if we had obtained all possible samples from the population do in fact contain the true parameter value. We hope this is one of the lucky ones.
  • It's been said that constructing a 95% confidence interval is analogous to running from an elephant while shooting an elephant gun over your shoulder. Did you hit it? You don't know but you know that 95% of the time an elephant gun is known to stop an elephant. You just hope this is one of those times.
  • The bottom line is that when θ is treated as real and fixed, then it's only our methods that can have probability associated with them.
• The frequentist perspective uses p-values as measures of evidence against hypotheses, but the p-value is a measure of how likely are the observed data under the null hypothesis, i.e., . This doesn't address the probability of the null hypothesis, for that we need . So the p-value addresses the wrong probability.
• A Bayesian has other problems with p-values beyond that they estimate the wrong probability. By definition a p-value is the probability under the null hypothesis of obtaining data as extreme or more extreme than the data one actually obtained. A Bayesian would wonder why we are basing our decision about the null hypothesis using data we didn't obtain. Shouldn't we base our decisions on the evidence at hand and not on the evidence we might have obtained?

The Bayesian alternative

• In the Bayesian approach, probability is a matter of opinion. It still must follow the standard rules of probability (the Kolmogorov axioms, for instance), but other than that things are pretty flexible. For this reason Bayesians are said to take a subjective approach to probability (as opposed to the "objective" approach of the frequentists).
• As far as the existence of a true value of θ in nature, Bayesians are of an open mind. θ may or may not be real, but in the Bayesian perspective it doesn't matter. All we know about θ is what we believe about it. As knowledge accumulates our beliefs about θ become more focused. Since the value of θ is a matter of belief, and probability is a matter of belief, for all intents and purposes θ can be viewed as a random quantity. As a result to a Bayesian parameters are random variables, not fixed constants.
• Consequently confidence intervals pose no philosophical dilemma for a Bayesian. Since parameters are random we can make probability statements about their values. Thus a confidence interval for θ is a probability statement about the likely value of θ. To avoid confusion with frequentist confidence intervals, Bayesians often call their intervals "credibility intervals".
• In the context of a statistical experiment there are only two opinions about θ that matter.
  1. Our opinion about θ before we carry out the experiment (i.e., before we collect data). This is called the prior probability and is denoted .
  2. Our opinion about θ after we carry out the experiment (collect data). This is called the posterior probability and is denoted .

The algebra of Bayesian inference

• In the logic of Bayesian inference, data modifies our opinion about nature. This is stated formally in the probability rule known as Bayes rule.
• Let A and B be two events. Using the definition of conditional probability we can write the following.

• Having solved for the joint probability in each expression we can set the results equal to each other.

• This latter formula is called Bayes rule. Now if we imagine a set of events that are pairwise disjoint and partition the sample space S such that , then we can write P(A) as follows.

• So Bayes rule becomes

or for B continuous

• Now let A = θ and B = data, then Bayes rule can be written as follows.

• Observe that is our old friend the likelihood. Having already identified the posterior and prior probabilities in this expression we can express Bayes rule as

where the constant of proportionality is give by the reciprocal of the integral shown above. Thus with Bayes rule we update our prior probability of θ via the likelihood to obtain the posterior probability of θ.

• A Bayesian takes pride in the fact that Bayes rule as formulated above essentially encapsulates inductive science in a single formula. In science we develop theories about nature. Observation and experiment then cause us to modify those theories about nature. Further data collection causes further modification. Thus science is a constant dynamic between prior and posterior probabilities, with prior probabilities becoming modified into posterior probabilities through experimentation at which point they become the prior probabilities of future experiments. Thus the Bayesian perspective accounts for the cumulative nature of science.

The frequentist rejoinder

• The frequentist retort is that this is a mischaracterization of science. Science is inherently objective and has no use for subjective quantities such as prior probabilities. Science should be based on data alone and not the prejudices of the researcher.
  • The Bayesian rejoinder is first that science does have a subjective component. The "opinions" of scientists dictate the kinds of research questions they pursue.
  • In any case if there is concern that a prior probability unfairly skews the results, the analysis can be rerun with other priors. In fact in Bayesian analysis it is fairly typical to carry out analyses with a range of priors to demonstrate that results are robust to the choice of prior.
• In fact both schools of thought are correct here. While Bayesians may have described the ideal scientific method, in truth consensus in science is informal at best. Perhaps there should be a current prior probability in vogue for everything in science, but typically there isn't. Without this consensus the inherent subjectivity of priors does seem to be a problem.
• The bottom line rejoinder offered by a pragmatist is that the entire argument is moot anyway, because very few interesting problems can be solved using Bayes rule. The reason for tht lies in the integral that appears in the denominator. For most higher dimensional scenarios evaluating this integral is an intractable problem.
• Historically the only problems that Bayesians were able to solve were trivial ones where the integral can be ignored because the form of the posterior probability can be obtained by inspection. These situations involve what are called conjugate priors, likelihood-prior choices that combine in a nice fashion so that the posterior probability can be determined without integration.
  • For instance, a normal prior and a normal likelihood yield a normal posterior of a known form.
  • A binomial likelihood coupled with a beta prior yield a beta posterior of a known form.
• There are a number of other examples such as this, but the list is not very long.

Bayesian statistics today

• The situation changed dramatically 20 years ago. Using methodology, Gibbs sampling and the Metropolis-Hastings algorithm, developed in physics much earlier it became possible to sample from the posterior distribution, for a given prior and likelihood, without the need to construct an explicit formula. These methods are now placed under the rubric Markov chain Monte Carlo. The mantra for this methodology is that anything you care to say about the posterior distribution you can obtain by sampling from it. A formula is unnecessary. Markov chain Monte Carlo allows us to characterize posterior probabilities by sampling from a set of values that are generated using only the prior and the likelihood.
• Suppose you had a way of sampling from the posterior distribution, . Let be a random sample from the posterior distribution. Then any quantity of interest can be estimated from this sample.
  1. Mean:
  2. Variance:
  3. Density: use the histogram of the sample
  4. p^th quantile is the solution of where I is the indicator function.
• We've already seen a parallel for this approach to statistical analysis in our use of the bootstrap distribution as a surrogate for the sampling distribution of a statistic.

Course Home Page