Lecture 17—Friday, February 10, 2006

What was covered?

• The Burnham & Anderson (2002) protocol for using AIC to select models

Terminology Defined

• AIC differences, Δ_i
• relative likelihood
• Akaike weight

A Protocol for Using AIC in Model Selection

• In what follows I assume that a set of biologically realistic modes has been assembled. We wish to use AIC to screen out those models that are not supported by the data.
• Before using AIC in model selection there are a number of things to keep in mind.

1. All the models in the set of candidates must use exactly the same set of observations and therefore be based on the same sample size n.
2. All the models must use exactly the same response (all y, or all log y, etc.)
3. If some of the models in the set assume different probability generating mechanisms, then it is crucial that AIC is calculated using the full likelihood. If you are doing the likelihood calculations yourself there is no problem, but if you are using a statistical package then you'll need to determine what it's doing. If terms have been dropped from the likelihood, your AIC comparisons will not be valid. For additional discussion of this point see lecture 10.

The Algorithm

Step 1

Calculate AIC for all models. If is small, use AIC_c instead of AIC. Note: it doesn't hurt to always use AIC_c.

Step 2

Identify the model with the smallest AIC. Denote its AIC as AIC_min. This is the best model. We could stop at this point but there is more information to extract from the models.

Step 3

Calculate the AIC differences, Δ_i, for each model.

Step 4

Compute the relative likelihood, , for each model.

Step 5

Compute Akaike weights for each model. These are just the normalized relative likelihoods.

• The Akaike weights can be interpreted as probabilities—the probability that the given model is the best model. If we were to go back and obtain more data from the population and refit the same models again, then the Akaike weight gives the probability that a given model would be judged the best model on repeated sampling.

A numerical example

• In the example below, five different nonlinear models were fit to a data set. The sample size is n = 47. Because < 40 for at least one of our models (actually for all of our models), we follow Burnham & Anderson's recommendation and use AIC_c to compare the models. I report AIC only for completeness.

• Because Model 2 has the smallest AIC_c we would judge it to be the best model. Notice that it does have one competitor. Based on the weights we would say that there is some evidence for model 1. On the other hand there is essentially no support for models 3, 4, and 5.
• Using the cross-validation property of AIC we would expect under repeated sampling that Model 2 would come out best 87% of the time but that 13% of the time Model 1 would be ranked best.
• Observe that for these data we would have reached the same conclusions if we had used AIC instead of AIC_c.
• In practice it is probably unnecessary to construct this full table (although Burnham & Anderson recommend it). The point of the Akaike weights is to place the Δ_i on a more interpretable scale. Essentially what you'll find can be summarized in the following table using only Δ_i (Burnham & Anderson 2002, p. 70).

Cited Reference

• Burnham, K. P. and D. R. Anderson. 2002. Model Selection and Multimodel Inference. Springer-Verlag, New York.

Course Home Page