Lecture 20 —Wednesday, February 15, 2006

What was covered?

• Introduction to generalized linear models
• Assumptions of general linear models
• Assumptions of generalized linear models
• The exponential family of probability distributions
• The Poisson distribution is a member of the exponential family

Terminology Defined

• general linear model (GLM)
• generalized linear model (GLIM)
• linear predictor (η)
• link function
• exponential family
• canonical parameter
• canonical link function

General linear models (GLMs)

• The general linear model is essentially just the ordinary linear regression model but with an expanded class of predictors. As is usual in regression models the basic starting point is that we have a random sample of observations of some random variable Y.
• In addition, associated with each observation we have measured a number of additional variables, x₁, x₂, … , x_p, the predictors, that we suspect might have some relationship to Y. These are not considered random and are assumed to be known without error. In fact, these variables may be fixed by design, as in an experiment where we control the levels of the treatment variables. In this case experimental units are randomly assigned to treatment levels thus gaining validity and ensuring that the set of responses can still be treated as a random sample.
• The "general" in GLM refers to the fact that the predictors in the model include both continuous variables and categorical variables. Thus the general linear model includes under its rubric those classical statistical models known as regression, analysis of variance, and analysis of covariance.
• The "linear" in GLM refers to the way in which the predictors enter the regression equation. Now while the predictors can be anything, quadratic terms, cubic terms, sines, cosines, etc., their effect on the response is linear. The predicted response is assumed to be obtained from a linear combination of the predictors, i.e., a sum of the form , where the βs are parameters to be estimated.
• In order to be able to do more than simply estimate the parameters in the model, it is usually assumed that response variable is normally distributed. This distributional assumption is the basis of the statistical tests that are used to assess the "quality" of the regression.
• The predictors are used to describe the mean of this normal distribution. Thus we assume that the mean of the normal distribution varies as a function of the values the predictors take. Although the mean of the normal distribution may vary its variance is required to remain constant. Thus the general linear model assumes we have a family of normal distributions all with the same variance but whose mean varies as determined by the regression equation.
• In order to contrast the general linear model with the generalized linear model that I will describe next, it is useful to reformulate the assumptions outlined above this time using the language of generalized linear models.

The assumptions of GLMS using the language of GLIMs

1. Random component: The response variables Y₁, Y₂, … , Y_n are assumed to be independent and normally distributed. More formally we assume . Observe that the mean is allowed to vary for each Y_i but that the variance does not.
2. Systematic component: The p covariates are combined to form a linear predictor. The linear predictor is represented by the Greek letter η (eta).
3. Link function: The systematic component and random component are linked together via a link function g. For GLMs the function g is taken to be the identity link. In other words we assume μ = η or as is more typically written

The potential limitations of GLMs

1. Response variables of interest in ecology are often not normally distributed nor do they tend to exhibit constant variance. While transformations can sometimes "fix" one or both of these problems, transformations introduce their own baggage and make the resulting model difficult to interpret. There's also no guarantee that a suitable transformation will be found.
2. Response variables may have a restricted range. Thus count data are required to be non-negative and proportions are required to be between zero and one. The normal distribution does not have such a range restriction and hence may not be an appropriate probability model for such data.
3. It is often observed in ecological data that the variance is a function of the mean. This is especially true of restricted range data for values near the boundary of the range. Such values will necessarily have a skewed distribution because their distribution is truncated by the boundary. This in turn limits the amount of variability they can display, at least in one direction.
4. The identity link can be problematic. Since the predictors do not have the same range restrictions as does the response variable, the systematic component may yield predicted values outside the legal range of the data.

Generalized linear models (GLIMs)

• Generalized linear models are designed to overcome the limitations described above. Proposed in the 1970s by McCullagh and Nelder, GLIMs serve to unify a large number of disparate modeling protocols under a single umbrella. While some revisionist statisticians think the significance of the GLIM paradigm is overstated (Lindsey 2004), I think it is fair to say that generalized linear models did for statistical modeling what plate tectonics did for geology. GLIMs provided a unifying vision to statistical models and brought a degree of order to what was otherwise a set of rather chaotic and seemingly arbitrary collection of procedures.
• To understand how GLIMs differ from GLMs I restate the three assumptions listed above but this time for GLIMs.
  1. Random component: The response variables Y₁, Y₂, … , Y_n are assumed to be independent and drawn from a member of the exponential family of distributions. The exponential family includes many of the standard probability distributions including the Poisson, normal, and binomial distributions. Thus the normality and constant variance assumptions are relaxed for GLIMs.
  2. Systematic component: The p covariates are combined to form a linear predictor. The linear predictor is typically denoted by the Greek letter η (eta). . This assumption is identical for both GLMs and GLIMs.
  3. Link function: The systematic component and random component are linked together via a link function g. For GLIMs the link function g can be selected from a large class of functions, although there are certain natural choices called canonical links. The only restrictions on g are that it be differentiable and monotonic. Thus for a GLIM we write

Because we require g to be monotonic, it follows that it is invertible. Thus we can also write this last expression in terms of the inverse link function, .

Written this way we can see that GLIMs have the potential of solving the range restriction problem. Since the linear predictor is connected to the mean through , a judicious choice of link function can constrain the predictions to map onto a desired range.

Exponential family of distributions

• Crucial to the definition of the generalized linear model is the concept of the exponential family of probability distributions. Formally, a probability distribution is a member of the exponential family if its probability density (mass) function can be written in the following form.

• In this expression θ is called the canonical parameter. It's also referred to as the location parameter of the distribution.
• The parameter φ is referred to as the dispersion parameter. It's also called a scale parameter.
• a, b, and c are functions whose form will vary between different members of the exponential family. There is no constraint on what form these functions can take except for which arguments they are allowed to take.
  • Observe that b is only a function of θ. In particular it is not a function of y.
  • a is a function of φ only.
  • c is a function of y and φ only. It is not a function of θ.
• Observe also that the variable y occurs multiplied by the canonical parameter θ.

• The fact that a large number of probability distributions can be written in this generic form means that an efficient way to obtain maximum likelihood estimates is to write a single optimization routine based on this generic form. Thus a single protocol can be used to find maximum likelihood estimates for regression parameters in models employing a diverse assortment of random components.

The Poisson distribution is a member of the exponential family

• I next demonstrate that the Poisson distribution can be written in the generic form of the exponential family. The probability mass function for the Poisson distribution is shown below.

• Getting this expression into the proper form for the exponential family is just an exercise in algebra. The first step in doing this is always the same. We use the fact that exp(x) and log(x) are inverse functions and hence .

• Using this last expression we can match up the generic functions and parameters in the formula for the exponential family with the specific values from the Poisson probability mass function. Thus we see
  • 
  • 
  • 
  • 
• Since θ is the canonical parameter for the exponential family we should call log λ the canonical parameter for the Poisson distribution.
• Recall that λ is the mean of the Poisson distribution. Thus the canonical parameter for the Poisson distribution can also be written from which it follows that .
• This suggests that a sensible choice for the link function is because then we would have, using the link function to link the systematic and random components, the following.

• From the last equation we see that although η can take on any value from –∞ to ∞, μ is constrained to be positive. Thus the constraint that the mean of the Poisson distribution has to be positive is satisfied if we use the logarithm as the link function. Because this choice of link function arose directly from the canonical parameter, it is called the canonical link for the Poisson distribution.
• Note: the canonical link is not the only link function one might choose, but it is the "natural" one. Using the canonical parameter as the canonical link also turns out to have some advantages in the numerical estimation of the parameters in the model.

Cited Reference

• Lindsey, J. K. 2004. Introduction to Applied Statistics: A Modeling Approach. Oxford University Press: Oxford.
• McCullagh, P. and J. A. Nelder. 1989. Generalized Linear Models, 2nd edition. Chapman & Hall: London.

Course Home Page