Lecture 21—Friday, February 17, 2006

What was covered?

• The role of b(θ) in the generic density formula for a member of the exponential family of distributions
• The normal distribution is a member of the exponential family
• The binomial distribution is a member of the exponential family
• Canonical link functions for probability distributions commonly used in generalized linear models

Terminology Defined

• variance function b′′(θ)
• logit function

The role of b(θ) in the generic exponential family density formula

• Last time, we defined the generic probability density function that defines the exponential family of probability distributions.

It turns out the function b(θ) in this expression plays an interesting role. I examine this role in what follows.

• Because this generic formula given above is a density, it integrates to 1.

where I'm assuming that by definition f(y) = 0 outside of its ordinary domain so that the above integral makes sense.

• Now suppose we differentiate both sides of this equation with respect to θ.

• Subject to what are called regularity conditions it is legitimate to interchange a derivative and an integral. Thus we have

• The chain rule tells us that . Thus we have

• But and . Thus we have

• So we see that the derivative of b(θ) is the mean of the distribution. Suppose we differentiate the integral a second time.

• This derivative in the last expression is the derivative of a product. Recall the product rule of differentiation.

• Thus we have

where in the last step I use the result discovered above that . Recall that . Thus we have

• Hence the second derivative of b(theta) is related to the variance of the distribution. is called the variance function and is usually denoted as . The notation makes sense because is a function of μ hence , being the derivative of , also is a function of μ.

Checking these results on the Poisson distribution

• Last time we demonstrated that the Poisson distribution is a member of the exponential family. We found for the Poisson that

from which we see that where θ is the so-called canonical parameter. From the equation for the canonical parameter we see that and therefore . Thus for the Poisson distribution we find

which matches the usual result. For the Poisson distribution the parameter λ is both the mean and the variance of the distribution.

The normal distribution is a member of the exponential family

• The probability density function for the normal distribution is

• I proceed to rewrite the density so that it matches the required pattern for the exponential family.

• From this last expression we can identify the following.

Thus for the normal distribution the canonical parameter θ is just the mean μ. Thus the canonical link function is just the identity function, .

• Checking the mean and variance I find the following.

The binomial distribution is a member of the exponential family

• The probability mass function for the binomial distribution is

• Taking the exponential of the log of this expression yields the following.

from which we can identify the following.

• So the canonical parameter for the binomial distribution is , a function known as the logit. By definition,

The logit function is also the canonical link function for the binomial. This is not immediately obvious because it doesn't appear that logit(p) is a function of μ, g(μ), as it should be. (Recall that the mean of the binomial distribution is μ = np.) But in fact it is as I now demonstrate.

• The logit turns out to be an excellent choice as a link function. Recall that one of the problems with using the identity link for proportions is that while proportions are required to lie in the interval [0, 1], using an identity link can lead to predictions outside this interval. Let's see what restrictions the logit function places on p. (Recall that the link function is the link between the random and the systematic component of a generalized linear model.)

Fig. 1  Inverse of the logit link

• Observe the following.
  1. Since η is the linear predictor there are no restrictionson its values and it in principle can range from –∞ to ∞.
  2. The exponential function is always positive. Therefore the expression is also always positive.
  3. Because is always positive it follows that . Hence .
  4. approaches 0 as and approaches 1 as .

  The graph of is shown to the right. Observe that it maps the linear predictor η nonlinearly into the interval (0, 1).

• An additional attractive feature of the logit link is that the logit is an interpretable quantity. We'll talk more about intepreting logits when we discuss regression problems with a binomial distribution as the random component.
• Finally I check the mean and variance results for the binomial. First I write b(θ) as a function of the canonical parameter .
• Above we found that if , then and hence

.

Therefore

• From this I can obtain the mean and variance

Members of the exponential family and their canonical links

• The table below lists the probability distributions typically used as the random component in generalized linear models along with their canonical link functions.

Probability Distribution	Canonical Link g(μ)
Poisson
Normal
Binomial
Gamma
Inverse Gaussian

• Conspicuous by its absence from this table is the negative binomial distribution.
  • The two-parameter negative binomial distribution is not a member of the exponential family. But if we treat the dispersion parameter as a known, fixed constant, then it is a member.
  • In order to fit a negative binomial regression model as a GLIM, an initial guess is made for the value of the dispersion parameter. Using this guess the parameters of the linear predictor, η, are estimated within the generalized linear model framework. Using the parameter estimates obtained for η, a new estimate is obtained for the dispersion parameter. This process is repeated until the estimated values of both sets of parameters no longer change.
  • The canonical link for the negative binomial distribution is rather complicated and hard to interpret, so it is rarely used. Instead to facilitate comparisons with the Poisson generalized linear model, a log link is typically used.

Course Home Page