Lecture 34—Monday, March 20, 2006

What was covered?

• Basics of logistic regression
• Interpreting parameters in terms of odds ratios
• Goodness of fit for logistic regression

Terminology defined

• contingency table
• grouped binomial data
• odds
• odds ratio
• prospective study
• retrospective study
• ungrouped binomial data

Logistic regression as a generalized linear model

• Logistic regression is the preferred method of an analylsis for situations in which the response has a Bernoulli or a binomial distribution, i.e., or . In logistic regression the situation when the Bernoulli distribution applies is referred to as ungrouped (or sparse) binomial data, while the binomial scenario is called grouped binomial data.
• Logistic regression is an example of a generalized linear model in which

1. The random component is the binomial (Bernoulli) distribution,
2. the linear predictor is , and
3. the link function is , also called the logit.

• Recall that other suitable link functions for binomial data are the probit, , and the complementary log-log link, . In Lecture 22 we showed that the logit, probit, and cloglog link are invertible and that their inverses map the linear predictor onto the interval [0, 1], an important requirement since p is a probability.

Interpretation of regression coefficients

Identity link

• When is the identity function, as is typically the case in ordinary linear regression, then

.

• If we increase x₁ by one unit, then we have

• So β₁ is interpreted as the amount that the mean increases when x₁ is increased by one unit.

Log link

• When is the log function, as in Poisson or negative binomial regression, then

• If we increase x₁ by one unit, then we have

• Thus on the original scale of the mean we have

• So we can interpret β₁ in terms of , which is the amount the mean is multiplied by when x₁ is increased by one unit.

Logit link

• The arguments used above for the identity and log links do not work for the logit link. The regression coefficients do not have simple interpretations in terms of p, the probability of success. On the other hand the regression coefficients in logistic regression do have a simple interpretation in terms of odds.
• Probability theory has its origin in the study of gambling. Gamblers typically speak in terms of the odds for or odds against events occurring, rather than in terms of probabilities, but the two notions are equivalent. Any statement made using odds can be reformulated in terms of probability.
• As an example, suppose it is stated that the odds in favor of an event, Y = 1, occurring are 3 to 1. This can be interpreted as follows.

Thus if the odds for an event are 3:1, then the event has a probability of occurring.

• Notice that it is odds, rather than probability, that is the argument of the logarithm function in the logit link. Thus the logit function is also called the log odds function. When the logit is viewed as a log odds, this immediately provides us with an interpretation for the regression parameter β₁ in the logistic regression model. Using the log odds notation we have the following.

• Solving for β₁ yields the following.

where the notation OR denotes the odds ratio, the ratio of two odds. In this case the odds ratio measures the effect of increasing x₁ by 1 on the odds that Y = 1.

• Notice that we can determine the direction of the effect of x₁ by looking at the sign of its regression coefficient.
  • If β₁ > 0 then > 1. Thus the odds of Y = 1 occurring go up when x₁ increases.
  • If β₁ < 0 then < 1. Thus the odds of Y =1 occurring go down when x₁ increases.
• Just as is the case with ordinary regression, Poisson regression, and negative binomial regression, we can use the sign of the regression coefficient in logistic regression to infer the effect a predictor has on the response.

The odds ratio in epidemiology

• The dominance of the logistic regression model for analyzing binomial data is due largely to its widespread acceptance in epidemiology, medicine, and public health. In these disciplines the objective is often to assess the effect of various risk behaviors on the odds of occurrence of a disease of interest.
• Consider the following example in which the objective is to determine if bottle-fed babies are at a higher risk for infant respiratory disease than are breast-fed infants. Suppose 458 bottle-fed babies and 494 breast-fed babies were followed over time to see if they contract a respiratory disease. Because we select the subjects at the onset of the study and then follow them over time, this is referred to as a prospective study.
• The data from such a study can be conveniently organized in what's called a contingency table. In the contingency table below we use the rows of the table to classify infants according to their risk behavior and the columns of the table to classify infants by their disease status.

• In a prospective study we can calculate the odds of disease for each risk group separately. Thus we have the following.
  • The odds of disease for bottle-fed babies is .
  • The odds of disease for breast-fed babies is .
  • Thus the odds ratio for disease, bottled-fed versus breast-fed, is the following.

• Now suppose we obtained the same data but this time it was from a retrospective study. In a retrospective study we obtain two groups of individuals, one with disease and one disease-free, and then retrospectively determine what their risk behaviors were. Thus in a retrospective study all we can calculate are the odds of the risk behavior, say being bottle-fed, given the disease status. Thus we obtain the following.
  • The odds of being bottle-fed for children with respiratory disease is .
  • The odds of being bottle-fed for children who are disease-free is .
  • Thus the odds ratio of being bottle-fed, diseased children versus disease-free children, is the following.

• Observe that even though the odds being calculated in the two studies are totally different, we obtain the same odds ratio from both. This is great news because retrospective studies are much easier to conduct than are prospective studies and although it would appear that the quantities that one can estimate from a retrospective study are less interesting, the odds ratio for risk status is identical to the odds ratio for disease. So we can estimate an interesting odds ratio by easily calculating an uninteresting one.
• Because the parameters of a logistic regression model estimate odds ratios and odds ratios are the natural language of epidemiology (due to their equivalence in prospective and retrospective studies), logistic regression has become the method of choice for analyzing binary data. As a result it is available in most statistical packages.
• For ecologists the epidemiological rationale for doing logistic regression is probably uninteresting. Ecologists use logistic regression because that's the tool that's available.

Assessing the fit of logistic regression models—Grouped data

• In discussing goodness of fit strategies it is necessary to distinguish the grouped binary data case from the ungrouped binary data case, for reasons we'll discuss next time.
• Recall the goodness of fit strategies we've used for other kinds of regression models
  • For normal-based models we examine the model residuals
  • For Poisson regression we use the deviance and compare it to a chi-squared distribution
  • For negative binomial regression we compare expected frequencies with the observed frequencies using a Pearson chi-squared test or a likelihood ratio G² test.
• For grouped binary data, we can construct a goodness of fit test that is very much like the one used for negative binomial regression. Let's begin with the following definitions.
  • Let i denote the group.
  • Let n_i denote the sample size in group i.
  • Let be the number of successes in group i. Here is a binary variable that is 0 or 1 depending on whether the j^th observation in group i is a failure or a success.
  • Based on the covariate pattern for group i (which is the same for all members of the group) we obtain the estimate for the probability of success. Thus for group i the expected number of success is and the expected number of failures is .
• Suppose there are g groups. Then the observed frequencies can be arranged in the following 2 × g table.

• The expected frequencies are as follows.

Expected	Group 1	…	Group g
Success		…
Failure		…

• From these we can form the usual Pearson chi-squared statistic, , or the likelihood ratio G² statistic, . The sum in each case is taken over both the successes and failures for a total of 2g terms.
• Observe that in each column the total is fixed—it's equal to n_i . Thus once we know the number of successes, we immediately know the number of failures (and vice versa). Thus there are only g degrees of freedom associated with the contingency table. If we estimate p parameters in obtaining the probability estimates for the expected frequencies we have g – p degrees of freedom left. Thus the Pearson chi-squared statistic and G² statistic have an asymptotic chi-squared distribution with g – p degrees of freedom, . The usual caveats on minimum cell sizes hold for the expected frequencies so we should have no more than 20% with 5 or fewer observations.

The connection between Pearson residuals and the Pearson chi-squared statistic

• In the goodness of fit test just constructed suppose we look only at the terms contributed by one column in the above tables. Suppose it is the i^th column. The contribution that the two terms in this column make to the Pearson chi-squared statistic is shown below.

• Thus the Pearson chi-squared goodness of fit test reduces to the following.

• Recall that for the binomial distribution, a member of the exponential family, we proved in Lecture 21 that

• Thus we see that

• The term on the right hand side of this last equation was defined in Lecture 24 to be the Pearson residual. Hence the Pearson chi-squared goodness of fit test for a logistic regression on grouped binomial data can be calculated by extracting the Pearson residuals from the logistic regression model, squaring them, and adding them up.
• An exactly parallel argument can be used to show that the G² statistic is equal to the sum of the squared deviance residuals. Recall that the sum of the squared deviance residuals is what is called the residual deviance. Thus just as in Poisson regression the residual deviance (as well as the sum of squared Pearson residuals) can be used as a goodness of fit statistic for grouped binary data.
• This result does not hold for ungrouped binary data. We'll consider goodness of fit tests for ungrouped binary data next time.

Course Home Page