Lecture 28—Wednesday, March 1, 2006

What was covered?

• Regression models with both continuous and categorical predictors
• Analysis of covariance
• Sequential testing of models in the analysis of covariance scenario

Terminology Defined

• allometric equation
• analysis of covariance
• coincident lines model
• covariate
• interaction
• non-parallel lines model
• parallel lines model

Regression with Both Continuous and Categorical Predictors

• I consider again the fictitious sociological example involving education, income, and gender. The goal of the study was to relate the income received in various lines of work to the level of education required for the job while accounting for the potentially modifying role of gender on the relationship. Suppose we use an indicator coding for gender as follows:

We consider the general problem of formulating the relationship as a regression model.

• Suppose we believe that income is linearly related to education, but that the relationship may be different depending upon gender. With a single continuous predictor and a single categorical predictor with two levels there are three sensible models we might consider. They are the following.
  1. coincident lines model,
  2. parallel lines model, and
  3. non-parallel lines model
• Model 1: Coincident lines model—no gender effect. Both genders show the same income-education relationship.

• Model 2: Parallel lines model—additive gender effect on intercept. Both genders show the same income-education relationship except that they have different baseline values.

• If the primary goal is to study the differences in income between males and females, then education becomes essentially just a control variable. Since we believe income and education are related, the question of an income difference between males and females is in general not well-defined if we don't also specify the education level. But if the income-education relationship is identical in males and females, then the notion of a gender effect on income is well-defined. It's just the vertical distance between the two lines, or β₂ in the model. But this constant difference doesn't manifest itself unless we simultaneously control for education.
• A model containing a control variable (education) and a categorical variable of interest (gender) where we assume the relationship between the response variable and the control variable is the same for all levels of the categorical variable is called an analysis of covariance model. A predictor that serves solely as a control variable is usually called a covariate.
• Analysis of covariance is perhaps the most important application of regression in observational studies. It is a straight-forward example of controlling for something statistically that wasn't controlled for at the experimental design stage.

• Model 3: Non-parallel lines model—additive gender effect on both intercept and slope. In this model the genders show different income-education relationships and have different baseline values. To fit this as a regression model we include a term that is the product of education and gender. This is called an interaction term. Thus the non-parallel lines model is often called an interaction model.

• Of course we could also consider a Model 0. In this model there is no relationship between income and education. I'll assume that this model has been rejected.

Testing the Models

• In the information-theoretic approach we would fit all three models and choose the model with the lowest AIC.
• In the significance testing approach, a premium is placed on coming to a decision while carrying out as few tests as possible. The rationale for this is as follows. Suppose we carry out our tests with α-level equal to .05. This means that each time we carry out the test we are willing to be wrong 5% of the time. (The α-level is the probability of rejecting the null hypothesis when the null hypothesis is true.) If we carry out multiple tests then this potential for error compounds itself. The compounding is not exactly additive, but it does increase with each test carried out so that in the end the realized α-level can far exceed .05. Thus it behooves us to minimize the number of tests we do.
• The recommended approach for testing these models is to use a sequential approach.
  1. Start with the most complex model (model 3) and see if it can be reduced to model 2. This means testing the hypothesis H₀: β₃ = 0, that the two lines have the same slope. If this hypothesis is not rejected, it is safe to reduce the model to model 2.
  2. If model 2 has been chosen, see if it can be reduced to model 1. This means testing the hypothesis H₀: β₂ = 0, that the two lines have the same intercept. If this hypothesis is not rejected, it is safe to reduce the model to model 1.
  3. If model 1 has been chosen conclude there is no gender effect.
• Notice that if we take the sequential approach then certain models are never considered.
  • For example, in model 3 why not test the hypothesis H₀: β₂ = 0? If this hypothesis is not rejected we could then drop β₂ from the model and conclude that the lines have the same intercept but different slopes.
  • We might call this: Model 2.5: Non-parallel lines, same intercept

• As you can see there's nothing wrong with this model per se, but it seems rather uninteresting. To put it more charitably I'd call this model "not broadly applicable".
• What's peculiar about Model 2.5 is that it contains the interaction term (education gender) but does not include the term gender all by itself. Such a model violates what's often called the principle of marginality.
• Principle of marginality: A model including a higher order term (e.g., an interaction term) should also include all lower order relatives (the so-called main effect terms) that comprise the interaction. Some authors take this principle as gospel and consider any violation of it to be a cardinal sin. This is what I was taught but I've come to realize it's just a point of view. In particular, if you treat this principle as a law then you're forced to reject most of much of what is done under the guise of regression and classification trees (CART). In CART it is typical to have interactions without main effects.

A More Complicated Analysis of Covariance Example

• We'll use some data on energy use by volant birds and mammals that were collected by Speakman and Racey (1991). Paralleling their work we'll try to determine whether echolocation is a costly drain on an animal's energy resources during flight.
• Echolocation as a trait has evolved in very few animal species perhaps because of the high energy costs involved in producing pulses. If the combined energy costs of echolocation and flight are additive one might expect that the flight energy costs of echolocating bats would be higher than that of non-echolocating bats and birds. Of course other uncontrolled factors might confound such a relationship.
• One such uncontrolled factor is size. In this study body mass was not randomly assigned to the "treatment" groups—animal types. As a result echolocating bats came small and non-echolocating bats came big. Of course mass is not something that can be randomly assigned to an animal (although perhaps an effort could have been made to find a greater size diversity). In fact non-echolocating bats comprise the megachiroptera a taxonomic group that includes the flying foxes who are some the world's largest bats.
• Energy use is typically related to body mass through a specific type of nonlinear equation called an allometric equation. This is an equation of the form

Here b is called the allometric parameter. If b = 1 then the equation is called isometric. For our data y is rate of energy usage and x is body mass.

• Note: the rest of the civilized world would probably call this equation a power equation, but the term allometric equation has become ensconced in the biological vernacular.

• Historically allometric equations were fit by first linearizing them using a log-log transformation. This yields the following equation.

Now the allometric equation and its log-transformed equivalent are very different equations. They yield different estimates of the parameters and make different assumptions about the error distributions of the residuals. Without doing a little more work it's not obvious to me which is the better equation to use with these data. But because Speakman and Racey (1991) used the log-transformed version, so will we.

• The bottom line is that body size is undermining the goal of this study because it's hopelessly confounded with the variable of interest (animal category type). But there is a solution to this problem. Even though we can't control for the bias through randomization and good experimental design we can still try to control for it after the fact by applying appropriate statistical methods.
• Regression is the standard technique for doing this. What we'll do is essentially correct for the bias by subtracting off (statistically) the effect due to mass and only then use the animal's category to account for whatever unexplained variability remains. We do this by fitting a single regression model including both body mass and animal category as predictors. Because body mass appears in the model largely to correct for a bias in the sample, we generally refer to it as a covariate (rather than a predictor).
• In order for this method to work, we need to be able to make the same correction for all animals. This is equivalent to being able to use the same slope in the regression of rate of energy use on body size for all three animal groups. Thus what we have is an analysis of covariance problem in which the continuous covariate is body mass and the predictor of interest is animal type, a categorical variable with three levels.

Cited reference

• Speakman, J. R. and P. A. Racey. 1991. No cost for echolocation for bats in flight. Nature 350: 421–423.

Course Home Page