Lecture 25—Friday, February 24, 2006

What was covered?

• Using multiple regression models to infer causal relations
• Using multiple regression to increase precision
• Using multiple regression to reduce bias
• Including categorical variables in regression models

Terminology Defined

• confounder
• ecological fallacy
• marginal relationship
• partial relationship
• Simpson's paradox

Some reasons for fitting multiple regression models

Inferring causal relations

• Using the motivating example I described last time, consider the following information gleaned from separate simple linear regressions using the variable prestige as the response.
• Simple linear regression of prestige on income
  • Fitted regression equation: prestige = 0.28 + 0.90*income
  • Interpretation: an increase in an occupation's income level leads to an increase in its perceived prestige.
  • Causal diagram interpretation

• Simple linear regression of prestige on education
  • Fitted regression equation: prestige = 2.46 + 1.08*education
  • Interpretation: an increase in an occupation's education level leads to an increase in its perceived prestige.
  • Causal diagram interpretation

• At this point interpreting the combined effect of income and education on prestige would be entirely speculative, but an obvious possibility is shown in the causal diagram below.

• Simultaneous (multiple) regression of prestige on both education and income
  • Fitted regression equation: prestige = –6.06 + 0.56*education + 0.60*income
  • Interpretation: an increase in either an occupation's education level or an increase in an occupation's income level leads to an increase in its perceived prestige.
  • Possible causal diagram interpretation: Notice that the nature of the effects of education and income on prestige are the same as we saw in the individual regressions. They're both positive. But notice that the magnitudes of the effects (as measured by the regression coefficients) have gotten smaller. This suggests a different causal interpretation of the relationships between the three variables such as the one that is shown below.

• From the point of view of income the original one-variable estimate of the causal effect of income on prestige was partly spurious. Part of the effect we observed was due to the mutual dependence of income and prestige on education. The reduction we see in the regression coefficient of income in the multiple regression model (from 0.90 to 0.60) is due to the removal of this spurious effect from the relationship.
• From the point of view of education income is an intervening variable. Part of the effect of education on prestige is due to its indirect effect on prestige through income.
• Thus in a causal framework multiple regression allows us to identify spurious relations and to distinguish indirect effects from direct effects.
• Note: the point of view taken here is that the regression model approximates the true structural model that relates the variables. Even if we reject this causal interpretation and view the regression model as purely an empirical model, the issues of indirect vs direct effects and spurious relations can still be important.
  • Suppose we fit two models, one with a regressor x and a second model with the regressors x and z.

• If we observe that the estimate of β₁ is markedly different in the two models, we call the variable z a confounder or an effect modifier. It's presence in the model changes the nature of the relationship between y and x.

Controlling precision

• I next consider the following fictitious example. In a study the occupational income, education level, and gender of various participants were obtained. The goal of the study was to relate income earned to education level achieved accounting for the potentially modifying role of gender on the relationship.
• Suppose the relationship between income and education is as shown in the diagram below.

• Whether we ignore gender or include it in the relationship between income and education we end up with the same slope for the regression line. As shown in the diagram, the regression lines for the partial relationships (separate lines for each gender) are parallel to the line for the marginal relationship (the single regression line ignoring gender).
• But if we include gender we increase the precision of our estimate of the slope. This can be seen by the fact that the vertical spread about the individual partial regressions is much less than the spread about the single marginal regression line.
• This is accomplished by fitting a multiple regression additive model with two predictors: education and gender.
  • We obtain the increase in precision by using all of the data to estimate a common slope and only splitting the data into genders to estimate the separate intercepts.
  • The multiple regression model allows us to test for a common slope for the genders. How to do this will be demonstrated in one of our computer sessions.
• We accomplish all of this because gender and education are independent of each other. The education distributions for the two genders are identical and hence do not depend on gender.

Avoiding bias

• Consider our fictitious example 2 again. Suppose this time the relationship between income and education is as shown in the diagram below.

• This time gender and education are not independent of each other. Females tend to be better educated than males.
• Also notice that the partial relationships are the same as before—a positive association between income and education.
• But this time the marginal relationship is very different. If we ignore gender we actually obtain a negative relationship between income and education.
  • The situation where the partial and marginal relationships differ in sign is an example of what's called Simpson's paradox. By fitting a multiple regression model with both education and gender as predictors we avoid the potential bias of Simpson's paradox.
• Note: If we estimate a marginal relationship and assume it also holds for unobserved partial relationships, we are committing what's called the ecological fallacy. In the ecological fallacy we assume the behavior of the aggregate also holds for the behavior of its components.

Using Categorical Variables in Multiple Regression

• Two of the uses of multiple regression described above involved using categorical variables as predictors. Of course we could have just fit simple linear regression models separately for males and females and uncovered the same patterns. So why bother with multiple regression?
  • When the number of categories is large and/or many categorical variables are being examined, the strategy of fitting separate simple linear regressions for all the different combinations of the different levels of the categorical variables is impractical.
  • Fitting separate linear regressions is also less precise. If a particular categorical variable has no effect in a multiple regression we can pool over the levels of that variable and use the increase in effective sample size to obtain more precise estimates of the remaining effects.
• So how should categorical variables be included in a regression model?

The wrong way to do it (except when there are only two levels)

Example 1—a categorical variable with two levels

• Consider again our second fictitious example using income as the response with education and gender as predictors. Suppose we treat gender as a numeric variable with females coded 1 and males coded 2.

• We fit the additive model: income = β₀ + β₁ education + β₂ gender
  • In this model we assume that both genders exhibit the same effect of education on income but they start from different baselines.
• Using the coding scheme above this yields the following equations for the genders.

• This translates into the following diagram

• Since β₀ and β₂ are estimated separately there are no constraints placed on the location of the intercepts. The location of the intercept for females does not influence the location of the intercept for males.
• A simpler interpretation of the coefficients can be obtained if we make the following identifications.

• This is identical to the coding scheme we used in fitting probability models to the slug data set. There we used a variable called field.dummy which was coded 0 if field = Nursery and was coded 1 if field = Rookery.
• Using this coding scheme yields the following equations for the genders.

• This translates into the following diagram

• Now we see that β₀ corresponds to the intercept for females and β₂ is the difference in the value of the intercepts between males and females. A test of β₂ = 0 would be a test of whether the lines for males and females are coincident.

Course Home Page