Lecture 41—Friday, March 31, 2006

What was covered?

Random versus fixed effects—contrasting the individual regressions returned from the multilevel model with the separate regressions model.
Terminology associated with multilevel modeling
A multilevel model imposes a correlation structure on a data set

Terminology defined

composite equation

Fixed effects versus random effects

Last time we presented the random slopes and intercepts model for the invertebrate development time hierarchical data set. In the multilevel model formulation there are separate equations at each level of the hierarchy: individual measurement equations at level 1 and species-level equations at level 2.

Because the level-2 equations serve only to define variables that appear in the level-1 equation, we can plug the level-2 equations into the level-1 equation to yield an alternate representation of the multilevel model. This representation in which the existence of the individual levels is suppressed is called the composite equation.

It's worth comparing this composite equation with the separate regressions model (using dummy variables for the various species) that we developed last time.

The presence of the dummy variables in this last model obscures the common features shared by the multilevel model and the separate regressions model. Although the interpretation of what β₀ and β₁ represent in the two equations is quite different, (in the multilevel model they correspond to the population values of the intercept and slope while in the separate regressions model they correspond to the intercept and slope for species 1), the individual slopes and intercepts for different species take exactly the same form.

Model	Intercepts	Slopes	# parameters
multilevel model			6
separate regressions			149

In the separate regressions model we need to know the values of 148 different quantities, , in order to specify the regression equations for all 74 species.
In the multilevel model we need to know the values of 150 different quantities, , in order to specify the regression equations for all 74 species.

The main difference though is that in the separate regressions model the 148 quantities are all fixed effects while in the multilevel model only two of the parameters, , are fixed effects while the remaining 148 quantities are random effects.
This is more than just semantics. In the frequentist approach to fitting multilevel models, the random effects are not parameters to be estimated. In fact there are only six model parameters estimated in fitting the multilevel model: (the population intercept and slope), (the level-1 variance), and (the parameters of the variance-covariance matrix of the random effects). In the separate regressions model, on the other hand, there are 149 parameters to be estimated (74 intercepts, 74 slopes, and the error variance).
- If it's desired one can obtain estimates (more properly called predictions) of the random effects after the multilevel model is fit. These predictions are generally referred to as empirical Bayes estimates.
This highlights the fundamental difference between these two modeling approaches. The separate regressions model treats slopes and intercepts as fixed effects and estimates them directly, while the multilevel model treats slopes and intercepts as random effects and estimates the parameters of their distribution instead. The implications of this choice for grouped data are summarized in the table below.

Separate regressions model	Random slopes and intercepts model
Treats the slopes and intercepts of individual groups (species) as fixed effects.	Treats the slopes and intercepts of individual groups (species) as random effects.
This is the natural choice if the focus is on the individual species that occur in the data set.	This is the natural choice if the focus is not the particular species that happen to occur in the sample, but instead is on characterizing the distribution of the species values.
This is a sensible choice any time the variable defining the structure is really a predictor of interest.	This is a sensible choice any time the variable defining the structure serves essentially as a blocking variable.
This method tends to overfit the data especially for those groups (species) that have very little data.	For groups (species) with very little data, the individual predictions are shrunk to the mean thus preventing overfitting.
This is a sensible approach if there are very few groups.	This is a sensible approach if there are very many groups.
Generally this is not the parsimonious alternative. When there are many groups present a large number of parameters will need to be estimated.	Usually this is the more parsimonious alternative. Regardless of the number of groups present, the number of parameters that need to be estimated will not change.
If there are a lot of groups present and hence many parameters that need to be estimated, the variances obtained for the individual parameter estimates will tend to be inflated. Thus precision is sacrificed.	If there are very few groups present then it may be difficult to estimate some of the parameters (variances and covariances) of the random effects distribution. This is more problematic with a frequentist approach to estimating multilevel models, less so with Bayesian methods.

Ecologists in particular seem to be overly concerned with the problem of deciding when a factor should be treated as fixed and when a factor should be treated random. See, e.g., Beck (1997). A quick Google search will bring up a large number of guidelines for distinguishing fixed from random effects. In my opinion the philosophical distinctions between these two approaches are overrated. Any time a factor has many levels it probably makes sense to treat it as a random effect.

Some terminology

I've been using the label "multilevel model" to describe a regression model fit to hierarchical data in which random effects are used to described the heterogeneity (correlation) inherent in such data. The term multilevel model is very popular in the social sciences, particularly in political science, sociology, and education research.
In other fields different terminology is used when referring to such models. Other commonly used terminology includes random effects models, random coefficients models, mixed effects models, mixed models, hierarchical models, and nested models. Generally I will treat these all as synonyms here. It's worth noting that random effects models can be used with a larger class of data sets than just those that are hierarchically structured. Random effects that are used with non-hierarchical data are generally referred to as crossed random effects.
The terms nested design and split plot design refer to specific kinds of experimental designs in which the data have a hierarchical structure and, particularly in split plot designs, where different treatments are applied to different levels of the hierarchy. Generally speaking such experimental designs can be analyzed using random effects models such as the ones we're describing here.
There is now a huge literature on the topic of multilevel models, most of it from fields outside of ecology. Textbook examples include Brown & Prescott (1999), Gelman & Hill (2007), Goldstein (1995), Hox (2002), Kreft & de Leeuw (1998), Leyland & Goldstein (2001), Pinheiro & Bates (2000), Raudenbush & Bryk (2002), Snijders & Bosker (1999), and Verbeke & Molenberghs (2000). Textbooks specializing in the analysis of longitudinal data (repeated measures data) also tend to have extensive coverage of multilevel models. Example from this group include Fitzmaurice et al. (2004), Frees (2004), Singer & Willett (2003), and Twisk (2003).

The connection between correlation and observational heterogeneity

In introducing structured data in lecture 38, I stressed the fact that what characterizes structured data is observational heterogeneity, that some groups of observations differ from other groups. The flip side of the observational heterogeneity coin is correlation. When some groups of observations differ from other groups that means that observations within that group must be more similar and hence correlated. I now demonstrate that both heterogeneity and correlation flow directly from our algebraic formulation of the multilevel model.
To use more generic notation let and . To simplify things let's focus on the random intercepts model, a model in which the slope is constant across groups. For this the 2-level model is the following.

or in composite form

As noted in the last equation, the parameters and the predictor are just fixed constants. Only and are random quantities that have a distribution. Recall that if a and b are fixed constants, and W, X, Y, and Z are random variables, then properties of the covariance operator yield the following.

Next consider two observations and potentially coming from different groups. The composite equations using the random intercepts model for these two observations are below.

The covariance between these two observations is given by the following.

Lines 2 and 3 follow from the covariance properties given above and line 4 follows from the fact that the level-2 random effects are independent of the level-1 errors.

Now the values of the remaining covariances vary depending on the values of i, j, k, and l. For instance, we have

From this it follows that

This result demonstrates the heterogeneity that we've been claiming exists. Observations from the same group of species (i = k) have a positive covariance, , while observations from different species (i ≠ k) have a covariance of zero. So observations coming from the same group are more similar (have positive covariance) than observations from different groups (zero covariance).
Using the definition of correlation we can translate this statement about heterogeneity into one about correlation. For distinct observations with j ≠ l we have

and so we see that observations coming from the same level-2 unit (species) are correlated while observations from different level-2 units are uncorrelated.

The random intercepts model yields the simplest possible correlation structure for a structured data set. If we allow the slopes to be random too, the correlation becomes more complicated including both the variance and covariance terms of the random effects, as well as the value of the predictor, temperature.

Cited references

Beck, M. W. (1997) Inference and generality in ecology: Current problems and an experimental solution. Oikos 78(2): 265–273.
Brown, H. & Prescott, R. (1999) Applied Mixed Models in Medicine (Wiley, New York).
Fitzmaurice G. M., Laird, N. M., & Ware, J. H. (2004) Applied Longitudinal Analysis (Wiley, New York).
Frees, E. W. (2004) Longitudinal and Panel Data (Cambridge University Press, Cambridge, UK).
Gelman, A. & Hill, J. (2007) Data Analysis Using Regression and Multilevel/Hierarchical Models (Cambridge University Press, Cambridge, UK).
Goldstein, H. (1995) Multilevel Statistical Models (Edward Arnold, London).
Hox, J. J. (2002) Multilevel Analysis: Techniques and Applications (Lawrence Erlbaum Publishers, Mahwah, NJ).
Kreft, I. G. G. & de Leeuw, J. (1998) Introducing Multilevel Modeling (Sage Publications, Thousand Oaks, CA).
Leyland, A. H. & Goldstein, H. (2001) Multilevel Modelling of Health Statistics (Wiley, New York).
Pinheiro, J. C. & Bates, D. M. (2000) Mixed-Effects Models in S and S-Plus. (Springer-Verlag, New York).
Raudenbush, S. W. & Bryk, A. S. (2002) Hierarchical Linear Models (Sage Publications, Thousand Oaks, CA).
Snijders, T. A. B. & Bosker, R. (1999) Multilevel Analysis (Sage Publications, Thousand Oaks, CA).
Singer, J. D. & Willett, J. B. (2003) Applied Longitudinal Data Analysis: Modeling Change and Event Occurrence (Oxford University Press, Oxford, UK).
Twisk, J. W. R. (2003) Applied Longitudinal Data Analysis for Epidemiology: A Practical Guide (Cambridge University Press, Cambridge, UK).
Verbeke, G. & Molenberghs, G. (2000) Linear Mixed Models for Longitudinal Data (Springer-Verlag, New York) .

Course Home Page

Jack Weiss
Phone: (919) 962-5930
E-Mail: jack_weiss@unc.edu
Address: Curriculum in Ecology, Box 3275, University of North Carolina, Chapel Hill, 27516
Copyright © 2006
Last Revised--August 15, 2008
URL: http://www.unc.edu/courses/2006spring/ecol/145/001/docs/lectures/lecture41.htm