Lecture 9 —Friday, January 27, 2006

What was covered?

• Terminology and notation of estimation theory
• The sampling distribution of a statistic
• Criteria for evaluating an estimator

Terminology Defined

• Parameter
• Statistic
• Estimator
• Estimate
• Sampling distribution
• Method of moments estimator
• Consistent estimator
• Unbiased estimator
• Bias
• Precise estimator
• Standard error of an estimate

Introduction to Estimation Theory

• Parameter: a numerical characteristic of a population
  • In ecology we might be interested in the diversity of plants in a habitat. Diversity is a parameter.
  • In statistics we think of populations as being defined by probability distributions as in "a normally distributed population".
  • The formula for the normal density function is

  

  • Since a normal distribution is fully defined by two parameters, the mean and the variance , this would be an example of a two-parameter population.
  • Typically, Greek letters are used to represent parameters. When we wish to focus on the parameters of a probability distribution we will specify the parameters as part of the notation for the density or probability mass function, e.g., . Thus a normal density function would be written as . This is just notation and doesn't imply a joint distribution or anything of the sort. Depending on context we may use to represent a scalar or a vector.
  • Similarly we've looked at the negative binomial distribution. In the ecologist's version of the formula and are the parameters so this is another two-parameter distribution. Using our new notation we would write it as follows.

• Statistic: a numerical characteristic of a sample
  • In ecology we might lay out a series of quadrats and calculate the diversity of plants in each quadrat or over all quadrats using perhaps the Shannon-Wiener index. Diversity in this case is a statistic.
  • If we take a sample of a random variable from a normally distributed population and calculate the mean of the random variable in the sample, this sample mean is a statistic.
• An estimator is a statistic (meaning that it is calculated from a sample) whose calculated value is used as an estimate (numerical surrogate) of a population parameter. Since an estimator is a function of the sample, it is a random variable. Each of the above examples of statistics are also estimators of the corresponding population quantity.
• A particular realization of the estimator (based on the particular sample we obtain) is called an estimate.
  • If is the population parameter of interest, an estimator of is denoted by .
  • We will use the symbol both for the random variable (estimator) as well as a particular realization of the random variable (estimate).
• Notation: If Greek letters are used for parameters, then typically the associated estimator of the parameter uses the equivalent Roman letter or, alternatively, the Greek letter with a carat (hat) on top. There are some exceptions to this rule.

Examples	Parameter	Estimator
hat notation	, generic parameter
Roman letter or hat notation	, regression coefficient	b or
Nonstandard notation	, population mean	, sample mean , sample median

Sampling Distributions and the Frequentist Interpretation of Statistics

• In the frequentist interpretation of statistics, parameters are fixed quantities in nature whose values we are trying to estimate. In the frequentist point of view there is no uncertainty associated with parameters; they are what they are.
• On the other hand, statistics, the quantities we calculate from samples, are random variables, i.e., they have uncertainty associated with them. More specifically, they have a probability distribution. The probability distribution arises from the variability in our statistic that we would expect to see under repeated sampling. Accordingly we call the probability distribution of a sample statistic its sampling distribution. Since an estimator is a random variable based on a sample and hence a statistic, it too has a sampling distribution.
• Comments about sampling distributions
  • The reality of sampling distributions is obvious. If we take a second sample in exactly the same way we took the first sample, the statistics we calculate on the second sample will differ from those we calculated using the first sample. Sample statistics vary from sample to sample.
  • Still, the sampling distribution is really only a theoretical construct. We generally don't take multiple samples. Hence our understanding of the nature of the sampling distribution of a particular statistic will derive from theoretical considerations or from computer simulations.
• The importance of the sampling distribution to us is that criteria for determining whether an estimator is good or not are based entirely on the characteristics of its sampling distribution.
• We always make the assumption that the sample was constructed using probabilistic methods. If this is not the case, the science of statistics has nothing constructive to say about the estimates we calculate from such a sample.

How Does One Choose an Estimator?

• A simple-minded approach is if a formula is used to calculate the population parameter, then use that same formula on the sample to construct an estimator. This approach is called the "method of moments".
• For a discrete random variable X, such as a Poisson, its expected value is calculated from

Now suppose we have obtained a sample of size n in which individual j has value x_j of X. Suppose n_k individuals (n_k of the x_j s) are observed to have value k. Then the sample estimate of would be just the fraction of individuals observed to have value X = k, namely . Therefore the method of moments estimator of μ would be

• For a discrete random variable X, such as a Poisson, its variance is calculated by

Given a sample of size n, the method of moments estimator of would use as an estimate of and then following the same logic as above would be the following.

But from your elementary statistics class you already know that this is not the estimator that is commonly used. We instead use the sample variance, in which the divisor is n – 1 rather than n, because it turns out to be a better estimator.

There is no theory to support using method of moments estimators in general, but they are often a good starting point in constructing an estimator. They are typically used as initial guesses in numerical algorithms.

• In some cases there are a number of sensible candidates for an estimator. For a normally distributed random variable, the population mean, median, and mode coincide. This would suggest that either the sample mean or sample median might be good choices (the sample mode would be nonsensical for a continuous variable). In fact, it can be shown that the sample mean in this case is a better estimator than the sample median.

Characteristics of Good Estimators

• Ideally we would like our estimate to improve with sample size, i.e., as sample size goes up, gets closer to the true value of . Such an estimator is said to be consistent.

• Our estimate should be neither to high nor two low. On average we want an estimator that is right on the true value. In terms of sampling distributions, we want an estimator whose sampling distribution is centered on the true population parameter value. Such an estimator is said to be unbiased.

• Bias measures whether a statistic is consistently too low or too high in its estimate of a parameter. Formally it is defined as follows.

.

The E that appears in this expression is the expectation operator. For all intents and purposes you can think of as the mean of the sampling distribution of . is said to be an unbiased estimator of if .

• Precision: Our estimator should have low variability. Precision is a measure of the variation of the sampling distribution of our statistic. The more variable is our statistic, the less precise it is. If an estimator has low precision, then the estimate we actually obtain could be far away from the true population value just by chance.

• The square root of the variance of an estimator is called its standard error.

• More precisely, the standard error an estimator is the square root of the variance of its sampling distribution. Since this is a mouthful, we just say standard error.
• Good estimators have small standard errors.

Comparing estimators

• Suppose estimators and for have the same precision. Suppose is unbiased for but is not. In this situation is clearly the best estimator. The diagram below illustrates sampling distributions for and that meet these conditions.

• Suppose estimators and for are both unbiased for but is more precise than is . In this situation is clearly the best estimator. The diagram below illustrates sampling distributions for and that meet these conditions.

• Things obviously get more complicated when different properties are at odds with each other. For example, in comparing an imprecise but unbiased estimator against one that is very precise, but biased, which one should one choose?
• Next time we'll begin our discussion of a very popular method of estimation called maximum likelihood estimation. Maximum likelihood estimates overall have very good properties. Unfortunately these good properties are only guaranteed to hold when the sample size is large. How large is large is not clear in general.

Course Home Page