Estimators
Introduction to Estimators
Recommended Prerequesites
- Probability
Introduction
An estimator is a function or rule that takes a sample of data and produces an estimate of some population parameter.
Let \(\theta\) be a parameter of interest (e.g., the population mean, variance, or proportion), and let
\(X_1,X_2,\dots,X_n\) be a random sample from a population. An estimator \(\hat{\theta}\) is a function of the sample:
$$\hat{\theta}=\hat{\theta}(X_1,X_2,\dots,X_n)$$
The estimate \(\hat{\theta}\) is a specific value obtained by applying the estimator to a given data set
Example
The sample mean, \(\bar{X}\), is an estimator for the population mean \(\mu\):
$$\bar{X}=\frac{1}{n}\sum_{i=1}^{n}X_i$$
Bias
The bias of an estimator measures how far the expectedvalue of the estimator is from the true parameter value.
$$\text{Bias}(\hat{\theta})=\mathbb{E}[\hat{\theta}]-\theta$$
- If \(\mathbb{E}[\hat{\theta}]=\theta\), the estimator is said to be unbiased.
- If \(\mathbb{E}[\hat{\theta}]\neq\theta\), the estimator is biased
Example
The sample variance \(S^2\) is an unbiased estimator for the population variance \(\sigma^2\).
However, the biased sample variance estimator \(\hat{\sigma}^2\):
$$\hat{\sigma}^2=\frac{1}{n}\sum_{i=1}^{n}(X_i-\bar{X})^2$$
has bias \(\frac{n-1}{n}\sigma^2\), making it a biased estimator of the population variance.
Variance of an Estimator
The variance of an estimator measures the expected deviation of the estimator from its expected value. The variance of \(\hat{\theta}^2\) is given by:
$$\text{Var}(\hat{\theta})=\mathbb{E}[(\hat{\theta}-\mathbb{E}[\hat{\theta}])^2]$$
Bias-Variance Trade-off
The mean squared error of an estimator can be broken down into the two previously mentioned quantities:
$$\text{MSE}(\hat{\theta})=\mathbb{E}[(\hat{\theta}-\theta)^2]=\text{Var}(\hat{\theta})+[\text{Bias}(\hat{\theta})]^2$$
Consistency
Formally stated, consistency is that the probability that the bias deviates from the true parameter by a given amount goes to 0 as the sample sizes goes to infinity:
$$\lim_{n\rightarrow\infty}P(|\hat{\theta}-\theta|>\varepsilon)=0$$
The previously mentioned biased sample variance estimator is consistent.
Sufficient Statistics
Marginal Likelihood Practice Problems