Method of Moments

Introduction to Method of Moments

Recommended Prerequesites

1. Probability
2. Probability 2

Definition

The Method of Moments is a technique used to estimate the parameters of a probability distribution by equating sample moments to the corresponding population moments. It serves as a less common alternative to MLE, which can often be simpler.

Sample Moments

Given a set of data \(X_1, X_2, \dots, X_n\), the sample moments are empirical counterparts of population moments. The k-th sample moment is defined as: $$m'_{k}=\frac{1}{n}\sum_{i=1}^{n}X_{i}^k$$ Similarly, the k-th sample central moment is given by: $$m_{k}=\frac{1}{n}\sum_{i=1}^{n}(X_{i}-\bar{X})^k$$ where \(\bar{X}\) is the sample mean.

Performing the Method of Moments

The idea behind the Method of Moments is to estimate the parameters of a probability distribution by matching the first few sample moments with their corresponding population moments. Suppose a distribution is characterized by a set of parameters \(\theta_1,\theta_2,\dots,\theta_k\). To estimate these parameters:

1. Compute the first k sample moments \(m'_1,m'_2,\dots,m'_k\)
2. Equate the sample moments to the corresponding population moments expressed as functions of \(\theta_1,\theta_2,\dots,\theta_k\)
3. Solve the resulting system of equations to find estimates of the parameters.

Example

The probability density of the gamma distribution is given by: $$f(x;\alpha,\beta)=\frac{x^{\alpha-1}e^{-x/\beta}}{\beta^\alpha\Gamma(\alpha)}\quad\text{for }x>0$$ where \(\alpha\) is the shape parameter and \(\beta\) is the scale parameter. The population moments of the gamma distribution are: $$\mu'_1=\alpha\beta$$ $$\mu'_2=\alpha(\alpha-1)\beta^2$$ To estimate \(\alpha\) and \(\beta\) using the Method of Moments, we equate the first two sample moments to the corresponding population moments: $$m'_1=\alpha\beta$$ $$m'_2=\alpha(\alpha-1)\beta^2$$ Solving for \(\beta\) using the first equation: $$\beta=\frac{m'_1}{\alpha}$$ And substitute into the second equation: $$m'_2=\alpha(\alpha+1)\left(\frac{m'_1}{\alpha}\right)^2$$ Simplify and solve for \(\alpha\), then use the solution to find \(\beta\).

Method of Moments Practice Problems

1. Suppose \(X_1, X_2,\dots, X_n\) are iid samples from a binomial distribution \(\text{Bin}(n,p)\).
   1. Write down the first moment (mean) of the binomial distribution in terms of n and p
   2. Use the method of moments to derive an estimator for p, assuming n is known.
   3. If \(n=10\) and the sample mean is 4, estimate p.
2. Let \(X_1,X_2,\dots,X_n\) be iid samples from an exponential distribution with parameter \(\theta\), where the PDF is given by: $$f(x;\theta)=\frac{1}{\theta}e^{-x/\theta}$$
   1. Write down the first moment of the exponential distribution
   2. Use the method of moments to derive an estimator for \(\theta\)
   3. If the sample mean is 5, estimate \(\theta\)
3. Suppose \(X_1, X_2,\dots,X_n\) are iid samples from a uniform distribution:
   1. Write down the first and second moments of the uniform distribution in terms of a and b.
   2. Use the method of moments to derive estimators for both a and b
   3. If the sample mean is 4 and the sample variance is 9, estimate a and b.
4. Suppose \(X_1, X_2,\dots,X_n\) are iid samples from a gamma distribution whose pdf is given by: $$f(x;k,\theta)=\frac{1}{\Gamma(k)\theta^k}x^{k-1}e^{-x/\theta};\quad x\gt 0$$ where k is known and \(\theta\) is the parameter of interest. The following integral may be useful: $$\int_{0}^{\infty}x^ne^{-ax}dx=\frac{n!}{a^{n+1}}$$
   1. Write down the first moment (mean) of the gamma distribution in terms of k and \(\theta\)
   2. Use the method of moments to derive an estimator for \(\theta\)
   3. Given k=2 and a sample mean of 6, estimate \(\theta\)
5. Suppose \(X_1, X_2,\dots,X_n\) are iid samples from a beta distribution whose pdf is given by: $$f(x;\alpha,\beta)=\frac{\Gamma(\alpha+\beta)}{\Gamma(\alpha)\Gamma(\beta)}x^{\alpha-1}(1-x)^{\beta-1}$$ The following integral may be useful: $$\int_0^{1}x^{a}(1-x)^{b}=\frac{B(a+1,b+1)}{a+b+1}$$
   1. Write down the first two moments of the beta distribution in terms of \(\alpha\) and \(\beta\)
   2. Use the method of moments to derive estimators for \(\alpha\) and \(\beta\).
   3. Give a sample mean of 0.4 and a sample variance of 0.1, estimate \(\alpha\) and \(\beta\).