What would be the learning outcome from this slecture?

• Basic Theory behind Maximum Likelihood Estimation (MLE)
• Derivations for Maximum Likelihood Estimates for parameters of Exponential Distribution, Geometric Distribution, Binomial Distribution, Poisson Distribution, and Uniform Distribution

Outline of the slecture

• Introduction
• Derivation for Maximum Likelihood Estimates (MLE) for parameters of:
  • Exponential Distribution
  • Geometric Distribution
  • Binomial Distribution
  • Poisson Distribution
  • Uniform Distribution
• Summary
• References

Introduction

The maximum likelihood estimate (MLE) is the value $ \hat{\theta} $ which maximizes the function L(θ) given by L(θ) = f (X₁,X₂,...,X_n | θ) where 'f' is the probability density function in case of continuous random variables and probability mass function in case of discrete random variables and 'θ' is the parameter being estimated.

In other words,$ \hat{\theta} $ = arg max_θ L(θ), where $ \hat{\theta} $ is the best estimate of the parameter 'θ' . Thus, we are trying to maximize the probability density (in case of continuous random variables) or the probability of the probability mass (in case of discrete random variables)

If the random variables X₁,X₂,...,X_n $ \epsilon $ R are Independent Identically Distributed (I.I.D.) then, L(θ) = _i=1 ∏ ⁿ f(x_i | θ) . We need to find the value $ \hat{\theta} $ $ \epsilon $ θ that maximizes this function.

In the course, Purdue ECE 662, Pattern Recognition and Decision Taking Processes, we have already looked at the Maximum Likelihood Estimates for for Normally distributed random variables and found that to be:

$ \widehat{\mu} = \frac{\sum_{i=1}^{n}x_{i}}{n} $ $ \hat{\sigma{}^{2}} = \frac{1}{n-1}\sum_{i=1}^{n}(x_{i}-\widehat{\mu})^{2} $

where, X_i's are the Normal (Gaussian) Random Variables $ \epsilon $ R , 'n' is the number of samples, and $ \widehat{\mu} $ and $ \hat{\sigma{}^{2}} $ are the estimated Mean and estimated Variance.

Maximum Likelihood Estimate (MLE) for :-

1. Exponential Distribution

Let X₁,X₂,...,X_n $ \epsilon $ R be a random sample from the exponential distribution with p.d.f.

f(x)=(1|θ) * exp(−x|θ)

The likelihood function L(θ) is a function of x₁, x₂, x₃,...,x_n, given by:

L(θ)=(1|θ) * exp(−x₁|θ) * (1|θ) * exp(−x₂|θ) * ... * (1|θ) * exp(−x_n|θ)

L(θ)= (1|θⁿ) * exp( _i=1∑ⁿ -x_i|θ)

We need to maximize L(θ) . The logarithm of this function will be easier to maximize.

ln [L(θ)] = -n . ln(θ) - (1|θ) _i=1∑ⁿ x_i

Setting its derivative with respect to the parameter (θ) to zero, we have:

(d|dθ) ln[L(θ)] = (-n|θ) + _i=1∑ⁿ (-x_i| θ²) = 0

which implies that

$ \hat{\theta} = \frac{\sum_{i=1}^{n}x_{i}}{n} = \overline{X} $ = Mean of x₁, x₂, x₃,...,x_n

This is the maximum likelihood estimate

2. Geometric Distribution

Let X₁,X₂,...,X_n $ \epsilon $ R be random samples from the geometric distribution with p.d.f.

f(x) = (1−p)^x-1.p ; where x=1,2,3,.... and $ 0 \leq p \leq 1 $

The likelihood function is given by:

L(p) = (1−p)^x₁-1.p.(1−p)^x₂-1.p.(1−)^x₃-1.p...(1−p)^x_n-1.p

L(p) = pⁿ.(1-p)^{_i=1∑n x_i-n}

The likelihood function is a function of x₁,x₂,...,x_n

The log-likelihood is:

ln L(p)=n . ln(p) + _i=1∑ⁿ x_i-n . ln(1-p)

Setting its derivative with respect to the parameter (p) to zero, we have:

(d|dp) ln. L(p) = (n|p) - (_i=1∑ⁿ x_i-n|(1-p)) = 0

which implies that

$ \hat{p} = \frac{n}{\sum_{i=1}^{n}x_{i}} = \frac{1}{\overline{X}} $

This is the maximum likelihood estimate.

This is intuitively correct as well. Geometric Distribution is used to model a random variable X which is the number of trials before the first success is obtained. So, for random variables X₁,X₂,...,X_n, these contain n successes in X₁+ X₂ +...+ X_n trials.

Intuitively, the estimate of 'p' is the number of successes divided by the total number of trials. This matches with the maximum likelihood estimate of the parameter 'p' got for Geometric Distribution.

3. Binomial Distribution

Let X₁,X₂,...,X_N $ \epsilon $ R be samples obtained from a Binomially Distribution.

Binomial Distribution is used to model 'x' successes in 'n' Bernoulli trials. Its p.d.f. is given by:

$ f(x) = \frac{n!}{x!(n-x)!} p^{^{x}} (1-p)^{n-x} $

The likelihood function L(p) is given by:

$ L(p) = \prod_{i=1}^{n} f(x_{i}) = \prod_{i=1}^{N} \frac{n!}{x_{i}!(n-xi)!} p^{x_{i}} (1-p)^{n-x_{i}} $

The log-likelihood is:

$ lnL(p) = \sum_{i=1}^{N} ln (n!) - \sum_{i=1}^{N} ln (x_{i}!) - \sum_{i=1}^{N} ln(n-x_{i}!) + \sum_{i=1}^{N} xi.ln(p) + (n- \sum_{i=1}^{N} xi) . ln(1-p) $

Setting its derivative with respect to p to zero,

$ \frac{d}{dp} lnL(p) = \frac{1}{p}. \sum_{i=1}^{N} xi - \frac{1}{1-p} \sum_{i=1}^{N} (n - xi) = 0 $

which implies,

$ \frac{1}{p}. \sum_{i=1}^{N} xi = (\frac{1}{1-p})( N.n - \sum_{i=1}^{N} xi) $

giving,

$ \hat{p} = \frac{1}{N} (\frac{\sum_{i=1}^{N}x_{i}}{n}) = \frac{1}{N} (\frac{X1}{n} + \frac{X2}{n} + ... + \frac{XN}{n}) $

which is the maximum likelihood estimate.

This is intuitively correct too, as this is the average of the ratio $ \frac{X_{i}}{n} $ for each X_i, which is intuitively average of 'p' for each X_i

4.Poisson Distribution

Let X₁,X₂,...,X_n $ \epsilon $ R be a random sample from a Poisson distribution

The p.d.f. of a Poisson Distribution is :

$ f(x) = \frac{\lambda^{x}e^{-\lambda}}{x!} $ ; where x =0,1,2,...

The likelihood function is:

$ L(\lambda ) = \prod_{i=1}^{n} \frac{\lambda^{x_{i}}e^{-\lambda}}{x_{i}!} = e^{-\lambda n} \frac{\lambda^{\sum_{i=1}^{n}x_{i}}}{\prod_{i=1}^{n}x_{i}} $

The log-likelihood is:

$ ln L(\lambda) = - \lambda n + \sum_{i=1}^{n} xi . ln(\lambda) - ln( \prod_{i=1}^{n}xi) $

Setting its derivative with respect to $ \lambda $ to zero, we have:

$ \frac{d}{d\lambda}ln L(\lambda) = -n + \sum_{i=1}^{n}xi .\frac{1}{\lambda} = 0 $

giving,

$ \widehat{\lambda} = \frac{\sum_{i=1}^{n}x_{i}}{n} = \overline{X} $

which is the maximum likelihood estimate

5.Uniform Distribution

For Uniformly Distributed random variables X₁,X₂,...,X_n $ \epsilon $ R, the p.d.f is given by:

f(x_i) = $ \frac{1}{\theta} $ ; if $ 0 \leq xi \leq \theta $

f(x) = 0 ; otherwise

If the uniformly distributed random variables are arranged in the following order

$ 0 \leq X1\leq X2 \leq X3 ... \leq Xn \leq \theta $,

The likelihood function is given by:

$ L(\theta) = \prod_{i=1}^{n} f(xi) = \prod_{i=1}^{n} \frac{1}{\theta} = \theta{}^{-n} $

The log-likelihood is:

$ ln L(\theta) = -n ln (\theta) $

Setting its derivative with respect to parameter $ \theta $ to zero, we get:

$ \frac{d}{d\theta} ln L(\theta) = \frac{-n}{\theta} $

which is < 0 for $ \theta $ > 0

Hence, L($ \theta $) is a decreasing function and it is maximized at $ \theta $ = x_n

The maximum likelihood estimate is thus,

$ \hat{\theta} $ = X_n

Summary

Using the usual notations and symbols,

1) Normal Distribution:

$ f(x,\mu,\sigma) = \frac{1}{\sigma \sqrt(2\pi)} exp(-\frac{1}{2}(\frac{x-\mu}{\sigma})^{2}) $

X₁,X₂,...,X_n $ \epsilon $ R

$ \widehat{\mu} = \frac{\sum_{i=1}^{n}x_{i}}{n} $ $ \hat{\sigma{}^{2}} = \frac{1}{n-1}\sum_{i=1}^{n}(x_{i}-\widehat{\mu})^{2} $

2) Exponential Distribution:

f(x,$ \lambda $)=(1|$ \lambda $)*exp(−x|$ \lambda $) ; X₁,X₂,...,X_n $ \epsilon $ R

$ \widehat{\lambda} = \frac{\sum_{i=1}^{n}x_{i}}{n} = \overline{X} $

3) Geometric Distribution:

f(x,p) = (1−p)^x-1.p ; X₁,X₂,...,X_n $ \epsilon $ R

$ \hat{p} = \frac{n}{\sum_{i=1}^{n}x_{i}} = \frac{1}{\overline{X}} $

4) Binomial Distribution:

$ f(x,p) = \frac{n!}{x!(n-x)!} p^{^{x}} (1-p)^{n-x} $

X₁,X₂,...,X_n $ \epsilon $ R

$ \hat{p} = \frac{1}{N} (\frac{\sum_{i=1}^{N}x_{i}}{n}) = \frac{1}{N} (\frac{X1}{n} + \frac{X2}{n} + ... + \frac{XN}{n}) $

5) Poisson Distribution:

$ f(x,\lambda) = \frac{\lambda^{x}e^{-\lambda}}{x!} $

X₁,X₂,...,X_n $ \epsilon $ R

$ \widehat{\lambda} = \frac{\sum_{i=1}^{n}x_{i}}{n} = \overline{X} $

6) Uniform Distribution:

For X₁,X₂,...,X_n $ \epsilon $ R

f(x_i) = $ \frac{1}{\theta} $ ; if $ 0 \leq xi \leq \theta $

f(x) = 0 ; otherwise

$ \hat{\theta} $ = X_n

References

1) A module on Maximum Likelihood Estimation - Examples by Ewa Paszek

2) Lecture on Maximum Likelihood Estimation by Dr. David Levin, Assistant Professor, Univeristy of Utah

3) Partially based on Dr. Mireille Boutin lecture notes for Purdue ECE 662 - Pattern Recognition and Decision Making Processes

Questions and comments

If you have any questions, comments, etc. please post them on this page.