| Line 15: | Line 15: | ||
---- | ---- | ||
---- | ---- | ||
| − | |||
| − | + | Content: | |
| + | ---- | ||
| + | |||
| + | 1 Bayes rule for Gaussian data | ||
| + | |||
| + | ---- | ||
| + | 2 Procedure | ||
| + | |||
| + | ---- | ||
| + | 3 Parameter estimation | ||
| + | |||
| + | Given a data set | ||
| + | <math>\mathbf{X}=(\mathbf{x}_1,...,\mathbf{x}_N)^T</math> in which the observations <math>\{{\mathbf{x}_n}\}</math> are assumed to be drawn independently from a multivariate Gaussian distribution (D dimension), we can estimate the parameters of the distribution by maximum likelihood. The log likelihood function is given by | ||
| + | |||
| + | <center><math> | ||
| + | ln p(\mathbf{x}|\mathbf{\mu, \Sigma}) = -\frac{ND}{2}ln(2\pi)-\frac{N}{2}ln(|\mathbf{\Sigma}|)-{\frac{1}{2}\sum\limits_{n=1}^{N}({\mathbf{x}}_n - \mathbf{\mu})^T\mathbf{\Sigma}^{-1}({\mathbf{x}}_n - \mathbf{\mu})}. | ||
| + | </math></center> | ||
| + | |||
| + | By simple rearrangement, we see that the likelihood function depends on the data set only through the two quantities | ||
| + | <center><math> | ||
| + | \sum\limits_{n=1}^{N}\mathbf{x}_n, \sum\limits_{n=1}^{N}{\mathbf{x}}_n{\mathbf{x}}_n^T. | ||
| + | </math></center> | ||
| + | |||
| + | These are the sufficient statistics for the Gaussian distribution. The derivative of the log likelihood with respect to <math>\mathbf{\mu}</math> is | ||
| + | <center><math> | ||
| + | \frac{\partial}{\partial\mathbf{\mu}} ln p(\mathbf{x}|\mathbf{\mu, \Sigma})= \sum\limits_{n=1}^{N}\mathbf{\Sigma}^{-1}(\mathbf{x}_n - \mathbf{\mu}) | ||
| + | </math></center> | ||
| + | |||
| + | and setting this derivative to zero, we obtain the solution for the maximum likelihood estimate of the mean | ||
| + | <center><math> | ||
| + | {\mathbf{\mu}}_{ML}=\frac{1}{N} \sum\limits_{n=1}^{N} {\mathbf{x}}_n. | ||
| + | </math></center> | ||
| + | |||
| + | Use similar method by setting the derivative of the log likelihood with respect to <math>\mathbf{\Sigma}</math> to zero, we obtain | ||
| + | and setting this derivative to zero, we obtain the solution for the maximum likelihood estimate of the mean | ||
| + | <center><math> | ||
| + | {\mathbf{\Sigma}}_{ML}=\frac{1}{N} \sum\limits_{n=1}^{N}({\mathbf{x}}_n - {\mathbf{\mu}}_{ML})({\mathbf{x}}_n - {\mathbf{\mu}}_{ML})^T. | ||
| + | </math></center> | ||
| + | |||
| + | |||
| + | |||
| + | ---- | ||
| + | 4 Example | ||
| + | |||
| + | ---- | ||
| + | 5 Conclusion | ||
---- | ---- | ||
Revision as of 10:50, 30 April 2014
Bayes rule in practice: definition and parameter estimation
A slecture by ECE student Chuohao Tang
Partly based on the ECE662 Spring 2014 lecture material of Prof. Mireille Boutin.
Content:
1 Bayes rule for Gaussian data
2 Procedure
3 Parameter estimation
Given a data set $ \mathbf{X}=(\mathbf{x}_1,...,\mathbf{x}_N)^T $ in which the observations $ \{{\mathbf{x}_n}\} $ are assumed to be drawn independently from a multivariate Gaussian distribution (D dimension), we can estimate the parameters of the distribution by maximum likelihood. The log likelihood function is given by
By simple rearrangement, we see that the likelihood function depends on the data set only through the two quantities
These are the sufficient statistics for the Gaussian distribution. The derivative of the log likelihood with respect to $ \mathbf{\mu} $ is
and setting this derivative to zero, we obtain the solution for the maximum likelihood estimate of the mean
Use similar method by setting the derivative of the log likelihood with respect to $ \mathbf{\Sigma} $ to zero, we obtain and setting this derivative to zero, we obtain the solution for the maximum likelihood estimate of the mean
4 Example
5 Conclusion
Questions and comments
If you have any questions, comments, etc. please post them on this page.
