Revision as of 08:26, 19 November 2010

5 Exams

Example. Addition of two independent Poisson random variables

Let $ \mathbf{Z}=\mathbf{X}+\mathbf{Y} $ where $ \mathbf{X} $ and $ \mathbf{Y} $ are independent Poisson random variables with means $ \lambda $ and $ \mu $ , respectively.

(a)

Find the pmf of $ \mathbf{Z} $ .

According to the characteristic function of Poisson random variable

$ \Phi_{\mathbf{X}}(\omega)=e^{-\lambda\left(1-e^{i\omega}\right)},\Phi_{\mathbf{Y}}(\omega)=e^{-\mu\left(1-e^{i\omega}\right)} $.

$ \mathbf{X} $ and $ \mathbf{Y} $ are independent $ \Longrightarrow \mathbf{X} $ and $ \mathbf{Y} $ are uncorrelated $ \Longrightarrow e^{i\omega\mathbf{X}} $ and $ e^{i\omega\mathbf{Y}} $ are uncorrelated.

$ \Phi_{\mathbf{Z}}(\omega)=E\left[e^{i\omega\mathbf{Z}}\right]=E\left[e^{i\omega\left(\mathbf{X}+\mathbf{Y}\right)}\right]=E\left[e^{i\omega\mathbf{X}}e^{i\omega\mathbf{Y}}\right]=E\left[e^{i\omega\mathbf{X}}\right]\cdot E\left[e^{i\omega\mathbf{Y}}\right] $$ =e^{-\lambda\left(1-e^{i\omega}\right)}\cdot e^{-\mu\left(1-e^{i\omega}\right)}=e^{-\left(\lambda+\mu\right)\left(1-e^{i\omega}\right).} $

Now, we know that \mathbf{Z} is a Poisson random variable with mean $ \lambda+\mu $ .

$ \therefore p_{\mathbf{Z}}(k)=\frac{e^{-\left(\lambda+\mu\right)}\left(\lambda+\mu\right)^{k}}{k!}. $

(b)

Show that the conditional pmf of $ \mathbf{X} $ conditioned on the event $ \left\{ \mathbf{Z}=n\right\} $ is binomially distributed, and determine the parameters of binomial distribution ($ n $ and $ p $ ).

$ P_{\mathbf{X}}\left(\mathbf{X}|\left\{ \mathbf{Z}=n\right\} \right)=P\left(\left\{ \mathbf{X}=k\right\} |\left\{ \mathbf{Z}=n\right\} \right)=\frac{P\left(\left\{ \mathbf{X}=k\right\} \cap\left\{ \mathbf{Z}=n\right\} \right)}{P\left(\left\{ \mathbf{Z}=n\right\} \right)}=\frac{P\left(\left\{ \mathbf{X}=k\right\} \cap\left\{ \mathbf{Y}=n-k\right\} \right)}{P\left(\left\{ \mathbf{Z}=n\right\} \right)} $$ =\frac{\frac{e^{-\lambda}\lambda^{k}}{k!}\cdot\frac{e^{-\mu}\mu^{n-k}}{\left(n-k\right)!}}{\frac{e^{-\left(\lambda+\mu\right)}\left(\lambda+\mu\right)^{n}}{n!}}=\left(\frac{n!}{k!\left(n-k\right)!}\right)\left(\frac{\lambda}{\lambda+\mu}\right)^{k}\left(\frac{\mu}{\lambda+\mu}\right)^{n-k} $$ =\left(\begin{array}{c} n\\ k \end{array}\right)\left(\frac{\lambda}{\lambda+\mu}\right)^{k}\left(\frac{\mu}{\lambda+\mu}\right)^{n-k}\;,\; k=0,\,1,\,2,\,\cdots $

This is a binomial pmf $ b(n,p) $ with parameters $ n $ and $ p=\frac{\lambda}{\lambda+\mu} $ .

Example. Addition of two independent Gaussian random variables

$ \mathbf{X}\sim\mathcal{N}\left(0,\sigma_{\mathbf{X}}^{2}\right),\;\mathbf{N}\sim\mathcal{N}\left(0,\sigma_{\mathbf{N}}^{2}\right),\;\mathbf{Y}=\mathbf{X}+\mathbf{N}. $

(a)

Correlation coefficient between $ \mathbf{X} $ and $ \mathbf{Y} $ .

$ \sigma_{\mathbf{Y}}=\sqrt{\sigma_{\mathbf{X}}^{2}+2r_{\mathbf{XN}}\sigma_{\mathbf{X}}\sigma_{\mathbf{N}}+\sigma_{\mathbf{N}}^{2}}=\sqrt{\sigma_{\mathbf{X}}^{2}+\sigma_{\mathbf{N}}^{2}} $

because $ \mathbf{X} $ and $ \mathbf{N} $ are independnet $ \Longrightarrow $ uncorrelated $ \Longrightarrow r_{\mathbf{XN}}=0 $ .

$ r_{\mathbf{XY}}=\frac{\text{cov}(\mathbf{X},\mathbf{Y})}{\sigma_{\mathbf{X}}\sigma_{\mathbf{Y}}}=\frac{E\left[\mathbf{XY}\right]-E\left[\mathbf{X}\right]E\left[\mathbf{Y}\right]}{\sigma_{\mathbf{X}}\sqrt{\sigma_{\mathbf{X}}^{2}+\sigma_{\mathbf{N}}^{2}}}=\frac{E\left[\mathbf{X}\left(\mathbf{X}+\mathbf{N}\right)\right]}{\sigma_{\mathbf{X}}\sqrt{\sigma_{\mathbf{X}}^{2}+\sigma_{\mathbf{N}}^{2}}}=\frac{E\left[\mathbf{X}^{2}\right]+E\left[\mathbf{XN}\right]}{\sigma_{\mathbf{X}}\sqrt{\sigma_{\mathbf{X}}^{2}+\sigma_{\mathbf{N}}^{2}}} $$ =\frac{\sigma_{\mathbf{X}}^{2}+E\left[\mathbf{X}\right]E\left[\mathbf{N}\right]}{\sigma_{\mathbf{X}}\sqrt{\sigma_{\mathbf{X}}^{2}+\sigma_{\mathbf{N}}^{2}}}=\frac{\sigma_{\mathbf{X}}}{\sqrt{\sigma_{\mathbf{X}}^{2}+\sigma_{\mathbf{N}}^{2}}}\qquad\because E\left[\mathbf{X}\right]=0. $

(b)

Conditional pmf of $ \mathbf{X} $ conditioned on the event $ \left\{ \mathbf{Y}=y\right\} $ .

$ f_{\mathbf{X}}\left(x|\left\{ \mathbf{Y}=y\right\} \right)=\frac{f_{\mathbf{XY}}\left(x,y\right)}{f_{\mathbf{Y}}(y)}=\frac{\frac{1}{2\pi\sigma_{\mathbf{X}}\sigma_{\mathbf{Y}}\sqrt{1-r^{2}}}\exp\left\{ \frac{-1}{2\left(1-r^{2}\right)}\left[\frac{x^{2}}{\sigma_{\mathbf{X}}^{2}}-\frac{2rxy}{\sigma_{\mathbf{X}}\sigma_{\mathbf{Y}}}+\frac{y^{2}}{\sigma_{\mathbf{Y}}^{2}}\right]\right\} }{\frac{1}{\sqrt{2\pi}\sigma_{Y}}\exp\left\{ \frac{-y^{2}}{2\sigma_{Y}^{2}}\right\} } $$ =\frac{1}{\sqrt{2\pi}\sigma_{\mathbf{X}}\sqrt{1-r^{2}}}\exp\left\{ \frac{-1}{2\left(1-r^{2}\right)}\left[\frac{x^{2}}{\sigma_{\mathbf{X}}^{2}}-\frac{2rxy}{\sigma_{\mathbf{X}}\sigma_{\mathbf{Y}}}+\frac{y^{2}}{\sigma_{\mathbf{Y}}^{2}}-\frac{\left(1-r^{2}\right)y^{2}}{\sigma_{\mathbf{Y}}^{2}}\right]\right\} $ $ =\frac{1}{\sqrt{2\pi}\sigma_{\mathbf{X}}\sqrt{1-r^{2}}}\exp\left\{ \frac{-1}{2\left(1-r^{2}\right)}\left[\frac{x^{2}}{\sigma_{\mathbf{X}}^{2}}-\frac{2rxy}{\sigma_{\mathbf{X}}\sigma_{\mathbf{Y}}}+\frac{r^{2}y^{2}}{\sigma_{\mathbf{Y}}^{2}}\right]\right\} $ $ =\frac{1}{\sqrt{2\pi}\sigma_{\mathbf{X}}\sqrt{1-r^{2}}}\exp\left\{ \frac{-1}{2\left(1-r^{2}\right)\sigma_{\mathbf{X}}^{2}}\left[x^{2}-\frac{2r\sigma_{\mathbf{X}}xy}{\sigma_{\mathbf{Y}}}+\frac{r^{2}\sigma_{\mathbf{X}}^{2}y^{2}}{\sigma_{\mathbf{Y}}^{2}}\right]\right\} $ $ =\frac{1}{\sqrt{2\pi}\sigma_{\mathbf{X}}\sqrt{1-r^{2}}}\exp\left\{ \frac{-1}{2\left(1-r^{2}\right)\sigma_{\mathbf{X}}^{2}}\left(x-\frac{r\sigma_{\mathbf{X}}y}{\sigma_{\mathbf{Y}}}\right)^{2}\right\} $

Noting that $ \sqrt{1-r^{2}}=\sigma_{\mathbf{X}}\sqrt{1-\left(\frac{\sigma_{\mathbf{X}}}{\sqrt{\sigma_{\mathbf{X}}^{2}+\sigma_{\mathbf{Y}}^{2}}}\right)^{2}}=\sqrt{1-\frac{\sigma_{\mathbf{X}}^{2}}{\sigma_{\mathbf{X}}^{2}+\sigma_{\mathbf{Y}}^{2}}}=\sqrt{\frac{\sigma_{\mathbf{X}}^{2}+\sigma_{\mathbf{N}}^{2}-\sigma_{\mathbf{N}}^{2}}{\sigma_{\mathbf{X}}^{2}+\sigma_{\mathbf{N}}^{2}}}=\frac{\sigma_{\mathbf{N}}}{\sqrt{\sigma_{\mathbf{X}}^{2}+\sigma_{\mathbf{N}}^{2}}} $ and

$ r\cdot\frac{\sigma_{\mathbf{X}}}{\sigma_{\mathbf{Y}}}=\frac{\sigma_{X}}{\sqrt{\sigma_{\mathbf{X}}^{2}+\sigma_{\mathbf{N}}^{2}}}\cdot\frac{\sigma_{\mathbf{X}}}{\sqrt{\sigma_{\mathbf{X}}^{2}+\sigma_{\mathbf{N}}^{2}}}=\frac{\sigma_{\mathbf{X}}^{2}}{\sigma_{\mathbf{X}}^{2}+\sigma_{\mathbf{N}}^{2}}. $ $ \therefore f_{\mathbf{X}}\left(x|\left\{ \mathbf{Y}=y\right\} \right)=\frac{1}{\sqrt{2\pi}\cdot\frac{\sigma_{\mathbf{X}}\sigma_{\mathbf{N}}}{\sqrt{\sigma_{\mathbf{X}}^{2}+\mathbf{\sigma}_{\mathbf{N}}^{\mathbf{2}}}}}\exp\left\{ \frac{-1}{2\frac{\sigma_{\mathbf{X}}^{2}\sigma_{\mathbf{N}}^{2}}{\sigma_{\mathbf{X}}^{2}+\sigma_{\mathbf{N}}^{2}}}\left(x-\frac{\sigma_{\mathbf{X}}^{2}}{\sigma_{\mathbf{X}}^{2}+\sigma_{\mathbf{N}}^{2}}\cdot y\right)^{2}\right\} $

(c)

What kind of pdf is the pdf you determined in part (b)? What is the mean and variance of a random variable with this pdf?

This is a Gaussian pdf with mean $ \frac{\sigma_{\mathbf{X}}^{2}}{\sigma_{\mathbf{X}}^{2}+\sigma_{\mathbf{N}}^{2}}\cdot y $ and variance $ \frac{\sigma_{\mathbf{X}}^{2}\sigma_{\mathbf{N}}^{2}}{\sigma_{\mathbf{X}}^{2}+\sigma_{\mathbf{N}}^{2}} $ .

(d)

What is the minimum mean-square estimate of $ \mathbf{X} $ given that $ \left\{ \mathbf{Y}=y\right\} $  ?

The minimum mean-square error estimate of $ \mathbf{X} $ given $ \mathbf{Y}=y $ is

$ \hat{x}_{MMS}(y)=E\left[\mathbf{X}|\left\{ \mathbf{Y}=y\right\} \right]=\int_{-\infty}^{\infty}x\cdot f_{\mathbf{X}}\left(x|\left\{ \mathbf{Y}=y\right\} \right)dx=\frac{\sigma_{\mathbf{X}}^{2}}{\sigma_{\mathbf{X}}^{2}+\sigma_{\mathbf{Y}}^{2}}\cdot y $ from part (b).

(e)

What is the maximum a posteriori estimate of $ \mathbf{X} $ given that $ \left\{ \mathbf{Y}=y\right\} $  ?

$ \hat{x}_{MAP}(y)=\arg\max_{x\in\mathbf{R}}\left\{ f_{\mathbf{X}}\left(x|\left\{ Y=y\right\} \right)\right\} =\frac{\sigma_{\mathbf{X}}^{2}}{\sigma_{\mathbf{X}}^{2}+\sigma_{\mathbf{Y}}^{2}}\cdot y $

as a Gaussian pdf takes on its maximum value at its mean.

(f)

Given that I observe $ \mathbf{Y}=y $ , what is $ E\left[\mathbf{X}|\left\{ \mathbf{Y}=y\right\} \right] $ ?

$ E\left[\mathbf{X}|\left\{ \mathbf{Y}=y\right\} \right]=\frac{\sigma_{\mathbf{X}}^{2}}{\sigma_{\mathbf{X}}^{2}+\sigma_{\mathbf{Y}}^{2}}\cdot y $ from part (d).

Example. Addition of two jointly distributed Gaussian random variables

Let $ \mathbf{X} $ and $ \mathbf{Y} $ be two jointly distributed Gaussian random variables. Assume $ \mathbf{X} $ has mean $ \mu_{\mathbf{X}} $ and variance $ \sigma_{\mathbf{X}}^{2} , \mathbf{Y} $ has mean $ \mu_{\mathbf{Y}} $ and variance $ \sigma_{\mathbf{Y}}^{2} $ , and that the correlation coefficient between $ \mathbf{X} $ and $ \mathbf{Y} $ is $ r $ . Define a new random variable $ \mathbf{Z}=\mathbf{X}+\mathbf{Y} $ .

(a)

Show that $ \mathbf{Z} $ is a Gaussian random variable.

If $ \mathbf{Z} $ is a Guassian random variable, then it has a characteristic function of the form

$ \Phi_{\mathbf{Z}}\left(\omega\right)=e^{i\mu_{\mathbf{Z}}\omega}e^{-\frac{1}{2}\sigma_{\mathbf{Z}}^{2}\omega^{2}}. $

$ \Phi_{\mathbf{Z}}\left(\omega\right) $

where $ \Phi_{\mathbf{XY}}\left(\omega_{1},\omega_{2}\right) $ is the joint characteristic function of $ \mathbf{X} $ and $ \mathbf{Y} $ , defined as

$ \Phi_{\mathbf{XY}}\left(\omega_{1},\omega_{2}\right)=E\left[e^{i\left(\mathbf{\omega_{1}X}+\omega_{2}\mathbf{Y}\right)}\right]. $

Now because $ \mathbf{X} $ and $ \mathbf{Y} $ are jointly Gaussian with the given parameters, we know that

$ \Phi_{\mathbf{XY}}\left(\omega_{1},\omega_{2}\right)=e^{i\left(\mu_{X}\omega_{1}+\mu_{Y}\omega_{2}\right)}e^{-\frac{1}{2}\left(\sigma_{X}^{2}\omega_{1}^{2}+2r\sigma_{X}\sigma_{Y}\omega_{1}\omega_{2}+\sigma_{Y}^{2}\omega_{2}^{2}\right)}. $

Thus,

$ \Phi_{\mathbf{Z}}\left(\omega\right)=\Phi_{\mathbf{XY}}\left(\omega,\omega\right)=e^{i\left(\mu_{X}\omega+\mu_{Y}\omega\right)}e^{-\frac{1}{2}\left(\sigma_{X}^{2}\omega^{2}+2r\sigma_{X}\sigma_{Y}\omega^{2}+\sigma_{Y}^{2}\omega^{2}\right)} $$ =e^{i\left(\mu_{X}+\mu_{Y}\right)\omega}e^{-\frac{1}{2}\left(\sigma_{X}^{2}+2r\sigma_{X}\sigma_{Y}+\sigma_{Y}^{2}\right)\omega^{2}}=e^{i\mu_{Z}\omega}e^{-\frac{1}{2}\sigma_{Z}^{2}\omega^{2}} $

where $ \mu_{Z}=\mu_{X}+\mu_{Y} $ and $ \sigma_{Z}^{2}=\sigma_{X}^{2}+2r\sigma_{X}\sigma_{Y}+\sigma_{Y}^{2} $ .

$ \mathbf{Z} $ is a Gaussian random variable with E\left[\mathbf{Z}\right]=\mu_{X}+\mu_{Y} and Var\left[\mathbf{Z}\right]=\sigma_{X}^{2}+2r\sigma_{X}\sigma_{Y}+\sigma_{Y}^{2} .

(b)

Find the variance of $ \mathbf{Z} $ .

As show in part (a) $ Var\left[\mathbf{Z}\right]=\sigma_{\mathbf{X}}^{2}+2r\sigma_{\mathbf{X}}\sigma_{\mathbf{Y}}+\sigma_{\mathbf{Y}}^{2} $ .

Example. Two jointly distributed random variables

Two joinly distributed random variables $ \mathbf{X} $ and $ \mathbf{Y} $ have joint pdf

$ f_{\mathbf{XY}}\left(x,y\right)=\begin{cases} \begin{array}{ll} c ,\text{ for }x\geq0,y\geq0,\textrm{ and }x+y\leq1\\ 0 ,\text{ elsewhere.} \end{array}\end{cases} $

(a)

Find the constant c such that $ f_{\mathbf{XY}}(x,y) $ is a valid pdf.

File:002.eps

$ \iint_{\mathbf{R}^{2}}f_{\mathbf{XY}}\left(x,y\right)=c\cdot Area=1 $ where $ Area=\frac{1}{2} $ .

$ \therefore c=2 $

(b)

Find the conditional density of \mathbf{Y} conditioned on \mathbf{X}=x .

f_{\mathbf{Y}}\left(y|\left\{ \mathbf{X}=x\right\} \right)=\frac{f_{\mathbf{XY}}\left(x,y\right)}{f_{\mathbf{X}}(x)}.

f_{\mathbf{X}}(x)=\int_{-\infty}^{\infty}f_{\mathbf{XY}}\left(x,y\right)dy=\int_{0}^{1-x}2dy=2\left(1-x\right)\cdot\mathbf{1}_{\left[0,1\right]}(x).

f_{\mathbf{Y}}\left(y|\left\{ \mathbf{X}=x\right\} \right)=\frac{f_{\mathbf{XY}}\left(x,y\right)}{f_{\mathbf{X}}(x)}=\frac{2}{2\left(1-x\right)}=\frac{1}{1-x}\textrm{ where }0\leq y\leq1-x\Longrightarrow\frac{1}{1-x}\cdot\mathbf{1}_{\left[0,1-x\right]}\left(y\right).

(c)

Find the minimum mean-square error estimator \hat{y}_{MMS}\left(x\right) of \mathbf{Y} given that \mathbf{X}=x .

\hat{y}_{MMS}\left(x\right)=E\left[\mathbf{Y}|\left\{ \mathbf{X}=x\right\} \right]=\int_{\mathbf{R}}yf_{\mathbf{Y}}\left(y|\left\{ \mathbf{X}=x\right\} \right)dy=\int_{0}^{1-x}\frac{y}{1-x}dy=\frac{y^{2}}{2\left(1-x\right)}\biggl|_{0}^{1-x}=\frac{1-x}{2}.

(d)

Find a maximum aposteriori probability estimator.

\hat{y}_{MAP}\left(x\right)=\arg\max_{y}\left\{ f_{Y}\left(y|\left\{ \mathbf{X}=x\right\} \right)\right\} but f_{Y}\left(y|\left\{ \mathbf{X}=x\right\} \right)=\frac{1}{1-x}\cdot\mathbf{1}_{\left[0,1-x\right]}\left(y\right) . Any \hat{y}\in\left[0,1-x\right] is a MAP estimator. The MAP estimator is NOT unique.

Example. Two jointly distributed independent random variables

Let \mathbf{X} and \mathbf{Y} be two jointly distributed, independent random variables. The pdf of \mathbf{X} is

f_{\mathbf{X}}\left(x\right)=xe^{-x^{2}/2}\cdot\mathbf{1}_{\left[0,\infty\right)}\left(x\right), and \mathbf{Y} is a Gaussian random variable with mean 0 and variance 1 . Let \mathbf{U} and \mathbf{V} be two new random variables defined as \mathbf{U}=\sqrt{\mathbf{X}^{2}+\mathbf{Y}^{2}} and \mathbf{V}=\lambda\mathbf{Y}/\mathbf{X} where \lambda is a positive real number.

(a)

Find the joint pdf of \mathbf{U} and \mathbf{V} . (Direct pdf method)

f_{\mathbf{UV}}\left(u,v\right)=f_{\mathbf{XY}}\left(x\left(u,v\right),y\left(u,v\right)\right)\left|\frac{\partial\left(x,y\right)}{\partial\left(u,v\right)}\right|

Solving for x and y in terms of u and v , we have u^{2}=x^{2}+y^{2} and v^{2}=\frac{\lambda^{2}y^{2}}{x^{2}}\Longrightarrow y^{2}=\frac{v^{2}x^{2}}{\lambda^{2}} .

Now, u^{2}=x^{2}+y^{2}=x^{2}+\frac{v^{2}x^{2}}{\lambda^{2}}=x^{2}\left(1+v^{2}/\lambda^{2}\right)\Longrightarrow x=\frac{u}{\sqrt{1+v^{2}/\lambda^{2}}}\Longrightarrow x\left(u,v\right)=\frac{u}{\sqrt{1+v^{2}/\lambda^{2}}} .

Thus, y=\frac{vx}{\lambda}=\frac{vu}{\lambda\sqrt{1+v^{2}/\lambda^{2}}}\Longrightarrow y\left(u,v\right)=\frac{vu}{\lambda\sqrt{1+v^{2}/\lambda^{2}}} .

Computing the Jacobian.

\frac{\partial\left(x,y\right)}{\partial\left(u,v\right)}

Because \mathbf{X} and \mathbf{Y} are statistically independent

f_{\mathbf{XY}}\left(x,y\right)=f_{\mathbf{X}}\left(x\right)f_{\mathbf{Y}}\left(y\right)=xe^{-x^{2}/2}\cdot\mathbf{1}_{\left[0,\infty\right)}\left(x\right)\cdot\frac{1}{\sqrt{2\pi}}e^{-y^{2}/2}=\frac{x}{\sqrt{2\pi}}e^{-\left(x^{2}+y^{2}\right)/2}\cdot\mathbf{1}_{\left[0,\infty\right)}\left(x\right).

Substituting these quantities, we get

f_{\mathbf{UV}}\left(u,v\right)

(b)

Are \mathbf{U} and \mathbf{V} statistically independent? Justify your answer.

\mathbf{U} and \mathbf{V} are statistically independent iff f_{\mathbf{UV}}\left(u,v\right)=f_{\mathbf{U}}\left(u\right)f_{\mathbf{V}}\left(v\right) .

Now from part (a), we see thatf_{\mathbf{UV}}\left(u,v\right)=c_{1}g_{1}\left(u\right)\cdot c_{2}g_{2}\left(v\right) where g_{1}\left(u\right)=u^{2}e^{-u^{2}/2}\cdot\mathbf{1}_{\left[0,\infty\right)}\left(u\right) and g_{2}\left(v\right)=\frac{1}{\left(\lambda^{2}+v^{2}\right)^{\frac{3}{2}}} with c_{1} and c_{2} selected such that f_{\mathbf{U}}\left(u\right)=c_{1}g_{1}\left(u\right) and f_{\mathbf{V}}\left(v\right)=c_{2}g_{2}\left(v\right) are both valid pdfs.

\therefore \mathbf{U} and \mathbf{V} are statistically independent.

Example. Two jointly distributed random variables (Joint characteristic function)

Let \mathbf{X} and \mathbf{Y} be tweo jointly distributed random variables having joint characteristic function

\Phi_{\mathbf{XY}}\left(\omega_{1},\omega_{2}\right)=\frac{1}{\left(1-i\omega_{1}\right)\left(1-i\omega_{2}\right)}.

(a)

Calculate E\left[\mathbf{X}\right] .

\Phi_{\mathbf{X}}\left(\omega\right)=\Phi_{\mathbf{XY}}\left(\omega,0\right)=\frac{1}{1-i\omega}=\left(1-i\omega\right)^{-1}

E\left[\mathbf{X}\right]=\frac{d}{d\left(i\omega\right)}\Phi_{\mathbf{X}}\left(\omega\right)|_{i\omega=0}=(-1)(1-i\omega)^{-2}(-1)|_{i\omega=0}=1

(b)

Calculate E\left[\mathbf{Y}\right]

E\left[\mathbf{Y}\right]=1

(c)

Calculate E\left[\mathbf{XY}\right] .

E\left[\mathbf{XY}\right]=\frac{\partial^{2}}{\partial\left(i\omega_{1}\right)\partial\left(i\omega_{2}\right)}\Phi_{\mathbf{XY}}\left(\omega_{1},\omega_{2}\right)|_{i\omega_{1}=i\omega_{2}=0}=\left(1-i\omega_{1}\right)^{-2}\left(1-i\omega_{2}\right)^{-2}|_{i\omega_{1}=i\omega_{2}=0}=1

(d)

Calculate E\left[\mathbf{X}^{j}\mathbf{Y}^{k}\right] .

E\left[\mathbf{X}^{j}\mathbf{Y}^{k}\right]

(e)

Calculate the correlation coefficient r_{\mathbf{XY}} between \mathbf{X} and \mathbf{Y} .

r_{\mathbf{XY}}=\frac{Cov\left(\mathbf{X},\mathbf{Y}\right)}{\sigma_{\mathbf{X}}\sigma_{\mathbf{Y}}}=\frac{E\left[\mathbf{XY}\right]-E\left[\mathbf{X}\right]E\left[\mathbf{Y}\right]}{\sigma_{\mathbf{X}}\sigma_{\mathbf{Y}}}=\frac{1-1\cdot1}{\sigma_{\mathbf{X}}\sigma_{\mathbf{Y}}}=0.

Example. Geometric random variable

Let \mathbf{X} be a random variable with probability mass function

p_{\mathbf{X}}\left(k\right)=\alpha\left(1-\alpha\right)^{k-1},k=1,2,3,\cdots

where 0<\alpha<1 .

Note

This is a geometric random variable with success probability \alpha .

(a) Find the characteristic function of \mathbf{X} .

\Phi_{\mathbf{X}}\left(\omega\right)

since \left|e^{i\omega}\left(1-\alpha\right)\right|<1 .

\because0<1-\alpha<1 and the real term of e^{i\omega}=\cos\omega+i\sin\omega is \left|\cos\omega\right|<1 .

(b) Find the mean of \mathbf{X} .

E\left[\mathbf{X}\right]

Note

You can see the other approach to find E\left[\mathbf{X}\right] and Var\left[\mathbf{X}\right] [CS1GeometricDistribution].

(c) Find the variance of \mathbf{X} .

E\left[\mathbf{X}^{2}\right]

because

\frac{d}{d\left(i\omega\right)}e^{i\omega2}\left(1-e^{i\omega}\left(1-\alpha\right)\right)^{-2}\left|_{i\omega=0}\right.

Var\left[\mathbf{X}\right]=E\left[\mathbf{X}^{2}\right]-\left(E\left[\mathbf{X}\right]\right)^{2}=\frac{2-\alpha}{\alpha^{2}}-\frac{1}{\alpha^{2}}=\frac{1-\alpha}{\alpha^{2}}.

Example. Secquence of binomially distributed random variables

Let \left\{ \mathbf{X}_{n}\right\} _{n\geq1} be a sequence of binomially distributed random variables, with the n_{th} random variable \mathbf{X}_{n} having pmf

P_{\mathbf{X}_{n}}(k)=P\left(\left\{ \mathbf{X}_{n}=k\right\} \right)=\left(\begin{array}{c} n\\ k \end{array}\right)p_{n}^{k}\left(1-p_{n}\right)^{n-k}\;,\; k=0,1,\cdots,n,\; p_{n}\in\left(0,1\right). Show that, if the p_{n} have the property that np_{n}\rightarrow\lambda as n\rightarrow\infty , where \lambda is a positive constant, then the sequence \left\{ \mathbf{X}_{n}\right\} _{n\leq1} converges in distribution to a Poisson random variable \mathbf{X} with mean \lambda .

Hint:

You may find the following fact useful:

\lim_{n\rightarrow\infty}\left(1+\frac{x}{n}\right)^{n}=e^{x}.

Solution

If \mathbf{X}_{n} converges to \mathbf{X} in distribution, then F_{\mathbf{X}_{n}}(x)\rightarrow F_{\mathbf{X}}(x) \forall x\in\mathbf{R} , where F_{\mathbf{X}}(x) is continuous. This occurs iff \Phi_{\mathbf{X}_{n}}(\omega)\rightarrow\Phi_{\mathbf{X}}(\omega) \forall x\in\mathbf{R} . We will show that \Phi_{\mathbf{X}_{n}}(\omega) converges to e^{-\lambda\left(1-e^{i\omega}\right)} as n\rightarrow\infty , which is the characteristic function of a Poisson random variable with mean \lambda .

\Phi_{\mathbf{X}_{n}}(\omega)

Now as n\rightarrow\infty , np_{n}\rightarrow\lambda\Rightarrow p_{n}\rightarrow\frac{\lambda}{n} .

\lim_{n\rightarrow\infty}\Phi_{\mathbf{X}_{n}}(\omega)=\lim_{n\rightarrow\infty}\left(1+p_{n}\left(e^{i\omega}-1\right)\right)^{n}=\lim_{n\rightarrow\infty}\left(1+\frac{\lambda}{n}\left(e^{i\omega}-1\right)\right)^{n}=e^{\lambda\left(e^{i\omega}-1\right)}=e^{-\lambda\left(1-e^{i\omega}\right)},

which is the characteristic function of Poisson random variable with mean \lambda .

c.f.

The problem 2 of the August 2007 QE [CS1QE2007August] is identical to this example.

Example. Secquence of exponentially distributed random variables

Let \mathbf{X}_{1},\mathbf{X}_{2},\cdots,\mathbf{X}_{n} be a collection of i.i.d. exponentially distributed random variables, each having mean \mu . Define

\mathbf{Y}=\max\left\{ \mathbf{X}_{1},\mathbf{X}_{2},\cdots,\mathbf{X}_{n}\right\}

and

\mathbf{Z}=\min\left\{ \mathbf{X}_{1},\mathbf{X}_{2},\cdots,\mathbf{X}_{n}\right\} .

(a) Find the pdf of \mathbf{Y} .

F_{\mathbf{Y}}\left(y\right)

f_{\mathbf{Y}}(y)

(b) Find the pdf of \mathbf{Z}

F_{\mathbf{Z}}(z)

f_{Z}(z)

(c) In words, give as complete a description of the random variable \mathbf{Z} as you can.

\mathbf{Z} is an exponetially distributed random variable with mean \frac{\mu}{n} .

Example. Secquence of uniformly distributed random variables

Let \mathbf{X}_{1},\mathbf{X}_{2},\cdots,\mathbf{X}_{n} be n i.i.d. jointly distributed random variables, each uniformly distributed on the interval \left[0,1\right] . Define the new random variables \mathbf{W}=\max\left\{ \mathbf{X}_{1},\mathbf{X}_{2},\cdots,\mathbf{X}_{n}\right\}

and

\mathbf{Z}=\min\left\{ \mathbf{X}_{1},\mathbf{X}_{2},\cdots,\mathbf{X}_{n}\right\} .

(a) Find the pdf of \mathbf{W} .

F_{\mathbf{W}}(w)

where f_{\mathbf{X}}(x)=\mathbf{1}_{\left[0,1\right]}(x) and F_{X}\left(x\right)=\left\{ \begin{array}{ll} 0 & ,x<0\\ x & ,0\leq x<1\\ 1 & ,x\geq1 \end{array}\right. .

f_{\mathbf{W}}\left(w\right)

(b) Find the pdf of \mathbf{Z} .

F_{\mathbf{Z}}(z)

f_{\mathbf{Z}}(z)

(c) Find the mean of \mathbf{W} .

E\left[\mathbf{W}\right]

Example. Mean of i.i.d. random variables

Let \mathbf{X}_{1},\mathbf{X}_{2},\cdots,\mathbf{X}_{n} be M jointly distributed i.i.d. random variables with mean \mu and variance \sigma^{2} . Let \mathbf{Y}_{M}=\frac{1}{M}\sum_{n=0}^{M}\mathbf{X}_{n} .

(a) Find the variance of \mathbf{Y}_{M} .

Var\left[\mathbf{Y}_{M}\right]=E\left[\mathbf{Y}_{M}^{2}\right]-\left(E\left[\mathbf{Y}_{M}\right]\right)^{2}.

E\left[\mathbf{Y}_{M}\right]=E\left[\frac{1}{M}\sum_{n=0}^{M}\mathbf{X}_{n}\right]=\frac{1}{M}\sum_{n=0}^{M}E\left[\mathbf{X}_{n}\right]=\frac{1}{M}\cdot M\cdot\mu=\mu.

E\left[\mathbf{Y}_{M}^{2}\right]=E\left[\frac{1}{M^{2}}\sum_{m=1}^{M}\sum_{n=1}^{M}\mathbf{X}_{m}\mathbf{X}_{n}\right]=\frac{1}{M^{2}}\sum_{m=1}^{M}\sum_{n=1}^{M}E\left[\mathbf{X}_{m}\mathbf{X}_{n}\right].

Now E\left[\mathbf{X}_{m}\mathbf{X}_{n}\right]=\begin{cases} \begin{array}{ll} E\left[\mathbf{X}_{m}^{2}\right] & ,m=n\\ E\left[\mathbf{X}_{m}\right]E\left[\mathbf{X}_{n}\right] & ,m\neq n \end{array}\end{cases} because when m\neq n , \mathbf{X}_{m} and \mathbf{X}_{n} are independent \Rightarrow \mathbf{X}_{m} and \mathbf{X}_{n} are uncorrelated.

E\left[\mathbf{Y}_{M}^{2}\right]=\frac{1}{M^{2}}\left[M\left(\mu^{2}+\sigma^{2}\right)+M\left(M-1\right)\mu^{2}\right]=\frac{\left(\mu^{2}+\sigma^{2}\right)+\left(M-1\right)\mu^{2}}{M}=\frac{M\mu^{2}+\sigma^{2}}{M}.

Var\left[\mathbf{Y}_{M}\right]=\frac{M\mu^{2}+\sigma^{2}-M\mu^{2}}{M}=\frac{\sigma^{2}}{M}.

(b) Now assume that the \mathbf{X}_{1},\mathbf{X}_{2},\cdots,\mathbf{X}_{n} are identically distributed with with mean \mu and variance \sigma^{2} , but they are only correlated rather than independent. Find the variance of \mathbf{Y}_{M} .

Again, Var\left[\mathbf{Y}_{M}\right]=\frac{\sigma^{2}}{M} , because only uncorrelatedness was used in part (a).

Example. A sum of a random number of i.i.d. Gaussians

Let \left\{ \mathbf{X}_{n}\right\} be a sequence of i.i.d. Gaussian random variables, each having characteristic function

\Phi_{\mathbf{X}}\left(\omega\right)=e^{i\mu\omega}e^{-\frac{1}{2}\sigma^{2}\omega^{2}}. Let \mathbf{N} be a Poisson random variable with pmf

p(n)=\frac{e^{-\lambda}\lambda^{n}}{n!},\; n=0,1,2,\cdots,\;\lambda>0, and assume \mathbf{N} is statistically independent of \left\{ \mathbf{X}_{n}\right\} . Define a new random variable

\mathbf{Y}=\mathbf{X}_{1}+\mathbf{X}_{2}+\cdots+\mathbf{X}_{N}.

Note

If \mathbf{N}=0 , then \mathbf{Y}=0 .

(a) Find the mean of \mathbf{Y} .

• Probability generating function of \mathbf{N} is P_{\mathbf{N}}\left(z\right)=E\left[z^{\mathbf{N}}\right]=\sum_{n=0}^{\infty}z^{n}\frac{e^{-\lambda}\lambda^{n}}{n!}=e^{-\lambda}\sum_{n=0}^{\infty}\frac{\left(z\lambda\right)^{n}}{n!}=e^{-\lambda}e^{z\lambda}=e^{-\lambda\left(1-z\right)}.

• The characteristic function of \mathbf{Y} is \Phi_{\mathbf{Y}}\left(\omega\right)=P_{\mathbf{N}}\left(z\right)\Bigl|_{z=\Phi_{\mathbf{X}}\left(\omega\right)}=e^{-\lambda\left(1-z\right)}\Bigl|_{z=e^{i\mu\omega}e^{-\frac{1}{2}\sigma^{2}\omega^{2}}}=e^{-\lambda\left(1-e^{i\mu\omega}e^{-\frac{1}{2}\sigma^{2}\omega^{2}}\right)}.

• Now, we can get the mean of \mathbf{Y} using the characteristic function. E\left[\mathbf{Y}\right]