Communication, Networking, Signal and Image Processing (CS)

Question 1: Probability and Random Processes

August 2015

## Contents

### Solution 1

$E(S_n)=E(\frac{1}{n}\sum_i^n X_i) =\frac{1}{n}\sum_i^n E(X_i)=0$

$E(X_i-S_n)=E(X_i-\frac{1}{n}\sum_k^n X_k) =E(X_i)-E(\frac{1}{n}\sum_k^n X_k)=0$

$E((X_i-S_n)S_n)=E(X_iS_n-S_n^2)$

As for any $i,j\in \{1,2,...,n\}$, we have $E(X_i\cdot X_j) = E(X_i)E(X_j)=0$

$E(X_iS_n-S_n^2) = E(X_iS_n)-E(S_n^2)\\ =E(\sum_k^nX_iX_K) - E(\sum_i^n\sum_k^nX_iX_K)\\ =\sum_k^nE(X_iX_K) - \sum_i^n\sum_k^nE(X_iX_K) \\ =0$

Thus $E(X_i-S_n)E(S_n)=E((X_i-S_n)S_n)$, $S_n$ and $X_i-S_n$ are uncorrelated.

### Solution 2

$S_n=\frac{1}{n}\sum_{j=1}{n}X_j$, note: in the problem statement, it should be $\frac{1}{n}, because [itex]S_n$ is the sample mean.

$E[S_n]=E[\frac{1}{n}\sum_{j=1}{n}X_j] = \frac{1}{n}\sum_{j=1}{n}E[X_j ] = \frac{1}{n}\sum_{j=1}{n} \mu = 0\\ E[(X_i-\mu)^2]=E[X_i^2]=\sigma^2$

$E[X_iX_j]=\int_{-\infty}^{+\infty}x_ix_jf_{X_iX_j}(x_i,x_j)dx_idx_j=\int_{-\infty}^{+\infty}x_if_{X_i}(x_i)x_jf_{X_j}(x_j)dx_idx_j=E[X_i]E[X_j]=\mu\cdot\mu=0$

$E[X_i-S_n]=E[X_i]-E[S_n]=0-0=0$

$E[X_i\cdot S_n]=E[\frac{1}{n}\sum_{j=1}^{n}X_j\cdot X_i]=\frac{1}{n}\sum_{j=1}^{n}E[X_j\cdot X_i]=\frac{1}{n}\cdot \sigma^2$

$E[S_n^2]=E[\frac{1}{n^2}\sum_{j=1}^{n}\sum_{i=1}^{n}X_j\cdot X_i]=\frac{1}{n^2}\sum_{j=1}^{n}E[X_i^2]+\frac{1}{n^2}\sum_{j=1}^{n}\sum_{i=1}^{n}E[X_i\cdot X_j]=\frac{1}{n^2}\cdot (n\cdot \sigma^2) + \frac{1}{n^2}\cdot 0 = \frac{\sigma^2}{n}$

Therefore,

$E[(S_n-0)(X_i-S_n-0)]=E[S_nX_i-S_n^2]= E[S_nX_i]-E[S_n^2]=\frac{\sigma^2}{n}-\frac{\sigma^2}{n}=0$

So $r = \frac{cov(S_n,X_i-S_n)}{\sigma_{S_n}\sigma_{X_i-S_n}}=0$

Thus $S_n$ and $X_i-S_n$ are uncorrelated.

### Solution 3

Our goal is to show that $S_n$ and $X_i$ - $S_n$ are uncorrelated $\forall i \in 1, 2, ..., n$. If we can show that the covariance between $S_n$ and $X_i - S_n$ is equal to 0 $\forall i$, then we will have shown the aforementioned property. Recalling that

$cov(X,Y) = E[XY] - E[X]E[Y],$

we aim to show that

$E[(S_n)(X_i -S_n)] - E[S_n]E[X_i-S_n] = 0.$

Let us consider the LHS of the above equation. This can be written as

$E[(S_n)(X_i -S_n)] - (E[S_n]E[X_i]-E[S_n]E[S_n]).$

We are given that $E[X_i] = 0 \forall i$, and since $S_n \triangleq \frac{1}{n}\sum^n_{j=1}X_j$ it is easy to show that $E[S_n] = 0$ as well. Thus the above becomes

$E[(S_n)(X_i -S_n)]$

or

$E[S_n X_i] - E[S_n^2].$

Recalling that the above expression must be equal to 0, our problem has reduced to showing that

$E[S_n X_i] = E[S_n^2].$

Let us first examine $E[S_n X_i]$. This can be rewritten as

$E[S_n X_i] = E\left[\left(\frac{1}{n}\sum^n_{j = 1}X_j \right)\cdot X_i\right] = \frac{1}{n}E[X_1 X_i + X_2 Xi + ... + X_i X_i + ... + X_n X_i].$

Since the sequence $X_1, X_2, ..., X_n$ is i.i.d, if $i\neq j, E[X_i X_j] = 0$. Thus, the above becomes

$\frac{1}{n}\left(0 + 0 + ...\, E[X_i^2] + ... + 0 + 0\right).$

Since $X_i$ is zero-mean $\forall i$, we know that $E[X_i^2] = var(X_i) = \sigma^2$ for any $i$, and that $E[S_n X_i] = \frac{\sigma^2}{n}$. Now let us examine $E[S_n^2]$. This can be rewritten as

$E[S_n^2] = E\left[\frac{1}{n}\left(X_1 + X_2 + ... + X_n\right)\frac{1}{n}\left(X_1 + X_2 + ... + X_n\right)\right] = \frac{1}{n^2}E\left[\left(X_1 + X_2 + ... + X_n\right)^2\right]$.

Squaring out the expression inside the expectation operator will result in an expression with $n$ square terms and $\sum^n_{j-1}(j-1)$ cross terms. Recall that the expectation values of the cross terms will all be zero since $X_i$ is zero-mean $\forall i$. Then we have

$\frac{1}{n^2}E\left[\left(X_1 + X_2 + ... + X_n\right)^2\right] = \frac{1}{n^2}\left(E[X_1^2] + E[X_2^2] + ... + E[X_n^2]\right) = \frac{1}{n}(n\sigma^2) = \frac{\sigma^2}{n}.$

Thus we have shown that $E[S_n X_i] = E[S_n^2] = \frac{\sigma^2}{n}$, and we are done.

### Similar Problem

Consider a similar problem, except in the case that $\mu\neq 0 \,\forall \,X_i, i=1,...,n$. Are the sample mean $S_n=\sum_{j=1}^{n}X_j$ and $X_i$ uncorrelated for every $i=1,...,n$?