Jump to Problem 2,3

Problem 2

Problem statement: Let X be a continuous or discrete random variable with mean μ and variance σ². Then, $ \forall \varepsilon >0 $, we have
$ P(|X-\mu| \geq \varepsilon) \leq \frac{\sigma^2}{\varepsilon^2} $

$ \color{blue}\text{Solution 1:} $

• Continuous Case:
  

Let X be a random variable with mean $ \mu $ and variance $ \sigma^2 $

Consider two functions g₁(x) and g₂(x)

$ g_{1}(x)= 1_{\{x: \mid x-\mu\mid \geq \varepsilon\}}(x) $

$ g_{2}(x)=\frac{(x-\mu)^2}{\varepsilon^2} $

Clearly,

$ g_{2}(x)-g_{1}(x) \geq 0\; \; \forall x \in \mathbb {R} $

$ E[g_{2}(x)-g_{1}(x) ] \geq 0 \; \; \forall x \in \mathbb {R} $

Consider,

$ E[g_{2}(x)-g_{1}(x) ] = E[g_{2}(x)]-E[g_{1}(x)] $

where, $ E[g_{2}(x)]=E[\frac{(x-\mu)^2}{\varepsilon^2}] = \frac{1}{\varepsilon^2} var(X) = \frac{\sigma^2}{\varepsilon^2} $

and

$ E[g_{1}(x)] = P \{\mid x - \mu\mid \geq \varepsilon \} $

Thus we get,

$ \frac{\sigma^2}{\varepsilon^2} - P\{\mid X - \mu \mid \geq \varepsilon\} \geq 0 $

Therefore,

$ P\{\mid X - \mu \mid \geq \varepsilon\} \leq \frac{\sigma^2}{\varepsilon^2} $

• Discrete Case:
  

The inequalities described above hold even if the x-axis is discretized. That is, if the random variable X is discrete. In other words, the discrete case is a proper subset of the continuous case in this problem. Thus, the Chebyshev inequality holds for discrete random variables.

$ {\color{red} \text{Basically, the whole proof is correct. But, there is a mistake in the following equation.}} $

$ {\color{red} E[g_{1}(x)] = P \{\mid x - \mu\mid \geq \varepsilon \} } $

$ {\color{red} \text{The probability measure } P( \cdot ) \text{ should evaluate on an event. Thus, I will rewrite this equation as follow}} $

$ {\color{red} E[g_{1}(x)] = P \left( \{x: X(x) - \mu\mid \geq \varepsilon \} \right) } $

$ \color{blue}\text{Solution 2:} $

• Discrete Case:
  

Let $ p_{X}(x) $ be the pmf of X. The probability that $ X $ differs from $ \mu $ by at least $ \varepsilon $ is

$ P(|X-\mu| \geq \varepsilon)= \sum_{x \in \{|X-\mu| \geq \varepsilon \} }p_{X}(x) $

Based on the definition of the variance, we have
$ \sigma^2 = E[(X-E[X])]=\sum_{x}(x-\mu)^2 p_{X}(x) $

Let a set $ A= \{ x|\,|x-\mu| \geq \varepsilon \} $. We have
$ \sigma^2 = \sum_{x}(x-\mu)^2 p_{X}(x)= \sum_{x \in A}(x-\mu)^2 p_{X}(x)+\sum_{x \notin A}(x-\mu)^2 p_{X}(x) $
$ \Rightarrow\sigma^2 \geq \sum_{x \in A}(x-\mu)^2 p_{X}(x) $
Since, in set A, we have $ |x-\mu| \geq \varepsilon $, we have
$ \Rightarrow\sigma^2 \geq \sum_{x \in A}\varepsilon^2 p_{X}(x)= \varepsilon^2 \sum_{x \in A}p_{X}(x)=\varepsilon^2 P(x \in A) =\varepsilon^2 P(|X-\mu| \geq \varepsilon) $
That is
$ P(|X-\mu| \geq \varepsilon) \leq \frac{\sigma^2}{\varepsilon^2} $

• Continuous Case:
  

Let f_X(x) be the pdf of X.
$ \sigma^2=\int_{-\infty}^{\infty}(x-\mu)^2f_{X}(x) \,dx \geq \int_{-\infty}^{\mu-\varepsilon}(x-\mu)^2f_{X}(x) \,dx+ \int_{\mu+\varepsilon}^{\infty}(x-\mu)^2f_{X}(x) \,dx $
The last inequality holds since we integrate a positive function. Since $ x \leq \mu-\varepsilon $ or $ x \geq \mu+\varepsilon $
$ \Rightarrow |x-\mu| \geq \varepsilon \Rightarrow (x-\mu)^2 \geq \varepsilon^2 $
Based on the above equation, we have
$ \sigma^2 \geq \int_{-\infty}^{\mu-\varepsilon}\varepsilon^2 f_{X}(x) \,dx+ \int_{\mu+\varepsilon}^{\infty} \varepsilon^2 f_{X}(x) \,dx $
$ = \varepsilon^2 \left( \int_{-\infty}^{\mu-\varepsilon}f_{X}(x) \,dx+ \int_{\mu+\varepsilon}^{\infty} f_{X}(x) \,dx \right) = \varepsilon^2 P \bigg( X \leq (\mu-\varepsilon)\, \text{or} \, X \geq (\mu+\varepsilon) \bigg) = \varepsilon^2 P(|X-\mu| \geq \varepsilon) $
$ \Rightarrow P(|X-\mu| \geq \varepsilon) \leq \frac{\sigma^2}{\varepsilon^2} $