Sample mean of a random variable

Intuition from some examples

给定一个随机变量$X$X满足某种分布, 我们可以通过sample它得到其mean or variance. 假设sample了$N$N个点, 那sample mean
$\overline{X}=\frac{1}{N}\sum_{n=1}^N X_n$X=N1​n=1∑N​Xn​
随着$N$N的增加,$\overline{X}$X应该越来越趋近于真实的mean $\mathbb{E}[X]$E[X] 最终相等. 但是simulation results indicate otherwise.
例1: Bernoulli distribution $X\sim Bernoulli(p=0.6)$X∼Bernoulli(p=0.6), 随着N的增大,sample mean的curve如下图所示

可以看到, 最终的mean确实好像是收敛到了$p=0.6$p=0.6. 但是如果我们放大来看的话会发现, 这条曲线实际上在抖动,就是说他并不是converge到一个点的.

例2: Gaussian distribution $X\sim N(0,1)$X∼N(0,1), sample mean的curve

可以看到, $10^6$106之后sample mean仍在抖动. 虽然抖动的很小, 但至少不是想象中的converge to a single point.

以上simulation表示, sample mean 收敛不到 population mean (i.e., 真实的mean) . It should be close to the population mean, but may not exactly equal the population mean.

Main Results

另一种表述方法是: 即使 $N$N足够大, 每次sample N次得到的 $\overline{X}$X仍然不是一个固定值, 而是一个distribution.

Theorem 1 (mean of sample mean). If $\mathbb{E}[X]=\mu$E[X]=μ, then $\mathbb{E}[\overline{X}]=\mu$E[X]=μ.

Theorem 1很好理解, 即 $X$X 的 sample mean 的 mean 即为 $X$X 的 mean. 这也很好verify:

$\mathbb{E}[\overline{X}]= \mathbb{E}\bigg[\frac{1}{N}\sum_{n=1}^N X_n\bigg] =\frac{1}{N}\sum_{n=1}^N \mathbb{E}[X_n]=\mathbb{E}[X]=\mu,$E[X]=E[N1​n=1∑N​Xn​]=N1​n=1∑N​E[Xn​]=E[X]=μ,

因为每次的sample都是i.i.d.的.

Theorem 2 (variance of sample mean). If $\text{var} [X]=\sigma^2$var[X]=σ2, then $\text{var}[\overline{X}]=\frac{\sigma^2}{N}$var[X]=Nσ2​.

$\text{var}[\overline{X}]= \text{var}\bigg[\frac{1}{N}\sum_{n=1}^N X_n\bigg] =\frac{1}{N^2}\sum_{n=1}^N \text{var}[X_n]=\frac{\sigma^2}{N},$var[X]=var[N1​n=1∑N​Xn​]=N21​n=1∑N​var[Xn​]=Nσ2​,

从Theorem 2中也可以看出, 多sample是有好处的, $N$N越大sample mean 的variance越小也就越趋近于population mean.

Conclusion

Overall, the sample mean is not a robust statistic, meaning that they are sensitive to outliers. We can only give a lower bound and an upper bound of the population mean, and say how confident we are (in %) that the population mean is between the lower bound and upper bound of the confidence interval.

Confidence interval is $\big[ \overline{X}-E, \overline{X}+E\big]$[X−E,X+E], where $E$E is called the margin of error, and is given by
$E=z_{\alpha/2}\frac{\sigma}{\sqrt{N}}$E=zα/2​N​σ​

$z$z: critical value, can be computed from standard normal distribution if given $\alpha/2$α/2.
$\alpha$α: significance level.
$CL=1-\alpha$CL=1−α: confidence level.

As shown in the figure,

1. Given a $CL=95\%$CL=95%;
2. Calculate $\alpha = 0.05$α=0.05 and $\alpha = 0.025$α=0.025;
3. Check norm distribution table and find $z_{\alpha/2}=z_{0.025}=1.96$zα/2​=z0.025​=1.96
4. Compute $E=z_{\alpha/2}\frac{\sigma}{\sqrt{N}}$E=zα/2​N​σ​, and the confidence interval.