Probability 4 -- Law of Large Number and Central Limit Theorem

This paper mainly comes fromProbability and Statistical Inference, Second Edition, Robert Bartoszynski, Magdalena Niewiadomska-Bugaj

In this chapter we collect two extremely import theorem in probability theorem -- Law of Large Numbers and Central Limit Theorem.

Weak Law of Large Numbers

These theorems assert convergence in probability of empirical frequencies of an event. Theorem 1 to 3 form a hierarchical relationship. Theorem 1 restricts sequence of random variables to be independent identical distribution. Theorem 2 improves the identical requirement to limit for the sum of variance. Theorem 3 further improves independent requirement to boundary of covariance. Theorem 4 generalizes theorem to distributions with variance not defined.

Theorem 1:

Let \(\{X_n\}\) be a sequence of i.i.d. random variables. Assume that \(E(X_i) = \mu, Var(X_i) = \sigma^2.\) Then for every \(\varepsilon > 0,\) \[ \lim_{n \rightarrow \infty} P \left(\left\{| \frac{\sum_{i=1}^n X_i}{n} - \mu | \ge \varepsilon \right\}\right) = 0. \]

Theorem 2:

Let \(\{X_n\}\) be a sequence of independent random variables. Assume that \(E(X_i) = \mu_i, Var(X_i) = \sigma_i^2.\) If \[ \lim_{n \rightarrow \infty} \frac{1}{n^2} \sum_{i=1}^n \sigma_i^2 = 0, \]then for every \(\varepsilon > 0,\) \[ \lim_{n \rightarrow \infty} P \left(\left\{| \frac{\sum_{i=1}^n X_i}{n} - \frac{\sum_{i=1}^n \mu_i}{n} | \ge \varepsilon\right\}\right) = 0. \]

Theorem 3:

Let \(\{X_n\}\) be a sequence of random variables with \(E(X_i) = \mu_i, Var(X_i) = \sigma_i^2,\) and such that \(Cov(X_i, X_j) \le 0,\) satisfying \[ \lim_{n \rightarrow \infty} \frac{1}{n^2} \sum_{i=1}^n \sigma_i^2 = 0. \]Then for every \(\varepsilon > 0,\) \[ \lim_{n \rightarrow \infty} P \left(\left\{| \frac{\sum_{i=1}^n X_i}{n} - \frac{\sum_{i=1}^n \mu_i}{n} | \ge \varepsilon\right\}\right) = 0. \]

Theorem 4:

Let \(\{X_n\}\) be a sequence of i.i.d. random variables. Assume that \(E(X_i) = \mu,\) and such that their common moments generating function exists in some neighborhood of \(t = 0.\) Then for every \(\varepsilon > 0,\) \[ \lim_{n \rightarrow \infty} P \left(\left\{| \frac{\sum_{i=1}^n X_i}{n} - \mu | \ge \varepsilon \right\}\right) = 0. \]

Strong Law of Large Numbers

These theorems assert almost sure convergence of sequences of random variables obtained by averaging some underlying sequences of random variables.

Theorem 5:

Let \(\{X_n\}\) be a sequence of independent random variables. Assume that \(E(X_i) = \mu_i, Var(X_i) = \sigma_i^2.\) If \[ \sum_{n=1}^{\infty} \frac{\sigma_n^2}{n^2} < \infty, \]then \[ \frac{1}{n}\sum_{i=1}^n\left(X_i - \mu_i\right) \overset{a.s.}{\longrightarrow} 0. \]

Theorem 6:

Let \(\{X_n\}\) be a sequence of i.i.d. random variables, and let \(S_n = X_1 + X_2 + ... + X_n.\) If \(\mu = E(X_i)\) exists, then \[ \frac{S_n}{n} \overset{a.s.}{\longrightarrow} \mu. \]

Central Limit Theorem

These theorems assert that the sums of large numbers of random variables, after standardization, have approximately standard distribution.

Theorem (Lindeberg and Levy):

Let \(\{X_n\}\) be a sequence of i.i.d. random variables. Assume that \(E(X_i) = \mu, Var(X_i) = \sigma^2, \sigma^2 \in (0, \infty).\) Then let \(S_n = X_1 + ... + X_n,\) for every \(x\) \[ \lim_{n \rightarrow \infty} P\left\{\frac{S_n - n\mu}{\sigma\sqrt{n}} \le x \right\} = \frac{1}{\sqrt{2\pi}}\int_{-\infty}^x e^{\frac{-t^2}{2}} dt. \]

Theorem(Liapunov):

Let \(\{X_n\}\) be a sequence of random variables. Assume that \(E(X_i) = \mu_i, Var(X_i) = \sigma_i^2,\) and \(\gamma_i = E(X_i - \mu_i)^3.\) Moreover, put \[ m_n = \sum_{j=1}^n \mu_j, s_n^2 = \sum_{j=1}^n \sigma_j^2, \Gamma_n = \sum_{j=1}^n \gamma_i, \] and let \(S_n = X_1 + ... + X_n\) be the corresponding sequence of partial sums. If all \(X_i\) have finite third moments, then the condition \[ \lim_{n \rightarrow \infty} \frac{\Gamma_n}{s_n^3} = 0 \]is sufficient for convergence \[ \frac{S_n - m_n}{s_n} \overset{d}{\longrightarrow} \mathcal{N}(0, 1). \]

Theorem(Lindeberg and Feller):

Let \(\{X_n\}\) be a sequence of independent random variables with finite second moments. Assume that \(s_n^2 \rightarrow \infty\) and \[ \lim_{n \rightarrow \infty}\max_{1 \le j \le n} \frac{\sigma_j^2}{s_n^2} = 0. \]Then \[ \frac{S_n - m_n}{s_n} \overset{d}{\longrightarrow} \mathcal{N}(0, 1) \]iff for every \(\varepsilon > 0,\) \[ \lim_{n \rightarrow \infty}\frac{1}{s_n^2}\sum_{j = 1}^n\int_{|x-\mu_j| \ge \varepsilon s_n} (x-\mu_j)^2dF_j(x) = 0. \]

The comparison between LLNs and CLT are always interesting. The Law of Large Numbers describes the sample mean will converge to theoretical mean, which is a scalar to another scalar. The Central Limit Theorem describes the distribution of standardized sum converge to normal distribution, which is a distribution to another distribution.

cite:

Probability and Statistical Inference, Second Edition, Robert Bartoszynski, Magdalena Niewiadomska-bugaj