Central Limit Theorem

The Law of Large Numbers

It is commonly believed that if a fair coin is tossed many times and the proportion of heads is calculated, that proportion will be close to 1/2 . John Kerrich, a South African mathematician, tested this belief empirically while detained as a prisoner during World War II. He tossed a coin 10,000 times and observed 5067 heads. The law of large numbers is a mathematical formulation of this belief. The successive tosses of the coin are modeled as independent random trials. The random variable $X_i$ takes on the value 0 or 1 according to whether the $i$th trial results in a tail or a head, and the proportion of heads in $n$ trials is

$\bar{X_n}=\frac{1}{n}\sum_i^n X_i$

The law of large numbers states that $\bar{X_n}$ approaches 1/2 in a sense that is specified by the following theorem.

Convergence in Distribution and the Central Limit Theorem

In applications, we often want to find $P(a < X < b)$ when we do not know the cdf of $X$ precisely; it is sometimes possible to do this by approximating $F_X$ . The approximation is often arrived at by some sort of limiting argument. The most famous limit theorem in probability theory is the central limit theorem, which is the main topic of this section. Before discussing the central limit theorem, we develop some introductory terminology, theory, and examples.

Moment-generating functions are often useful for establishing the convergence of distribution functions. We know that a distribution function $F_n$ is uniquely determined by its mgf, $M_n$. The following theorem, which we give without proof, states that this unique determination holds for limits as well.

Central Limit Theorem

In probability theory, the central limit theorem (CLT) establishes that, in many situations, when independent random variables are summed up, their properly normalized sum tends toward a normal distribution even if the original variables themselves are not normally distributed.

The theorem is a key concept in probability theory because it implies that probabilistic and statistical methods that work for normal distributions can be applicable to many problems involving other types of distributions.

This theorem has seen many changes during the formal development of probability theory. Previous versions of the theorem date back to 1811, but in its modern general form, this fundamental result in probability theory was precisely stated as late as 1920, thereby serving as a bridge between classical and modern probability theory.

If ${\textstyle X_{1},X_{2},\dots ,X_{n},\dots }$ are random samples drawn from a population with overall mean ${\textstyle \mu }$ and finite variance  ${\textstyle \sigma ^{2}}$, and if  ${\textstyle {\bar {X}}_{n}}$ is the sample mean of the first ${\textstyle n}$ samples, then the limiting form of the distribution, 

${\textstyle Z=\lim _{n\to \infty }{\left({\frac {{\bar {X}}_{n}-\mu }{\frac {\sigma }{\sqrt {n}}}}\right)}}$

, is a standard normal distribution.

For example, suppose that a sample is obtained containing many observations, each observation being randomly generated in a way that does not depend on the values of the other observations, and that the arithmetic mean of the observed values is computed. If this procedure is performed many times, the central limit theorem says that the probability distribution of the average will closely approximate a normal distribution. A simple example of this is that if one flips a coin many times, the probability of getting a given number of heads will approach a normal distribution, with the mean equal to half the total number of flips. At the limit of an infinite number of flips, it will equal a normal distribution.

Let ${\textstyle \{X_{1},\ldots ,X_{n},\ldots \}}$ be a sequence of random samples — that is, a sequence of independent and identically distributed (i.i.d.) random variables drawn from a distribution of expected value given by ${\textstyle \mu }$ and finite variance given by ${\textstyle \sigma ^{2}}$. Suppose we are interested in the sample average ${\displaystyle {\bar {X}}_{n}\equiv {\frac {X_{1}+\cdots +X_{n}}{n}}}$ of the first ${\textstyle n}$ samples.

By the law of large numbers, the sample averages converge almost surely (and therefore also converge in probability) to the expected value ${\textstyle \mu }$ as ${\textstyle n\to \infty }$.

The classical central limit theorem describes the size and the distributional form of the stochastic fluctuations around the deterministic number ${\textstyle \mu }$ during this convergence. More precisely, it states that as ${\textstyle n}$ gets larger, the distribution of the difference between the sample average ${\textstyle {\bar {X}}_{n}}$ and its limit ${\textstyle \mu }$, when multiplied by the factor ${\textstyle {\sqrt {n}}}$ (that is ${\textstyle {\sqrt {n}}({\bar {X}}_{n}-\mu )})$  approximates the normal distribution with mean 0 and variance ${\textstyle \sigma ^{2}}$. For large enough $n$, the distribution of ${\textstyle {\bar {X}}_{n}}$ is close to the normal distribution with mean ${\textstyle \mu }$ and variance ${\textstyle \sigma ^{2}/n}$.