1.1 Predominance of a normal distribution

1. It represents a natural extension of the univariate normal distribution and provides a suitable model for many real-life problems concerning vector-valued data.

2. Even if, in an experiment, the original data cannot be fitted satisfactorily with a multivariate normal distribution, by the central limit theorem the distribution of the sample mean vector is asymptotically normal. Thus the multivariate normal distribution can be used for approximating the distribution of the sample mean vector in the large sample size.

3. The density function of a multivariate normal distribution is uniquely determined by the mean vector and the covariance matrix of the random vector.

4. Zero correlations imply independence; that is, if all the correlation coefficients between two sets of components of a multivariate normal vector are zero, the two sets of components are independent.

5. The distributions of linear transformations or linear combinations of multivariate normal vectors are again multivariate normal.

6. The marginal distribution of any subset of components of a multivariate normal vector is also multivariate normal.

7. The conditional distribution in a multivariate normal distribution is multivariate normal. Furthermore, the conditional mean vector is a linear function and the conditional covariance matrix depends only on the covariance matrix of the joint distribution. This property yields simple and useful results in regression analysis and correlation analysis.

1.2 Historical remarks

Studies of the bivariate normal distribution began in the 19-th century, and moved forward dramatically when Francis Galton (1888) published his work on the applications of correlation analysis in genetics. As Karl Pearson noted, ”In 1885 F.Galton had completed the theory of bivariate normal correlation” but, because ”he was very modest and throughout his life underrated his own mathematical powers, he did not at once write down the equation” of the bivariate normal density function (K.Pearson, 1920). Before F.Galton, the bivariate normal density function was also investigated by R.Adrian (1808), P.S.Laplace (1810), G.A.Plana (1813), C.F.Gauss (1823), A.Bravais (1846). But nobody of them gave the definition of the correlation coefficient as a measure of the dependence between variables, characteristics of the conditional distributions etc.

It was K.Pearson (1896) who ”gave a definitive mathematical formulation” of the bivariate normal distribution (H.L.Seal, 1967). The development of the multivariate normal distribution theory originated mainly from the studies of regression analysis and multiple and partial correlation analysis, and was treated comprehensively for the first time by F.Y.Edgeworth (1892). The developments of the sampling distribution theory under the assumption of normality (as in Fisher’s work on the distributions of sample correlation coefficients, Hotelling’s T^{2} distribution, and the Wishart distribution) then followed. Today, multivariate normal theory has become a fully grown area in statistics and plays a central role in statistical applications.

1.3 Definition

Definition 1. A two-dimensional random vector X = (X_{1}, X_{2})′ is said to have a non-singular bivariate normal distribution if its density function is of the form

f(x; μ, Σ) = \frac{1}{2π|Σ|^{1/2}} exp \{−(x − μ)′Σ^{−1}(x − μ)/2\} , x ∈ ℝ^{2},

where μ = (μ_{1}, μ_{2})′,

Σ = ( \begin{matrix}σ_{1}^{2} & σ_{12} \\ σ_{12} & σ_{2}^{2} \end{matrix} ) , σ_{1}^{2} > 0, σ_{2}^{2} > 0, |σ_{12}| < σ_{1}σ_{2}.

The matrix Σ is positive definite (Σ > 0), i.e. for any x ∈ ℝ^{2}, x ≠ 00 0,

x′Σx > 0.

A bivariate normal distribution is called standard if μ=00 0, σ_{1} =σ_{2} =1. A level curve for the density f(x; μ, Σ) (called a concentration ellipse):

{x ∈ ℝ^{2} : (x − μ)′Σ^{−1}(x − μ) = c}, c > 0.

Properties of the density function: f(x; μ, Σ) = f(−x; μ, Σ) ⟺ μ = 00 0. f((x_{1}, x_{2})′; 00 0, Σ) = f((x_{1}, −x_{2})′; 00 0, Σ) = f((−x_{1}, x_{2})′; 00 0, Σ) ⟺ σ_{12} = 0. f((x_{1}, x_{2})′; 00 0, Σ) = f((x_{2}, x_{1})′; 00 0, Σ) ⟺ σ_{1} = σ_{2}.

Definition 2. A two-dimensional random vector X = (X_{1}, X_{2})′ is said to have a singular bivariate normal distribution if there exist real numbers

σ_{1}, σ_{2}, μ_{1}, μ_{2} such that X_{1} =^{d} σ_{1}Z + μ_{1}, X_{2} =^{d} σ_{2}Z + μ_{2}, where Z has

an N(0, 1) distribution.

A singular bivariate normal distribution is concentrated on a line.

1.4 Calculations

Let ρ = σ_{12}/(σ_{1}σ_{2}) ∈ (−1, 1). Then |Σ| = σ_{1}^{2}σ_{2}^{2}(1 − ρ^{2}) and

Σ^{−1} = \frac{1}{σ_{1}^{2}σ_{2}^{2}(1 − ρ^{2})} ( \begin{matrix}σ_{2}^{2} & −ρσ_{1}σ_{2} \\ −ρσ_{1}σ_{2} & σ_{1}^{2} \end{matrix} ) = \frac{1}{1 − ρ^{2}} ( \begin{matrix}\frac{1}{σ_{1}^{2}} & −\frac{ρ}{σ_{1}σ_{2}} \\ −\frac{ρ}{σ_{1}σ_{2}} & \frac{1}{σ_{2}^{2}} \end{matrix} ) .

Thus

f(x; μ, Σ) = \frac{1}{2πσ_{1}σ_{2}\sqrt{1 − ρ^{2}}} exp \{−\frac{1}{2(1 − ρ^{2})} [(\frac{x_{1} − μ_{1}}{σ_{1}})^{2}

−2ρ (\frac{x_{1} − μ_{1}}{σ_{1}}) (\frac{x_{2} − μ_{2}}{σ_{2}}) + (\frac{x_{2} − μ_{2}}{σ_{2}})^{2}]\} , x ∈ ℝ^{2}.

We have a symmetry in this formula! From the identity

(\frac{x_{1} − μ_{1}}{σ_{1}})^{2} − 2ρ (\frac{x_{1} − μ_{1}}{σ_{1}}) (\frac{x_{2} − μ_{2}}{σ_{2}}) + (\frac{x_{2} − μ_{2}}{σ_{2}})^{2}

= (1−ρ^{2}) (\frac{x_{2} − μ_{2}}{σ_{2}})^{2}+[(\frac{x_{1} − μ_{1}}{σ_{1}}) − ρ (\frac{x_{2} − μ_{2}}{σ_{2}})]^{2} ,

we obtain

f(x; μ, Σ) = \frac{1}{σ_{2}}ϕ (\frac{x_{2} − μ_{2}}{σ_{2}}) \frac{1}{σ_{1}\sqrt{1 − ρ^{2}}}ϕ (\frac{x_{1} − (μ_{1} + ρ(σ_{1}/σ_{2})(x_{2} − μ_{2}))}{σ_{1}\sqrt{1 − ρ^{2}}}) .

So, the random variables

Y_{1} = \frac{X_{1} − μ_{1} − ρ(σ_{1}/σ_{2})(X_{2} − μ_{2})}{σ_{1}\sqrt{1 − ρ^{2}}} and Y_{2} = \frac{X_{2} − μ_{2}}{σ_{2}}

are independent and N(0, 1) distributed.

1.5 Marginal distributions

The marginal distribution of X_{i} is N(μ_{i}, σ_{i}^{2}), i = 1, 2. Indeed, the marginal density of X_{2} is

f_{X_{2}}(x_{2}) = ∫_{−∞}^{∞} f((x_{1}, x_{2})′; μ, Σ)dx_{1} = \frac{1}{σ_{2}}ϕ (\frac{x_{2} − μ_{2}}{σ_{2}})

× ∫_{−∞}^{∞} \frac{1}{σ_{1}\sqrt{1 − ρ^{2}}}ϕ (\frac{x_{1} − (μ_{1} + ρ(σ_{1}/σ_{2})(x_{2} − μ_{2}))}{σ_{1}\sqrt{1 − ρ^{2}}}) dx_{1} = \frac{1}{σ_{2}}ϕ (\frac{x_{2} − μ_{2}}{σ_{2}}) ,

where x_{2} ∈ ℝ. By symmetry, the marginal density of X_{1} is

f_{X_{1}}(x_{1}) = ∫_{−∞}^{∞} f((x_{1}, x_{2})′; μ, Σ)dx_{2} = \frac{1}{σ_{1}}ϕ (\frac{x_{1} − μ_{1}}{σ_{1}}) , x_{1} ∈ ℝ.

Thus EX = μ.

The next example shows that if two marginal distributions are normal then the bivariate distribution need not be the normal one! Example. Consider the distribution with the density

f_{X}(x) = \frac{1}{2}f(x; 00 0, Σ_{1}) + \frac{1}{2}f(x; 00 0, Σ_{2}),

where

Σ_{1} = ( \begin{matrix}1 & ρ_{1} \\ ρ_{1} & 1 \end{matrix} ) , Σ_{2} = ( \begin{matrix}1 & ρ_{2} \\ ρ_{2} & 1 \end{matrix} ) , ρ_{1} ≠ ρ_{2}.

Clearly, the distribution of X is not normal. But X_{1} ∼ N(0, 1) since

f_{X_{1}}(x_{1}) = ∫_{−∞}^{∞} f_{X}(x_{1}, x_{2})dx_{2}

= \frac{1}{2} ∫_{−∞}^{∞} f((x_{1}, x_{2})′; 00 0, Σ_{1})dx_{2} + \frac{1}{2} ∫_{−∞}^{∞} f((x_{1}, x_{2})′; 00 0, Σ_{1})dx_{2}

= \frac{1}{2}ϕ(x_{1}) + \frac{1}{2}ϕ(x_{1}) = ϕ(x_{1}), x_{1} ∈ ℝ.

Similarly, X_{2} ∼ N(0, 1).

1.6 Correlation coefficient

Let us calculate the covariance of X_{1} and X_{2}. Since

∫_{−∞}^{∞} \frac{z − μ}{σ}ϕ (\frac{z − μ}{σ}) dz = 0, ∫_{−∞}^{∞} z^{2}ϕ(z)dz = 1,

we have

Cov (X_{1}, X_{2})=E[(X_{1}−μ_{1})(X_{2}−μ_{2})]=∫_{−∞}^{∞}∫_{−∞}^{∞}(x_{1}−μ_{1})(x_{2}−μ_{2})f(x; μ, Σ)dx_{1}dx_{2}

=∫_{−∞}^{∞}∫_{−∞}^{∞} \frac{x_{2} − μ_{2}}{σ_{2}}ϕ(\frac{x_{2} − μ_{2}}{σ_{2}})\frac{x_{1} − μ_{1}}{σ_{1}\sqrt{1 − ρ^{2}}}ϕ(\frac{x_{1} − (μ_{1} + ρ(σ_{1}/σ_{2})(x_{2} − μ_{2}))}{σ_{1}\sqrt{1 − ρ^{2}}})dx_{1}dx_{2}

= σ_{1}σ_{2} ∫_{−∞}^{∞} z_{2}ϕ(z_{2}) ∫_{−∞}^{∞} \frac{z_{1}}{\sqrt{1 − ρ^{2}}}ϕ (\frac{z_{1} − ρz_{2}}{\sqrt{1 − ρ^{2}}}) dz_{1}dz_{2}

= σ_{1}σ_{2} ∫_{−∞}^{∞} z_{2} \frac{ρz_{2}}{\sqrt{1 − ρ^{2}}}ϕ(z_{2}) ∫_{−∞}^{∞} ϕ (\frac{z_{1} − ρz_{2}}{\sqrt{1 − ρ^{2}}}) dz_{1}dz_{2}

= ρσ_{1}σ_{2} ∫_{−∞}^{∞} z_{2}^{2}ϕ(z_{2})dz_{2} = ρσ_{1}σ_{2}.

Thus, Corr (X_{1}, X_{2}) = ρ. If ρ = 0 then X_{1} and X_{2} are uncorrelated. But in this case they are also independent since

f(x; μ, Σ) = \frac{1}{σ_{1}}ϕ (\frac{x_{1} − μ_{1}}{σ_{1}}) \frac{1}{σ_{2}}ϕ (\frac{x_{2} − μ_{2}}{σ_{2}}) .  (1)

On the other hand, if X_{1} and X_{2} are independent then (1) holds and, therefore, ρ = 0. By the way,

Σ = ( \begin{matrix}V ar X_{1} & Cov (X_{1}, X_{2}) \\ Cov (X_{1}, X_{2}) & V ar X_{2} \end{matrix} ) = E[(X − μ)(X − μ)′].

Remark. For a singular normal distribution |E[(X − μ)(X − μ)′]| = 0. Furthermore, P (X_{1} = cX_{2} + b) = 1 for some c ≠ 0 and b. Thus, |Corr (X_{1}, X_{2})| = 1.

1.7 Conditional distributions. Regression

The conditional distribution of X_{1} given X_{2} = x_{2} is N(μ_{1}+ρ(σ_{1}/σ_{2})(x_{2}− μ_{2}), σ_{1}^{2}(1 − ρ^{2})). Indeed, the conditional density of X_{1} given X_{2} = x_{2} is

f_{X_{1}|X_{2}}(x) = \frac{f(x; μ, Σ)}{f_{X_{2}}(x_{2})} = \frac{1}{σ_{1}\sqrt{1 − ρ^{2}}}ϕ (\frac{x_{1} − (μ_{1} + ρ(σ_{1}/σ_{2})(x_{2} − μ_{2}))}{σ_{1}\sqrt{1 − ρ^{2}}}) .

By symmetry, the conditional distribution of X_{2} given X_{1} = x_{1} is N(μ_{2} + ρ(σ_{2}/σ_{1})(x_{1} − μ_{1}), σ_{2}^{2}(1 − ρ^{2})).

In other words,

E(X_{2}|X_{1} = x_{1}) = μ_{2} + ρ\frac{σ_{2}}{σ_{1}}(x_{1} − μ_{1}),

V ar (X_{2}|X_{1} = x_{1}) = σ_{2}^{2}(1 − ρ^{2}).

A random variable E(X_{2}|X_{1} = x_{1}) is called the regression of X_{2} on X_{1}. It is a function of x_{1} only. Similarly, E(X_{1}|X_{2} = x_{2}) is called the regression of X_{1} on X_{2}.

As we see, in case of a normal distribution regressions are the linear functions! The first who obtained those relations was F.Galton.

Assume that X = (X_{1}, X_{2})′ ∼ N_{2}(00 0, Σ), where X_{1} and X_{2} are the deviations of father’s and son’s heights from their mean values, respectively. Then as we know X_{2} ∼ N(0, σ_{2}^{2}). But if we fix the height of fathers (X_{1} = x_{1}), then X_{2}|X_{1} = x_{1} ∼ N(ρ(σ_{2}/σ_{1})x_{1}, σ_{2}^{2}(1 − ρ^{2})) and σ_{2}^{2}(1 − ρ^{2}) < σ_{2}^{2}. That is why F.Galton used the word ”regression”.

1.8 Linear transformations

Let X ∼ N_{2}(μ, Σ),

C = ( \begin{matrix}c_{11} & c_{12} \\ c_{21} & c_{22} \end{matrix} ) , b = (b_{1}, b_{2})′.

Then Y = CX + b ∼ N_{2}(Cμ + b, CΣC′).

If |C| ≠ 0 :

f_{Y}(u) = |C^{−1}|f_{X}(C^{−1}(u − b)); μ, Σ)

= \frac{1}{2π|C||Σ|^{1/2}} exp\{−(C^{−1}(u − b) − μ)′Σ^{−1}(C^{−1}(u − b) − μ)/2\}

= \frac{1}{2π|CΣC′|^{1/2}} exp\{−(u − b − Cμ)′(C^{−1})′Σ^{−1}C^{−1}(u − b − Cμ)/2\}

= \frac{1}{2π|CΣC′|^{1/2}} exp\{−(u − [Cμ + b])′(CΣC′)^{−1}(u − [Cμ + b])/2\}

= f_{X}(u; Cμ + b, CΣC′).

In a particular case taking b = −Cμ and

C = ( \begin{matrix}(1 − ρ)^{−1/2} & 0 \\ 0 & (1 + ρ)^{−1/2} \end{matrix} ) ( \begin{matrix}1/\sqrt{2} & −1/\sqrt{2} \\ 1/\sqrt{2} & 1/\sqrt{2} \end{matrix} ) ( \begin{matrix}σ_{1}^{−1} & 0 \\ 0 & σ_{2}^{−1} \end{matrix} ) ,

we obtain CΣC′ = I_{2} (I_{2} denotes the identity 2 × 2 matrix) and Y = CX + b ∼ N_{2}(00 0, I_{2}).

By reversing this process we can always obtain a bivariate normal vector X with any mean vector μ, variances σ_{1}^{2}, σ_{2}^{2} and correlation coefficient ρ ∈ (−1, 1) through a transformation of two independent N(0, 1) variables.

1.9 Characterizations

1. Let U_{1}, U_{2} be independent uniform [0, 1] variables, and define

Y_{1} = \sqrt{−2 ln U_{1}} cos(2πU_{2}), Y_{2} = \sqrt{−2 ln U_{1}} sin(2πU_{2}).

Then Y_{1}, Y_{2} are independent N(0, 1) variables, that is Y = (Y_{1}, Y_{2})′ ∼ N_{2}(00 0, I_{2}). Indeed,

U_{1} = exp\{−(Y_{1}^{2} + Y_{2}^{2})/2\}, U_{2} = \frac{1}{2π}arc cot \frac{Y_{1}}{Y_{2}}.

Since for U = (U_{1}, U_{2})′ the density function has the form

f_{U}(u_{1}, u_{2}) = 1_{[0,1]}(u_{1})1_{[0,1]}(u_{2}),

we obtain

f_{Y}(y_{1}, y_{2}) = |A|f_{U}(h_{1}(u_{1}, u_{2}), h_{2}(u_{1}, u_{2})) = \frac{1}{2π}e^{−(y_{1}^{2}+y_{2}^{2})/2} = ϕ(y_{1})ϕ(y_{2}).

Therefore, X ∼ N_{2}(μ, Σ) iff it is a random vector corresponding to the following experiment:

(i) Observe a value of a random variable V from an exponential distribution with mean 2;

(ii) for given V = v, observe a point at random on the circle with radius \sqrt{v}, and identify this point in ℝ^{2} as Y = (Y_{1}, Y_{2})′;

(iii) apply the transformation X = CY+μ, where C satisfies CC′ = Σ.

2. X = (X_{1}, X_{2})′ has a standard bivariate normal distribution with correlation ρ iff

X_{1} = \sqrt{1 − ρ^{2}}Y + ρX_{2},

where Y and X_{2} are independent standard normal variables.

Generalization: X = (X_{1}, X_{2})′ ∼ N_{2}(μ, Σ) iff

X_{1} = σ_{1}(\sqrt{1 − ρ^{2}}Y_{1} + ρY_{2}) + μ_{1}, X_{2} = σ_{2}Y_{2} + μ_{2},

where Y_{1} and Y_{2} are independent standard normal variables.

3. Let Z_{0}, Z_{1}, Z_{2} be independent N(0, 1) variables. For arbitrarily but fixed μ and Σ, let

X_{1} = σ_{1}(\sqrt{1 − |ρ|}Z_{1} + \sqrt{|ρ|}Z_{0}) + μ_{1},
X_{2} = σ_{2}(\sqrt{1 − |ρ|}Z_{2} + δ_{ρ}\sqrt{|ρ|}Z_{0}) + μ_{2},

where

δ_{ρ} = \{ \begin{matrix}1 & if ρ ≥ 0 \\ −1 & if ρ < 0. \end{matrix}

Then X = (X_{1}, X_{2})′ ∼ N_{2}(μ, Σ).

4. X = (X_{1}, X_{2})′ has a bivariate normal distribution iff any linear function of X_{1} and X_{2} has a normal distribution.

5. Let X_{1} and X_{2} are independent, Y_{1} = a_{1}X_{1} + a_{2}X_{2}, Y_{2} = b_{1}X_{1} + b_{2}X_{2} and coefficients a_{i}, b_{i} are non-zeros, i = 1, 2. If Y_{1} and Y_{2} are independent then X_{1}, X_{2}, Y_{1}, Y_{2} are normal.

1.11 Simulation

Simulation studies constitute an integral part of statistical computing, and may be applied to yield empirical results when analytical results are difficult to obtain.

In most simulation studies we begin with the generation of ”random numbers”. A sequence of (pseudo) independent uniform [0, 1] variables U_{1}, U_{2}, … is usually called a sequence of random numbers. The generation of such a sequence on a computer involves an algorithm for generating a sequence of real numbers deterministically, given the seed number. Since this is a well-researched area, we will assume that there already exists a built-in random number generator on the computer that can generate enough random numbers U_{1}, U_{2}, …

1. A Box-Muller method of generating N_{2}(00 0, I_{2}) random vectors (see 1.9, p. 1). Having independent uniform [0, 1] variables U_{1}, U_{2}, we calculate

X_{1} = \sqrt{−2 ln U_{1}} cos(2πU_{2}), X_{2} = \sqrt{−2 ln U_{1}} sin(2πU_{2}).

It is easy to understand that in such a way we can also produce a random vector having the distribution N_{2}(μ, Σ) with ρ = 0. It is enough to calculate

X_{1} = μ_{1} + σ_{1}\sqrt{−2 ln U_{1}} cos(2πU_{2}), X_{2} = μ_{2} + σ_{2}\sqrt{−2 ln U_{1}} sin(2πU_{2}).

2a. A method of generating N_{2}(μ, Σ) random vectors from N_{2}(00 0, I_{2}) random vectors (see 1.9, p. 1 and 1.10, p. 2). Having Y ∼ N_{2}(00 0, I_{2}), let us calculate X = CY + μ, where C satisfies CC′ = Σ. The matrix C, for instance, can be taken as

C = ( \begin{matrix}σ_{1}\sqrt{1 − ρ^{2}} & σ_{1}ρ \\ 0 & σ_{2} \end{matrix} ) .

2b. A method of generating N_{2}(μ, Σ) random vectors from independent N(0, 1) random variables (see 1.10, p. 3). Having independent N(0, 1) variables Z_{0}, Z_{1}, Z_{2} let us calculate

X_{1} = σ_{1}(\sqrt{1 − |ρ|}Z_{1} + \sqrt{|ρ|}Z_{0}) + μ_{1},
X_{2} = σ_{2}(\sqrt{1 − |ρ|}Z_{2} + δ_{ρ}\sqrt{|ρ|}Z_{0}) + μ_{2},

where

δ_{ρ} = \{ \begin{matrix}1 & if ρ ≥ 0 \\ −1 & if ρ < 0. \end{matrix}

1.13 Sampling distributions

Let X_{1}, …, X_{n} be a random sample of size n from an N_{2}(μ, Σ) distribution, that is X_{1}, …, X_{n} are independent N_{2}(μ, Σ) vectors, X_{m} = (X_{1m}, X_{2m})′, m = 1, …, n. The vector X̄ = (X̄_{1}, X̄_{2})′, where

X̄_{i} = \frac{1}{n} ∑_{m=1}^{n} X_{im}, i = 1, 2,

is called the sample mean.

The distribution of X̄ is N_{2}(μ, (1/n)Σ).

As we know, the random variables

X_{1} − μ_{1} − ρ(σ_{1}/σ_{2})(X_{2} − μ_{2}) ∼ N(0, σ_{1}^{2}(1 − ρ^{2})), X_{2} − μ_{2} ∼ N(0, σ_{2}^{2})

are independent (see 1.4). Therefore,

X̄_{1} − μ_{1} − ρ(σ_{1}/σ_{2})(X̄_{2} − μ_{2}) ∼ N(0, \frac{σ_{1}^{2}(1 − ρ^{2})}{n}), X̄_{2} − μ_{2} ∼ N(0, \frac{σ_{2}^{2}}{n})

are independent. Thus,

Y_{1} = \frac{[X̄_{1} − μ_{1} − ρ(σ_{1}/σ_{2})(X̄_{2} − μ_{2})]\sqrt{n}}{σ_{1}\sqrt{1 − ρ^{2}}}, Y_{2} = \frac{[X̄_{2} − μ_{2}]\sqrt{n}}{σ_{2}}

are independent and N(0, 1) distributed. So, f_{Y}(y_{1}, y_{2}) = ϕ(y_{1})ϕ(y_{2}). From this formula coming back to the variables X̄_{1} and X̄_{2}, we obtain (cf. 1.4)

fX̄(x_{1}, x_{2})= \frac{n}{σ_{1}σ_{2}\sqrt{1 − ρ^{2}}}ϕ(\frac{[x_{2} − μ_{2}]\sqrt{n}}{σ_{2}})ϕ(\frac{[x_{1} − μ_{1} − ρ(σ_{1}/σ_{2})(x_{2} − μ_{2})]\sqrt{n}}{σ_{1}\sqrt{1 − ρ^{2}}}) .

An immediate corollary: n(X̄ − μ)′Σ^{−1}(X̄ − μ) ∼ χ^{2}(2).

Indeed, let C be a non-singular 2 × 2 matrix such that CΣC′ = I_{2}. Define Z = \sqrt{n}C(X̄ − μ). Then by 1.8, Z ∼ N_{2}(00 0, I_{2}) and, therefore, Z′Z ∼ χ^{2}(2). But by \sqrt{n}(X̄ − μ) = C^{−1}Z we have

n(X̄ − μ)′Σ^{−1}(X̄ − μ) = Z′(C^{−1})′Σ^{−1}C^{−1}Z = Z′(CΣC′)^{−1}Z = Z′Z.

The matrix

S = ( \begin{matrix}S_{11} & S_{12} \\ S_{12} & S_{22} \end{matrix} ) , S_{ij} = \frac{1}{n − 1} ∑_{m=1}^{n}(X_{im}−X̄_{i})(X_{jm}−X̄_{j}), i, j = 1, 2,

is called the sample covariance matrix. Its distribution was established by J.Wishart (1928). Proposition 1. The density function of (n − 1)S, n > 2, is given by

f(W) = \frac{|W|^{(n−4)/2}}{c_{n−1}|Σ|^{(n−1)/2}}e^{−tr Σ^{−1}W/2}

for W in the set of all 2 × 2 positive definite matrices,

c_{n−1} = 2^{n−1}\sqrt{π}Γ (\frac{n − 1}{2}) Γ (\frac{n − 2}{2}) .

In fact, the Wishart distribution is the joint distribution of the 3 variables (n−1)S_{11}, (n−1)S_{12}, (n−1)S_{22}, which are the elements of random matrix (n − 1)S. It is easy to see that (n − 1)S = XX′ − nX̄X̄′. It can be shown that there exist independent N_{2}(00 0, Σ) random vectors Y_{1}, Y_{2}, …, Y_{n−1} such that the distributions of (n − 1)S and ∑_{m=1}^{n−1} Y_{m}Y′_{m} are the same. Therefore, for any a ∈ ℝ^{2} the distribution of (n − 1)a′Sa is (a′Σa)χ^{2}(n − 1). In univariate case S is the sample variance, and, as it is known, (n − 1)S has an σ^{2}χ^{2}(n − 1) distribution.

Making statistical inference on μ based on X̄, the covariance matrix Σ must be known. If Σ is unknown then a new statistic is needed. This was proposed and studied by H.Hotelling (1931): T^{2} = n(X̄−μ)′S^{−1}(X̄−μ). Proposition 2. The statistic ((n − 2)/((n − 1)2))T^{2}, n > 2, has an F (2, n − 2) distribution.

The statistic

r = \frac{S_{12}}{\sqrt{S_{11}S_{22}}} = \frac{∑_{m=1}^{n}(X_{1m} − X̄_{1})(X_{2m} − X̄_{2})}{[∑_{m=1}^{n}(X_{1m} − X̄_{1})^{2} ∑_{m=1}^{n}(X_{2m} − X̄_{2})^{2}]^{1/2}}

is known as the sample correlation coefficient. The distribution of r does not depend on μ, σ_{1}, σ_{2}. Proposition 3. If ρ = 0 and n ≥ 4 then the density function of r is

h(u) = \frac{Γ((n − 1)/2)}{\sqrt{π}Γ((n − 2)/2)}(1 − u^{2})^{(n−4)/2}, u ∈ (−1, 1).

Equivalently, r\sqrt{n − 2}/\sqrt{1 − r^{2}} ∼ t_{n−2}.

If ρ ≠ 0 then the density function of r is

h_{ρ}(u) = \frac{2^{n−3}(1 − ρ^{2})^{(n−1)/2}}{π(n − 3)!}(1−u^{2})^{(n−4)/2} ∑_{j=0}^{∞} \frac{(2ρu)^{j}}{j!} Γ^{2} (\frac{n + j − 1}{2}) , u ∈ (−1, 1).

R.A.Fisher (1915) proposed another form of h_{ρ}(u):

h_{ρ}(u) = \frac{(1 − ρ^{2})^{(n−1)/2}}{π(n − 3)!}(1 − u^{2})^{(n−4)/2} [ \frac{∂^{n−2}}{∂x^{n−2}} \frac{arccos(−x)}{\sqrt{1 − x^{2}}}|_{x=ρu}] .

R.A.Fisher (1921) also proposed an approximation if n is large. Proposition 4. As n → ∞, the limiting distribution of \sqrt{n − 3}(z − a) is N(0, 1), where

z = \frac{1}{2} ln (\frac{1 + r}{1 − r}) , a = \frac{1}{2} ln (\frac{1 + ρ}{1 − ρ}) .

Given the sample correlation coefficient r, we can test the hypothesis H_{0} : ρ = 0 of no correlation against the alternative H_{1} : ρ ≠ 0 (or ρ > 0 or ρ < 0) with the probability α of a first type error by the rule:

accept H_{0} if |r|\sqrt{\frac{n − 2}{1 − r^{2}}} ≤ t_{α/2;n−2}

accept H_{1} if |r|\sqrt{\frac{n − 2}{1 − r^{2}}} > t_{α/2;n−2},

where t_{α/2;n−2} is the α/2-th percentage point of t_{n−2} distribution such that P (t_{n−2} > t_{α/2;n−2}) = α/2.

If n is large then we adopt Proposition 4 for testing hypothesis. Namely:

accept H_{0} if |z|\sqrt{n − 3} ≤ z_{α/2}
accept H_{1} if |z|\sqrt{n − 3} > z_{α/2},

where z_{α/2} is the α/2-th percentage point of N(0, 1) distribution such that P (N(0, 1) > z_{α/2}) = α/2.

The reformulation of the rules in case of the alternative ρ > 0 or ρ < 0 is evident.

1.17 Exercises

1. Assuming μ_{1} = μ_{2} and σ_{1} = σ_{2}, show that f(x; μ, Σ) is a decreasing function of |x_{1} − x_{2}| for every fixed x_{1} + x_{2} = λ.

2. Calculate the density function of the random variable R = \sqrt{X_{1}^{2} + X_{2}^{2}}, where X = (X_{1}, X_{2})′ ∼ N_{2}(00 0, I_{2}) (Rayleigh distribution).

3. An expert marksman firing at a target produces a random scatter of shots with approximately 50% of the shots in the bull’s eye. Assuming that the coordinates of a shot (X, Y) are N_{2}(00 0, I_{2}) distributed,

(i) find the approximate fraction of shots inside a circle with the same center as the bull’s eye, but twice the radius; (ii) find the approximate average distance of the marksman’s shots from the center of the bull’s eye.

4. Let Σ be a non-singular covariance matrix given in the definition 1.3. Find a 2 × 2 orthogonal matrix C such that CΣC′ is a diagonal matrix with positive diagonal elements. Find also a 2 × 2 matrix H such that HΣH′ = I_{2}.

5. Calculate P (X_{1} ≤ 0, X_{2} ≤ 0) for a standard bivariate normal vector X = (X_{1}, X_{2})′.

6. A study on the resemblances between fathers and sons have been curried out. It is found out that μ_{1} = 175cm (mean height of fathers), μ_{2} = 178cm (mean height of sons), and σ_{1} = σ_{2} = 6cm (standard deviations). Assuming that height of fathers and height of sons are normally distributed with ρ = 0, 5 predict the height of the son of a father who is 188cm tall. What is the chance that your prediction is off by more than 2, 5cm?

7. Show that if X = (X_{1}, X_{2})′ has a standard bivariate normal distribution then

E(max\{X_{1}, X_{2}\}) = \sqrt{\frac{1 − ρ}{π}}.

8. Show that r\sqrt{n − 2}/\sqrt{1 − r^{2}} ∼ t_{n−2}, where r is the sample correlation coefficient and ρ = 0 (see 1.15, Proposition 3).

9. Show that both formulae for h_{ρ}(u), ρ ≠ 0 (see 1.15) reduce to h(u) given in Proposition 3.

10. Assuming ρ = 0, let r be the sample correlation coefficient with n = 28 and let r_{α} be the α-th percentage point such that P (r > r_{α}) = α. Find r_{α} for α = 0, 05 and α = 0, 01 using Student’s t table. Use Proposition 4 and the normal table to approximate r_{α} for large values of n.

11. How many observations we should collect in order to reject the null hypothesis ρ = 0 for the alternative hypothesis ρ > 0 if r = 0, 2 and α = 0, 05?