2.1 Definitions

Definition 1. A d-dimensional random vector X = (X_{1}, …, X_{d})′ with mean vector μ and covariance matrix Σ is said to have a non-singular multivariate normal distribution, in symbols X ∼ N_{d}(μ, Σ), if Σ > 0 and the density function of X is of the form

f(x; μ, Σ) = \frac{1}{(2π)^{d/2}|Σ|^{1/2}} exp \{−(x − μ)′Σ^{−1}(x − μ)/2\} , x ∈ ℝ^{d},

where μ = (μ_{1}, …, μ_{d})′,

Σ = ( \begin{matrix}σ_{1}^{2} & σ_{12} & … & σ_{1d} \\ σ_{12} & σ_{2}^{2} & … & σ_{2d} \\ … & … & … & … \\ σ_{1d} & σ_{2d} & … & σ_{d}^{2} \end{matrix} ) .

For this definition to be consistent, we must verify that if X has the density function f(x; μ, Σ), then the mean vector and the covariance matrix of X are indeed μ and Σ, respectively. This is postponed and will be given in 2.3.

Let X ∼ N_{d}(μ, Σ), Σ > 0, and consider the linear transformation Y = CX + b, where C = (c_{ij}) is an d × d non-singular real matrix and b ∈ ℝ^{d}. Then similarly as in the case d = 2 (see 1.8) it can be shown that Y ∼ N_{d}(Cμ + b, CΣC′), CΣC′ > 0.

The reverse process (see again 1.8) also holds, therefore: X ∼ N_{d}(μ, Σ), Σ > 0, iff there exists a non-singular d × d matrix C such that CC′ = Σ and X =^{d} CZ + μ, where Z ∼ N_{d}(00 0, I_{d}).

Definition 2. A d-dimensional random vector X = (X_{1}, …, X_{d})′ with mean vector μ and covariance matrix Σ is said to have a singular multivariate normal distribution if Σ ≥ 0 and for some r < d there exists a d × r real matrix C such that X =^{d} CZ_{r} + μ, where Z_{r} ∼ N_{r}(00 0, I_{r}).

2.2 More definitions

A singular bivariate normal distribution is concentrated on a subspace of the dimension smaller than d.

Now we can combine both definitions and say that a d-dimensional random vector X with mean vector μ and covariance matrix Σ has a multivariate normal distribution if either Definition 1 or Definition 2 holds. This is equivalent to:

Definition 3. A d-dimensional random vector X with mean vector μ and covariance matrix Σ is said to have a multivariate normal distribution if there exists a d × r real matrix C with rank r ≤ d such that CC′ = Σ and X =^{d} CZ_{r} + μ, where Z_{r} ∼ N_{r}(00 0, I_{r}).

The next definition is also possible and equivalent to the previous one.

Definition 4. A d-dimensional random vector X with mean vector μ and covariance matrix Σ is said to have a multivariate normal distribution if a′X ∼ N(a′μ, a′Σa) for all a ∈ ℝ^{d}.

2.3 Marginal distributions

For fixed k < d consider the partitions of X, μ, Σ:

X = ( \begin{matrix}X_{1} \\ X_{2} \end{matrix} ) , μ = ( \begin{matrix}μ & 1 \\ μ & 2 \end{matrix} ) , Σ = ( \begin{matrix}Σ_{11} & Σ_{12} \\ Σ′_{12} & Σ_{22} \end{matrix} ) ,

where X_{1} = (X_{1}, …, X_{k})′, X_{2} = (X_{k+1}, …, X_{d})′, μ_{1} = (μ_{1}, …, μ_{k})′, μ_{2} = (μ_{k+1}, …, μ_{d})′, Σ_{ii} is the covariance matrix of X_{i}, i = 1, 2, and Σ_{12} = (Cov (X_{i}, X_{j})).

Proposition 1. If X ∼ N_{d}(μ, Σ) then the marginal distributions of X_{1} and X_{2} are N_{k}(μ_{1}, Σ_{11}) and N_{d−k}(μ_{2}, Σ_{22}), respectively.

From linear algebra it follows that Σ > 0 yields Σ_{11} > 0, Σ_{22} > 0.

From Proposition 1 it follows immediately that the mean vector and the covariance matrix of X are μ and Σ, respectively. This observation shows that Definition 1 is indeed consistent.

Let a matrix R = (ρ_{ij}) be such that

ρ_{ij} = \frac{σ_{ij}}{σ_{i}σ_{j}} , i, j = 1, …, d.

Then R is called the correlation matrix of X. The diagonal elements of R are 1 and the off-diagonal elements are the correlation coefficients. Furthermore, X_{1} and X_{2} are independent iff Σ_{12} = 0.

2.4 Linear transformations

If X ∼ N_{d}(μ, Σ) and Y = CX + b, where C is any given m × d real matrix and b ∈ ℝ^{m}, then Y ∼ N_{m}(Cμ + b, CΣC′). For m = d the proof is similar to that in 1.8. For m < d consider the transformation

( \begin{matrix}Y \\ Y_{2} \end{matrix} ) = ( \begin{matrix}C \\ B \end{matrix} ) X + ( \begin{matrix}b \\ 0 & 0 & 0 & d−m \end{matrix} ) ,

where B is any given (d − m) × d matrix. Therefore,

( \begin{matrix}Y \\ Y_{2} \end{matrix} ) ∼ N_{d} (( \begin{matrix}Cμ + b \\ Bμ \end{matrix} ) , ( \begin{matrix}CΣC′ & CΣB′ \\ BΣC′ & BΣB′ \end{matrix} ))

and Y = CX + b ∼ N_{m}(Cμ + b, CΣC′). For m > d, by Definition 3, there exists a d × r real matrix C^{∗} such that X =^{d} C^{∗}Z_{r} + μ, where r ≤ d is the rank of Σ and Z_{r} ∼ N_{r}(00 0, I_{r}). Thus CX + b =^{d} CC^{∗}Z_{r} + (Cμ + b) and has a singular N_{m}(Cμ + b, CΣC′) distribution again by Definition 3. Note that if Σ > 0 and C is m × d real matrix with rank m ≤ d, then CΣC′ > 0. Let C_{1}, C_{2} be m × k and m × (d − k) matrices, respectively. If X ∼ N_{d}(μ, Σ) then Y = C_{1}X_{1} + C_{2}X_{2} ∼ N_{m}(μ_{Y}, Σ_{Y}), where μ_{Y} = C_{1}μ_{1} + C_{2}μ_{2}, Σ_{Y} = C_{1}Σ_{11}C′_{1} + C_{2}Σ_{22}C′_{2} + C_{1}Σ_{12}C′_{2} + C_{2}Σ′_{12}C′_{1}.

2.5 Conditional distributions

For X_{1} and X_{2} as in 2.3 consider the linear transformation

Y = ( \begin{matrix}Y_{1} \\ Y_{2} \end{matrix} ) = ( \begin{matrix}I_{k} & −B \\ 00 0 & I_{d−k} \end{matrix} ) ( \begin{matrix}X_{1} \\ X_{2} \end{matrix} ) = CX,

where Y_{1} and Y_{2} are k-dimensional and (d − k)-dimensional random vectors, respectively, and B is a k × (d − k) real matrix. If X ∼ N_{d}(μ, Σ), Σ > 0, then Y ∼ N_{d}(μ_{Y}, Σ_{Y}) by 2.4, where

μ_{Y} = ( \begin{matrix}μ_{1} − Bμ_{2} \\ μ & 2 \end{matrix} ) ,

Σ_{Y} = ( \begin{matrix}Σ_{11} + BΣ_{22}B′ − BΣ′_{12} − Σ_{12}B′ & Σ_{12} − BΣ_{22} \\ (Σ_{12} − BΣ_{22})′ & Σ_{22} \end{matrix} ) .

If we choose B to satisfy Σ_{12} − BΣ_{22} = 00 0, that is if B = Σ_{12}Σ_{22}^{−1}, then Y_{1} and Y_{2} are uncorrelated (and thus independent), and we have

( \begin{matrix}Y_{1} \\ Y_{2} \end{matrix} ) = ( \begin{matrix}X_{1} − Σ_{12}Σ_{22}^{−1}X_{2} \\ X_{2} \end{matrix} ) ∼ N_{d} (( \begin{matrix}μ & 1·2 \\ μ & 2 \end{matrix} ) , ( \begin{matrix}Σ_{11·2} & 00 0 \\ 00 0 & Σ_{22} \end{matrix} )) ,

where μ_{1·2} = μ_{1} − Σ_{12}Σ_{22}^{−1}μ_{2}, Σ_{11·2} = Σ_{11} − Σ_{12}Σ_{22}^{−1}Σ′_{12}. Therefore, Y_{1} = X_{1} − Σ_{12}Σ_{22}^{−1}X_{2} and Y_{2} = X_{2} are independent normal vectors with the densities (cf. 1.4)

f_{Y_{1}}(y_{1}) = \frac{1}{(2π)^{k/2}|Σ_{11·2}|^{1/2}} exp \{−(y_{1} − μ_{1·2})′Σ_{11·2}^{−1}(y_{1} − μ_{1·2})/2\} ,

f_{Y_{2}}(y_{2}) = \frac{1}{(2π)^{(d−k)/2}|Σ_{22}|^{1/2}} exp \{−(y_{2} − μ_{2})′Σ_{22}^{−1}(y_{2} − μ_{2})/2\} ,

respectively.

Since

( \begin{matrix}X_{1} \\ X_{2} \end{matrix} ) = ( \begin{matrix}I_{k} & −Σ_{12}Σ_{22}^{−1} \\ 00 0 & I_{d−k} \end{matrix} )^{−1} ( \begin{matrix}Y_{1} \\ Y_{2} \end{matrix} ) = ( \begin{matrix}I_{k} & Σ_{12}Σ_{22}^{−1} \\ 00 0 & I_{d−k} \end{matrix} ) ( \begin{matrix}Y_{1} \\ Y_{2} \end{matrix} ) ,

we obtain

f_{X}(x_{1}, x_{2}) = f_{Y}(x_{1} − Σ_{12}Σ_{22}^{−1}x_{2}, x_{2}) = f_{Y_{1}}(x_{1} − Σ_{12}Σ_{22}^{−1}x_{2})f_{Y_{2}}(x_{2}).

The density f_{Y_{2}}(x_{2}) = f_{X_{2}}(x_{2}) is the marginal density of X_{2}. Therefore, the conditional distribution of X_{1} given X_{2} = x_{2} is N_{k}(μ_{1} + Σ_{12}Σ_{22}^{−1}(x_{2} − μ_{2}), Σ_{11·2}) (cf. 1.7). Similarly, we can write the conditional distribution of X_{2} given X_{1} = x_{1}.

2.6 Regression

As you remember (cf. 1.7), E(X_{1}|X_{2} = x_{2}) is called the regression of X_{1} on X_{2}. We have

E(X_{1}|X_{2} = x_{2}) = μ_{1} + Σ_{12}Σ_{22}^{−1}(x_{2} − μ_{2}),

and the covariance matrix of X_{1} given X_{2} = x_{2} is Σ_{11·2}. Again we see that E(X_{1}|X_{2} = x_{2}) is a linear function of x_{2} while the covariance matrix does not depend on x_{2}. Assume that k = 1 and try to find the best predictor of X_{1} given X_{2} = x_{2}, that is to find a function λ^{∗}(x_{2}), for which

inf_{λ} E[(X_{1} − λ(x_{2}))^{2}|X_{2} = x_{2}] = E[(X_{1} − λ^{∗}(x_{2}))^{2}|X_{2} = x_{2}]

for all x_{2} ∈ ℝ^{d−k}. We have

E[(X_{1} − λ(x_{2}))^{2}|X_{2} = x_{2}] = E[((X_{1} − E[X_{1}|X_{2} = x_{2}])
+(E[X_{1}|X_{2} = x_{2}] − λ(x_{2})))^{2}|X_{2} = x_{2}]
= V ar [X_{1}|X_{2} = x_{2}] + (E[X_{1}|X_{2} = x_{2}] − λ(x_{2}))^{2}.  (1)

The first term in (1), as we know, does not depend on x_{2}. Therefore, in order to minimize (1) it is enough to take λ(x_{2}) = E[X_{1}|X_{2} = x_{2}]. So. the best predictor is given by a linear function!

2.7 Multiple and partial correlation coefficients

Let X_{1} and X_{2} be as in 2.3. In certain applications, we are interested in the best linear combination of the components of X_{2} such that the correlation coefficient between X_{i}, 1 ≤ i ≤ k, and a′X_{2} is maximized.

Definition 5. The multiple correlation coefficient between X_{i}, 1 ≤ i ≤ k, and X_{2} is defined by

R_{i·k+1,...,d} = sup_{a∈ℝ^{d−k}} Corr (X_{i}, a′X_{2}).  (2)

Proposition 2. The vector a which maximizes the right-hand side of (2) is the same vector that yields the best predictor for X_{i} when X_{2} = x_{2} is given, namely a′ = σ_{i}Σ_{22}^{−1}, where σ_{i} = (σ_{i,k+1}, …, σ_{i,d}) is the i-th row of the submatrix Σ_{12}, and

R_{i·k+1,...,d} = \frac{(σ_{i}Σ_{22}^{−1}σ′_{i})^{1/2}}{σ_{i}}.

As we see, R_{i·k+1,...,d} ∈ [0, 1]. Furthermore, for a non-singular distribution R_{i·k+1,...,d} = 0 iff σ_{i} = 00 0, that is iff X_{i} and X_{2} are independent.

The partial correlation coefficient between two random variables is their correlation coefficient after allowing for the effects of a set of other variables.

Definition 6. For given X_{2} = x_{2} the partial correlation coefficient between X_{i} and X_{j}, i ≠ j, i, j = 1, …, k, is given by

ρ_{ij·k+1,...,d} = \frac{Cov ((X_{i}, X_{j})|X_{2} = x_{2})}{(V ar (X_{i}|X_{2} = x_{2}) · V ar (X_{j}|X_{2} = x_{2}))^{1/2}}.

Proposition 3.

ρ_{ij·k+1,...,d} = \frac{σ_{ij} − σ_{i}Σ_{22}^{−1}σ′_{j}}{((σ_{i}^{2} − σ_{i}Σ_{22}^{−1}σ′_{i}) · (σ_{j}^{2} − σ_{j}Σ_{22}^{−1}σ′_{j}))^{1/2}}.

2.8 Partial correlation coefficients. Example

As we see, partial correlation coefficient is non-negative iff σ_{ij} ≥ σ_{i}Σ_{22}^{−1}σ′_{j}. Thus it is possible to have a covariance matrix Σ such that the correlation coefficient between X_{i} and X_{j} (which is ρ_{ij} = σ_{ij}/(σ_{i}σ_{j})) is positive while the partial correlation coefficient is negative.

Example. Let X = (X_{1}, X_{2}, X_{3})′ be distributed according to N_{3}(00 0, Σ) distribution, where

Σ = ( \begin{matrix}1 & 1 − 2ε & 1 − ε \\ 1 − 2ε & 1 & 1 − ε \\ 1 − ε & 1 − ε & 1 \end{matrix} ) , 0 < ε < \frac{1}{2}.

One can verify that Σ > 0. The conditional distribution of (X_{1}, X_{2})′ given X_{3} = x_{3} is bivariate normal with the covariance matrix

Σ_{11·2} = ( \begin{matrix}1 & 1 − 2ε \\ 1 − 2ε & 1 \end{matrix} ) − (1 − ε)^{2} ( \begin{matrix}1 & 1 \\ 1 & 1 \end{matrix} ) = ε ( \begin{matrix}2 − ε & −ε \\ −ε & 2 − ε \end{matrix} ) .

Thus ρ_{12} = 1 − 2ε > 0 while ρ_{12·3} = −ε/(2 − ε) < 0.

2.9 A couple of interesting properties

1. If X = (X_{1}, …, X_{d})′ is a normally distributed random vector then there exists a vector a ∈ ℝ^{d}, ∥a∥ = 1, such that

V ar (a_{1}X_{1} + … + a_{d}X_{d}) ≥ V ar (b_{1}X_{1} + … + b_{d}X_{d})

for all b ∈ ℝ^{d}, ∥b∥ = 1. If a = (1, 0, …, 0)′ then X_{1} and all X_{j}, j > 1, are independent.

2. If X is normally distributed then there exists an orthogonal matrix C such that the components of the random vector CX are independent.

2.10 Sampling distributions

Let X_{1}, …, X_{n} be a random sample of size n from an N_{d}(μ, Σ) distribution, that is X_{1}, …, X_{n} are independent N_{d}(μ, Σ) random vectors.

(i) The sample mean vector X̄ has an N_{d}(μ, (1/n)Σ) distribution.

(ii) The scaled sample covariance matrix (n−1)S, n > d, has the Wishart distribution with the density

f(W) = \frac{|W|^{(n−d−2)/2}}{c_{n−1}|Σ|^{(n−1)/2}}e^{−tr Σ^{−1}W/2}

for W in the set of all d × d positive definite matrices and

c_{n−1} = 2^{d(n−1)/2}π^{d(d−1)/4} ∏_{j=1}^{d} Γ (\frac{n − j}{2} ) .

(iii) The statistic ((n − d)/((n − 1)d))T^{2} has an F (d, n − d) distribution, where T^{2} = n(X̄ − μ)′S^{−1}(X̄ − μ) is Hotelling’s statistic, n > d.

(iv) Consider the partition of the vector X as X_{1} and X_{2} = (X_{2}, …, X_{d})′ and a corresponding partition of the sample covariance matrix given by

S = ( \begin{matrix}S_{11} & S_{12} \\ S′_{12} & S_{22} \end{matrix} ) .

The sample mutiple correlation coefficient (see Definition 5 in 2.7) is

R̂_{1·2,...,d} = (\frac{S_{12}S_{22}^{−1}S′_{12}}{S_{11}})^{1/2} .

(Fisher, 1928): If R_{1·2,...,d} = 0 then (n − d)R̂_{1·2,...,d}^{2}/((d − 1)(1 − R̂_{1·2,...,d}^{2})) has an F (d − 1, n − d) distribution, n > d; if R_{1·2,...,d} ≠ 0 then the density function of R̂_{1·2,...,d}^{2} is

g(u) = \frac{(1 − u)^{(n−d−2)/2}(1 − R^{2})^{(n−1)/2}}{Γ((n − d)/2)Γ((n − 1)/2)} ∑_{j=1}^{∞} \frac{R^{2j}u^{(d−1)/2+j−1}Γ^{2}((n − 1)/2 + j)}{j!Γ((d − 1)/2 + j)}

for u ≥ 0, n > d, where R = R_{1·2,...,d}.

(v) Let X_{1} and X_{2} be as in 2.3 and consider a corresponding partition of the sample covariance matrix S. The sample partial correlation coefficient (see Definition 6 in 2.7) is

r_{ij·k+1,...,d} = \frac{s_{ij·k+1,...,d}}{(s_{ii·k+1,...,d} · s_{jj·k+1,...,d})^{1/2}},

where s_{ij·k+1,...,d} is the (i, j)th element of the matrix S_{11} − S_{12}S_{22}^{−1}S′_{12}. (Fisher, 1924): The distribution of r_{ij·k+1,...,d} is the same as that of the sample correlation coefficient between two variables based on a sample of size n − (d − k) from a standard bivariate normal distribution with

ρ = \frac{σ_{ij·k+1,...,d}}{(σ_{ii·k+1,...,d} · σ_{jj·k+1,...,d})^{1/2}},

where σ_{ij·k+1,...,d} is the (i, j)th element of the matrix Σ_{11} − Σ_{12}Σ_{22}^{−1}Σ′_{12}.

2.12 Exercises

1. Suppose X_{j} has a normal N(0, j^{2}) distribution, j = 1, 2, 3, and these three random variables are independent. Find P (X_{1} + X_{2} + X_{3} ≤ 4).

2. Suppose X_{1}, X_{2}, X_{3} are independent normal N(0, 1) variables. Find P (X_{1} < X_{2} < X_{3}) and E(X_{1}X_{2}X_{3}).

3. A point is picked randomly in ℝ^{3}. Its three coordinates X, Y, Z are independent normal N(0, 1) variables. Let R = \sqrt{X^{2} + Y^{2} + Z^{2}} be the distance of the point from the origin. Find: (i) the density of R^{2}; (ii) the density of R; (iii) E(R); (iiii) V ar (R).

4. Studying sentiment of people toward racially integrated neighborhoods, let X_{1} denote time an individual has lived in a given neighborhood; X_{2} denotes his feelings toward an integrated neighborhood (measured on a 100 point scale from −50 to 50); X_{3} denotes number of years of education of the individual. Many observations have been taken allowing to assume that X = (X_{1}, X_{2}, X_{3})′ ∼ N_{3}(μ, Σ) with

μ = ( \begin{matrix}3 \\ 10 \\ 8 \end{matrix} ) , Σ = ( \begin{matrix}1 & 3 & 1 \\ 3 & 16 & 2 \\ 1 & 2 & 4 \end{matrix} ) .

Find the conditional distribution of: (i) X_{1} given (X_{2}, X_{3})′ = (x_{2}, x_{3})′; (ii) (X_{1}, X_{2})′ given X_{3} = x_{3}; (iii) X_{3} given (X_{1}, X_{2})′ = (x_{1}, x_{2})′.

5. Verify the relation (1) in 2.6. Write down the formula for E(X_{1}|X_{2} = x_{2}) in this case.

6. Taking Definition 5 in 2.7 show that the multiple correlation coefficient R_{i·k+1,...,d} is non-negative and is bounded above by one.

7. Let X ∼ N_{4}(μ, Σ) with

Σ = ( \begin{matrix}1 & ρ & ρ & ρ \\ ρ & 1 & ρ & ρ \\ ρ & ρ & 1 & ρ \\ ρ & ρ & ρ & 1 \end{matrix} ) .

Calculate the multiple correlation coefficient R_{1·34} and the partial correlation coefficient ρ_{12·34}.

8. Let X ∼ N_{4}(μ, Σ) with

Σ = ( \begin{matrix}1 & ρ & ρ^{2} & ρ^{3} \\ ρ & 1 & ρ & ρ^{2} \\ ρ^{2} & ρ & 1 & ρ \\ ρ^{3} & ρ^{2} & ρ & 1 \end{matrix} ) .

Calculate the partial correlation coefficients ρ_{24·3}, ρ_{14·2}.

9. Let X ∼ N_{4}(00 0, Σ) with

Σ = ( \begin{matrix}1 & ρ_{2} & ρ_{1} & ρ_{1} \\ ρ_{2} & 1 & ρ_{1} & ρ_{1} \\ ρ_{1} & ρ_{1} & 1 & ρ_{2} \\ ρ_{1} & ρ_{1} & ρ_{2} & 1 \end{matrix} ) .

Consider the partition X_{1} = (X_{1}, X_{2})′, X_{2} = (X_{3}, X_{4})′. Find:

(i) the conditional distribution of X_{1} given X_{2} = x_{2}; (ii) R_{1·34}, R_{2·34}; (iii) ρ_{12·34}.

10. Prove the second property in 2.9.

11. Show that when d = 2, the distribution of the sample multiple correlation coefficient R̂_{1·2,...,d} given in 2.11 reduces to that given in Proposition 3 (see 1.15).