Lecture 3 : Linear Regression Model. Best Linear Unbiased Estimator. Least Squares Estimator. Optimal Confidence Set. Optimal Confidence Interval for Regression Function at a Given Point.

Linear Regression Model

Let we have a vector of observations y = (y_{1}, y_{2}, …, y_{n})′ with the expectation E(y) = Xθ and the covariance matrix D(y) = σ^{2}I_{n}, where

– X = (x_{1}, x_{2}, …, x_{n})′ is the n × k-matrix of full rank k ≤ n, x_{i} ∈ R^{k}, i = 1, …, n;

– θ = (θ_{1}, θ_{2}, …, θ_{n})′ is the k × 1-vector of unknown parameters ;

– σ^{2} = V ar(y_{i}) > 0 is unknown ;

– I_{n} is the n × n identity matrix . The function f : R^{k} ↦ R, f(x) = x′θ is called the regression function. Clearly, E(y_{i}) = x′_{i}θ = f(x_{i}).

A linear unbiased estimator (LUE) of θ is of the form θ̂(y) = Ay and satisfies the condition

E(Ay) = AXθ = θ ∀θ ∈ R^{k} ⟺ AX = I_{k}.

Definition 1. A LUE θ̂(y) of θ is said to be the best linear unbiased estimator (BLUE) of θ if D(θ̂(y)) ≤ D(θ̃(y)) for any other LUE θ̃(y).

It is easy to show that the inequality D(θ̂(y)) ≤ D(θ̃(y)) is equivalent to that of V ar(a′θ̂(y)) ≤ V ar(a′θ̃(y)) for all a ∈ R^{k} (Exercise 1).

That is why if θ̂(y) is the BLUE of θ then f̂(x) = x′θ̂ is the LUE of f(x) at a given point x, having the minimal variance among all LUE.

Least Squares Estimator

Define

Q(θ) = (y − Xθ)′(y − Xθ) = n ; ∑ ; i=1 (y_{i} − k ; ∑ ; j=1 X_{ij}θ_{j})^{2} .

Definition 2. Any point θ̂(y) from the set

arg min_{θ} Q(θ)

is said to be the least squares estimator (LSE) of θ.

The LSE of θ in the linear regression model is written as

θ̂(y) = (X′X)^{−1}X′y.  (1)

Indeed, due to the necessary condition for an extremum,

grad Q(θ) = 2 (X′Xθ − X′y) = 0 ⟺ X′Xθ = X′y.

The last equation is known as the normal equation. It possesses the unique solution (1) for the model considered. Since

Hess Q(θ) = 2X′X > 0 ∀θ,

this solution is the point of minimum for Q(θ).

It is easy to see that the LSE θ̂(y) of θ is a LUE. Moreover, its covariance matrix is of the form

D(θ̂(y)) = σ^{2}(X′X)^{−1}.

The well-known Gauss-Markov Theorem states that within the linear regression model the LSE θ̂(y) coincides with the BLUE.

Least Squares Estimator under Constraints

Assume we know in advance that Hθ = 0, where H is a given m × kmatrix of full rank m ≤ k. What is the LSE of θ in this context ?

Consider the Lagrangian function

L(θ, λ) = (y − Xθ)′(y − Xθ) + λ′Hθ.

Due to the necessary condition for an extremum,

\{ \begin{matrix}2X′Xθ − 2X′y + H′λ & = & 0 \\ Hθ & = & 0 \end{matrix} ⟺ ( \begin{matrix}X′X & H′ \\ H & 0 \end{matrix} ) ( \begin{matrix}θ \\ \frac{1}{2}λ \end{matrix} ) = ( \begin{matrix}X′y \\ 0 \end{matrix} ) .

(2)

Since

( \begin{matrix}X′X & H′ \\ H & 0 \end{matrix} )^{−1} =

(\begin{matrix}(X′X)^{−1}−(X′X)^{−1}H′[H(X′X)^{−1}H′]^{−1}H(X′X)^{−1} & (X′X)^{−1}H′[H(X′X)^{−1}H′]^{−1} \\ (H(X′X)^{−1}H′)^{−1}H(X′X)^{−1} & −[H(X′X)^{−1}H′]^{−1} \end{matrix}) ,

equation (2) possesses with respect to θ a unique solution of the form

θ̂(y) = (X′X)^{−1}X′y − (X′X)^{−1}H′[H(X′X)^{−1}H′]^{−1}H(X′X)^{−1}X′y. It is a point of minimum since Hess L_{θθ}(θ, λ) = 2X′X > 0.

Again, the LSE θ̂(y) of θ is a LUE since E(θ̂(y)) = θ − (X′X)^{−1}H′[H(X′X)^{−1}H′]^{−1}Hθ = θ ∀θ ∈ R^{k}.

Moreover, its covariance matrix is of the form D(θ̂(y)) = σ^{2} [(X′X)^{−1} − (X′X)^{−1}H′[H(X′X)^{−1}H′]^{−1}H(X′X)^{−1}] .

Confidence Sets and Confidence Intervals

Assume that y = (y_{1}, y_{2}, …, y_{n})′ follows a multivariate normal distribution N_{n}(Xθ, σ^{2}I_{n}). Let α ∈ (0, 1) be a given number which we call the confidence level.

Definition 3. A random set B(y) ∈ B^{k} which does not depend on θ and σ^{2} is said to be a confidence set of level α for θ if

P (θ ∈ B(y)) ≥ α  ∀θ ∈ R^{k}.  (3)

It is said to be a strong confidence set if

P (θ ∈ B(y)) = α ∀θ ∈ R^{k}.

In the case of estimation f(x) = x′θ at a given point x, we speak about a confidence interval [t_{1}(y), t_{2}(y)] of level α for which

P (t_{1}(y) ≤ f(x) ≤ t_{2}(y)) ≥ α (= α),

where t_{1}(y), t_{2}(y) are statistics.

The quality of a confidence set is reasonable to characterize by the risk function

R(B, θ) = E (λ(B(y))) ,

where λ(·) is the Lebesque measure in R^{k}.

Definition 4. A confidence set B^{∗}(y) of level α for θ is said to be optimal if

B^{∗}(y) ∈ arg min_{B} R(B, θ) ∀θ ∈ R^{k},

where minimum is taken over all sets B satisfying (3).

Similarly, we define the notion of an optimal confidence interval.

Confidence Set Basing on the Fisher Distribution

Recall that within the linear regression model the BLUE of θ is the same as the LSE and is of the form

θ̂(y) = (X′X)^{−1}X′y.

Denote

R_{1}^{2}(y) = ∥y − Xθ̂(y)∥^{2} = (y − Xθ̂(y))′(y − Xθ̂(y)),
R_{2}^{2}(y) = ∥Xθ̂(y) − Xθ∥^{2} = (θ − θ̂(y))′X′X(θ − θ̂(y))

and assume that n > k. It is not difficult to see that :

• θ̂(y) − θ ∼ N_{k}(0, σ^{2}(X′X)^{−1}),

• R_{1}^{2}(y) ∼ σ^{2}χ_{n−k}^{2},

• R_{2}^{2}(y) ∼ σ^{2}χ_{k}^{2}, and θ̂(y) − θ, R_{1}^{2}(y), R_{2}^{2}(y) are independent.

As it is known,

\frac{\frac{1}{k}R_{2}^{2}}{\frac{1}{n−k}R_{1}^{2}} ∼ F (k, n − k).

Let α be a given confidence level and F_{k,n−k,α} be the quantile of level α of F (k, n − k) distribution. Then

P (\frac{(θ − θ̂(y))′X′X(θ − θ̂(y))}{R_{1}^{2}(y)} ≤ \frac{k}{n − k}F_{k,n−k,α}) = α ⟺

P ((θ − θ̂(y))′X′X(θ − θ̂(y)) ≤ \frac{k}{n − k}F_{k,n−k,α}R_{1}^{2}(y)) = α.

Hence, the set

B(y) = \{u ∈ R^{k} : (u − θ̂(y))′X′X(u − θ̂(y)) ≤ \frac{k}{n − k}F_{k,n−k,α}R_{1}^{2}(y)\}

is a strong confidence set of level α for θ. It is an ellipse centered at the point θ̂(y). Is it optimal in any sense ?

Optimal Confidence Set

Denote

η = \frac{1}{σ} (\frac{θ̂_{1} − θ_{1}}{\sqrt{((X′X)^{−1})_{11}}}, …, \frac{θ̂_{k} − θ_{k}}{\sqrt{((X′X)^{−1})_{kk}}} )

and

M_{ij} = \frac{((X′X)^{−1})_{ij}}{\sqrt{((X′X)^{−1})_{ii}}\sqrt{((X′X)^{−1})_{jj}}} .

Then η ∼ N_{k}(0, M) and the vector

\frac{η}{\sqrt{R_{1}^{2}/(σ^{2}(n − k))}} = \frac{\sqrt{n − k}}{R_{1}} (\frac{θ̂_{1} − θ_{1}}{\sqrt{((X′X)^{−1})_{11}}}, …, \frac{θ̂_{k} − θ_{k}}{\sqrt{((X′X)^{−1})_{kk}}} )

has a multidimensional Student’s distribution t(k, M, n − k), which density is of the form

p(u) = \frac{Γ (\frac{n}{2})}{(π(n − k))^{k/2}Γ (\frac{n−k}{2} ) \sqrt{det M}} (1 + \frac{1}{n − k}u′M^{−1}u)^{−n/2}, u ∈ R^{k}.

Let α be a given confidence level and B_{0} ⊂ R^{k} be a Borel set such that

P (\frac{\sqrt{n − k}}{R_{1}(y)} (\frac{θ̂_{1}(y) − θ_{1}}{\sqrt{((X′X)^{−1})_{11}}}, …, \frac{θ̂_{k}(y) − θ_{k}}{\sqrt{((X′X)^{−1})_{kk}}} ) ∈ B_{0}) = α. (4)

Then a strong confidence set of level α for θ is given as

B(y) = θ̂(y) − \frac{R_{1}(y)}{\sqrt{n − k}}B′_{0},  (5)

where

B′_{0} = \{u ∈ R^{k} : (\frac{u_{1}}{\sqrt{((X′X)^{−1})_{11}}}, …, \frac{u_{k}}{\sqrt{((X′X)^{−1})_{kk}}} ) ∈ B_{0}\} .

Optimal Confidence Set (continuation)

Moreover,

E(λ(B(y))) = \frac{λ(B′_{0})}{(n − k)^{k/2}}E(R_{1}^{k}(y)) = \frac{(∏_{i=1}^{k}((X′X)^{−1})_{ii})^{1/2}}{(n − k)^{k/2}}λ(B_{0})E(R_{1}^{k}(y)).

That is why a set B^{∗}(y) of the form (5) is the optimal confidence set of level α for θ if the relevant set B_{0}^{∗} has the minimal area (volume) among all sets satisfying (4). Apply Lemma on Minimal Area to the function p(u) satisfying all the conditions. As B_{0}^{∗} we should take

B_{0}^{∗} = {u ∈ R^{k} : p(u) ≥ z_{α}} ,

where z_{α} is defined by the equation _{B_{0}^{∗}}^{∫} p(u)du = α. Taking into account

the formula for p(u), we can rewrite

B_{0}^{∗} = {u ∈ R^{k} : u′M^{−1}u ≤ (n − k)r_{α}^{2}} ,

where

r_{α}^{2} = (\frac{c_{n}}{z_{α}\sqrt{det M}} )^{2/n} − 1, c_{n} = \frac{Γ (\frac{n}{2})}{(π(n − k))^{k/2}Γ (\frac{n−k}{2} )}.

Finally, the optimal confidence set is of the form

B^{∗}(y)=\{u ∈ R^{k} : \frac{\sqrt{n − k}}{R_{1}(y)} (\frac{θ̂_{1}(y) − u_{1}}{\sqrt{((X′X)^{−1})_{11}}}, …, \frac{θ̂_{k}(y) − u_{k}}{\sqrt{((X′X)^{−1})_{kk}}} )∈B_{0}^{∗}\}

= \{u ∈ R^{k} : k ; ∑ ; i,j=1 \frac{u_{i} − θ̂_{i}(y)}{\sqrt{((X′X)^{−1})_{ii}}} \frac{u_{j} − θ̂_{j}(y)}{\sqrt{((X′X)^{−1})_{jj}}} (M^{−1})_{ij} ≤ r_{α}^{2}R_{1}^{2}(y)\}

= {u ∈ R^{k} : k ; ∑ ; i,j=1(u_{i} − θ̂_{i}(y))(u_{j} − θ̂_{j}(y))(X′X)_{ij} ≤ r_{α}^{2}R_{1}^{2}(y)} .

It is possible to show that r_{α}^{2} = \frac{k}{n−k}F_{k,n−k,α} (Exercise 6) and thus B^{∗}(y) is the same set as that basing on the Fisher distribution.

Confidence Interval for Regression Function at a Given Point

Basing on the optimal confidence set B^{∗}(y) of level α for θ, one can propose a confidence interval for f(x) = x′θ at a given point x as

[ min_{θ∈B^{∗}(y)} x′θ, max_{θ∈B^{∗}(y)} x′θ] .

Consider the Lagrangian function

L(θ_{1}, …, θ_{k}, λ) = k ; ∑ ; j=1 θ_{j}x_{j} + λ ( k ; ∑ ; i,j=1(X′X)_{ij}(θ_{i} − θ̂_{i})(θ_{j} − θ̂_{j}) − r_{α}^{2}R_{1}^{2}) .

Due to the necessary condition for an extremum,

\{ \begin{matrix}x + 2λX′X(θ − θ̂) & = & 0 \\ (θ − θ̂)′X′X(θ − θ̂) & = & r_{α}^{2}R_{1}^{2} \end{matrix} ⟺

\{ \begin{matrix}θ − θ̂ & = & −\frac{1}{2λ}(X′X)^{−1}x \\ \frac{1}{4λ^{2}}((X′X)^{−1}x)′X′X(X′X)^{−1}x & = & r_{α}^{2}R_{1}^{2} \end{matrix} ⟺

\{ \begin{matrix}θ & = & θ̂ − \frac{1}{2λ}(X′X)^{−1}x \\ \frac{1}{2λ} & = & ±\frac{r_{α}R_{1}}{\sqrt{x′(X′X)^{−1}x}} \end{matrix} .

Finally, we have two solutions with respect to θ :

θ^{∗} = θ̂ ∓ \frac{r_{α}R_{1}}{\sqrt{x′(X′X)^{−1}x}}(X′X)^{−1}x.

Due to the sufficient condition for an extremum, the solution with ”−” corresponds to the minimum while that with ”+” to the maximum.

Hence, a confidence interval (non-strong) of level α for f(x) = x′θ at a given point x is of the form

[x′θ̂(y) − r_{α}R_{1}(y)\sqrt{x′(X′X)^{−1}x}, x′θ̂(y) + r_{α}R_{1}(y)\sqrt{x′(X′X)^{−1}x}] .

Is it optimal in any sense ?

Optimal Confidence Interval

We have

x′θ̂ ∼ N(x′θ, σ^{2}x′(X′X)^{−1}x) ⟹ \frac{x′θ̂ − x′θ}{σ\sqrt{x′(X′X)^{−1}x}} ∼ N(0, 1).

Hence,

\frac{x′θ̂ − x′θ}{σ\sqrt{x′(X′X)^{−1}x}\sqrt{R_{1}^{2}/(σ^{2}(n − k))}} = \frac{\sqrt{n − k}(x′θ̂ − x′θ)}{R_{1}\sqrt{x′(X′X)^{−1}x}} ∼ t(n − k),

where t(n−k) is the Student‘s distribution with n−k degrees of freedom, which density is of the form

p(u) = \frac{Γ (\frac{n−k+1}{2})}{\sqrt{π(n − k)}Γ (\frac{n−k}{2} )} (1 + \frac{u^{2}}{n − k})^{−(n−k+1)/2} , u ∈ R.

Apply Lemma on Minimal Area to the function p(u) satisfying all the conditions. Let t_{n−k,\frac{1+α}{2}} be the quantile of level \frac{1+α}{2} of t(n − k) distribution. Then

P (−t_{n−k,\frac{1+α}{2}} ≤ \frac{\sqrt{n − k}(x′θ̂(y) − x′θ)}{R_{1}(y)\sqrt{x′(X′X)^{−1}x}} ≤ t_{n−k,\frac{1+α}{2}} ) = α.

Thus, the optimal confidence interval (strong) of level α for f(x) = x′θ at a given point x is of the form

[x′θ̂(y) − \frac{t_{n−k,\frac{1+α}{2}}}{\sqrt{n − k}}R_{1}(y)\sqrt{x′(X′X)^{−1}x}, x′θ̂(y) + \frac{t_{n−k,\frac{1+α}{2}}}{\sqrt{n − k}}R_{1}(y)\sqrt{x′(X′X)^{−1}x}] .

It is possible to show that for k ≥ 2,

t_{n−k,\frac{1+α}{2}}^{2} < kF_{k,n−k,α}

(Exercise 8). It means that the last confidence interval has the shorter length than the previous one.

Questions and Exercises after Lecture 3

1. Prove that for two LUE θ̂(y) and θ̃(y) of θ within the linear regression model,

D(θ̂(y)) ≤ D(θ̃(y)) ⟺ V ar(a′θ̂(y)) ≤ V ar(a′θ̃(y)) ∀a ∈ R^{k}.

2. Construct the LSE of θ = (θ_{1}, θ_{2})′ in the case of a linear fit model when

X′ = ( \begin{matrix}1 & 1 & . . . . . . & 1 \\ z_{1} & z_{2} & . . . . . . & z_{n} \end{matrix} ) , z_{i} ∈ R, n ≥ 2.

3. Construct the LSE of θ = (θ_{1}, θ_{2})′ under the constraint θ_{1} + θ_{2} = 0 in the linear fit model from Exercise 2.

4. Apply another method for construction of estimator in the linear regression model with the constraint from Exercise 3. Namely, consider the matrix

K = ( \begin{matrix}1 & 1 \\ 0 & 1 \end{matrix} ) and denote β = Kθ = ( \begin{matrix}θ_{1} + θ_{2} \\ θ_{2} \end{matrix} ) = ( \begin{matrix}0 \\ θ_{2} \end{matrix} ) .

Now substitute θ = K^{−1}β to the initial linear regression model and obtain

E(y) = Xθ = XK^{−1}β or E(y_{i}) = (XK^{−1})_{i2}β_{2}, i = 1, …, n.

In such a case we transform the model with the constraint to that of without constraint. Obtaining by usual least squares method an estimator β̂_{2}, consider then the estimator θ̂ of the initial vector θ as

θ̂ = K^{−1} ( 0 ; β̂_{2} ) . Does such an estimator differ from that in Exercise 3 ?

5. What could you say about the relation between two confidence sets (intervals) of levels α_{1} and α_{2} for unknown θ when, say, α_{1} > α_{2} ?

6. Prove that for a given 0 < α < 1 and n > k the equality r_{α}^{2} = \frac{k}{n−k}F_{k,n−k,α} holds, where F_{k,n−k,α} is the quantile of level α of F (k, n − k) distribution, while r_{α}^{2} is defined as on page 8.

7. Let the function g(θ), g : R^{k} ↦ R is to be estimated within the linear regression model. Basing on the optimal confidence set B^{∗}(y) of level α for θ, propose a method for construction of a confidence interval for g(θ).

8. Prove that for a given 0 < α < 1 and n > k ≥ 2 the inequality t_{n−k,\frac{1+α}{2}}^{2} < kF_{k,n−k,α} holds, where t_{n−k,\frac{1+α}{2}} is the quantile of level \frac{1+α}{2} of t(n−k) distribution, while F_{k,n−k,α} is the quantile of level α of F (k, n−k) distribution.

9. How does the optimal confidence interval of level α for f(x) = x′θ at x = (1, 0)′ look like in the case of a linear fit model from Exercise 2 ?