Lecture 4 : Optimal Confidence Region for Regression Parameters. Confidence Interval for Regression Function at Given Point.

(and in Given Interval. Optimal Designs.)

Optimal Confidence Region for Regression Parameters

Let t_{1}, t_{2}, …, t_{n}, be given numbers (at least two of them should be distinct) and assume that we observe

x_{i} = θ_{1} + θ_{2}t_{i} + ε_{i},

where ε_{i}, i = 1, 2, …, n, are independent identically distributed random variables having N(0, σ) distribution, while θ_{1} and θ_{2} are unknown parameters. So, in this case we have x = (x_{1}, x_{2}, …, x_{n}) and x_{i} ∼ N(θ_{1} + θ_{2}t_{i}, σ) are independent but not identically distributed.

The first problem which usually arises in this context is that of estimation of unknown parameters. We apply the least square method to estimate θ = (θ_{1}, θ_{2}). According to it, as the estimators we should take θ̂ = (θ̂_{1}, θ̂_{2}) such that

(θ̂_{1}, θ̂_{2}) = arg min_{(u,v)} Q(u, v),

where Q(u, v) = n ; ∑ ; i=1(x_{i} − u − vt_{i})^{2}.

For searching of θ̂_{1}, θ̂_{2} we need to solve with respect to (u, v) the system of equations

\{ \begin{matrix}\frac{∂Q(u,v)}{∂u} = 0 \\ \frac{∂Q(u,v)}{∂v} = 0 \end{matrix} ⟺ \{ \begin{matrix}n ; ∑ ; i=1(x_{i} − u − vt_{i}) = 0 \\ n ; ∑ ; i=1(x_{i} − u − vt_{i})t_{i} = 0 \end{matrix} ⟺ \{ \begin{matrix}u + t̄v = x̄ \\ t̄u + μ^{2}v = \frac{1}{n} n ; ∑ ; i=1 x_{i}t_{i} \end{matrix} ,

where

t̄= \frac{1}{n} n ; ∑ ; i=1 t_{i}, x̄ = \frac{1}{n} n ; ∑ ; i=1 x_{i}, μ^{2} = \frac{1}{n} n ; ∑ ; i=1 t_{i}^{2}.

The solutions θ̂_{1}, θ̂_{2} are given as

θ̂_{1} = \frac{μ^{2}x̄ − \frac{t̄}{n} n ; ∑ ; i=1 t_{i}x_{i}}{s_{t}^{2}},  θ̂_{2} = \frac{\frac{1}{n} n ; ∑ ; i=1 t_{i}x_{i} − t̄x̄}{s_{t}^{2}},

where s_{t}^{2} = \frac{1}{n} n ; ∑ ; i=1(t_{i} − t̄)^{2} = μ^{2} − t̄^{2}.

The minimal value R_{0}^{2} of Q(u, v) is written as

R_{0}^{2} = Q(θ̂_{1}, θ̂_{2}) = ns_{x}^{2} (1 − \frac{(\frac{1}{n} n ; ∑ ; i=1 t_{i}x_{i} − t̄x̄)^{2}}{s_{t}^{2}s_{x}^{2}} ) ,

where s_{x}^{2} = \frac{1}{n} n ; ∑ ; i=1(x_{i} − x̄)^{2}. Clearly, θ̂_{1}, θ̂_{2} can also be represented in the form

θ̂_{1} = \frac{μ^{2}ε̄− \frac{t̄}{n} ∑_{i=1}^{n} t_{i}ε_{i}}{s_{t}^{2}} + θ_{1}, θ̂_{2} = \frac{\frac{1}{n} ∑_{i=1}^{n} t_{i}ε_{i} − t̄ε̄}{s_{t}^{2}} + θ_{2},

where ε̄ = \frac{1}{n} n ; ∑ ; i=1 ε_{i}. Hence,

E(θ̂_{1}) = θ_{1}, E(θ̂_{2}) = θ_{2}

(the estimators are unbiased). Their variances have the form

V ar(θ̂_{1}) = \frac{1}{s_{t}^{4}}V ar (μ^{2}ε̄− \frac{t̄}{n} n ; ∑ ; i=1 t_{i}ε_{i}) = \frac{1}{s_{t}^{4}}V ar ( n ; ∑ ; i=1 (\frac{μ^{2}}{n} − \frac{t̄}{n}t_{i}) ε_{i})

= \frac{σ^{2}}{n^{2}s_{t}^{4}} n ; ∑ ; i=1 (μ^{2} − t̄t_{i})^{2} = \frac{σ^{2}}{n^{2}s_{t}^{4}} (nμ^{4} − nμ^{2}t̄^{2}) = \frac{σ^{2}μ^{2}}{ns_{t}^{2}},

V ar(θ̂_{2}) = \frac{1}{s_{t}^{4}}V ar (\frac{1}{n} n ; ∑ ; i=1 t_{i}ε_{i} − t̄ε̄) = \frac{1}{s_{t}^{4}}V ar ( n ; ∑ ; i=1 (\frac{t_{i}}{n} − \frac{t̄}{n}) ε_{i})

= \frac{σ^{2}}{n^{2}s_{t}^{4}} n ; ∑ ; i=1 (t_{i} − t̄)^{2} = \frac{σ^{2}}{ns_{t}^{2}}.

Since ε_{i}, i = 1, 2, …, n, have Normal distribution, the distribution of the estimators θ̂_{1}, θ̂_{2} is also Normal, namely

θ̂_{1} ∼ N (θ_{1}, \frac{σμ}{\sqrt{n}s_{t}} ) , θ̂_{2} ∼ N (θ_{2}, \frac{σ}{\sqrt{n}s_{t}} ) ,

or equivalently,

θ̂_{1} − θ_{1} ∼ \frac{σμ}{\sqrt{n}s_{t}}N(0, 1), θ̂_{2} − θ_{2} ∼ \frac{σ}{\sqrt{n}s_{t}}N(0, 1).

Let us denote

ξ = \frac{\sqrt{n}s_{t}}{σμ}(θ̂_{1} − θ_{1}), η = \frac{\sqrt{n}s_{t}}{σ}(θ̂_{2} − θ_{2}).

Then both ξ and η have N(0, 1) distribution.

It is also known that

ζ^{2} = \frac{R_{0}^{2}}{σ^{2}} ∼ χ_{n−2}^{2}

and ζ is independent of (ξ, η).

The vector (ξ, η) has two-dimensional Normal distribution with mean

vector equals to zero and covariance matrix ( \begin{matrix}1 & ρ \\ ρ & 1 \end{matrix} ) .

The correlation coefficient ρ for ξ and η is the same as that for θ̂_{1} and θ̂_{2} since

ρ = \frac{cov(ξ, η)}{\sqrt{V ar(ξ)V ar(η)}} = E(ξη) = \frac{ns_{t}^{2}}{σ^{2}μ}E((θ̂_{1}−θ_{1})(θ̂_{2}−θ_{2})) = \frac{cov(θ̂_{1}, θ̂_{2})}{\sqrt{V ar(θ̂_{1})V ar(θ̂_{2})}}.

Let us calculate ρ. As far as (θ̂_{1} − θ_{1}) + t̄(θ̂_{2} − θ_{2}) = ε̄, we have

cov(θ̂_{1}, θ̂_{2}) = E[(θ̂_{1} − θ_{1})(θ̂_{2} − θ_{2})] = E[(ε̄− t̄(θ̂_{2} − θ_{2}))(θ̂_{2} − θ_{2})]

= E(ε̄(θ̂_{2} − θ_{2})) − t̄E(θ̂_{2} − θ_{2})^{2} = −t̄V ar(θ̂_{2}) = −\frac{σ^{2}t̄}{ns_{t}^{2}}

and

ρ = −\frac{σ^{2}t̄}{ns_{t}^{2}}\frac{ns_{t}^{2}}{σ^{2}μ} = −\frac{t̄}{μ}.

It is known that in this case the vector

\sqrt{n − 2} (\frac{ξ}{ζ} , \frac{η}{ζ} ) = \frac{\sqrt{(n − 2)n}s_{t}}{μ} (\frac{θ̂_{1} − θ_{1}}{R_{0}}, \frac{μ(θ̂_{2} − θ_{2})}{R_{0}}) ,

which does not depend on σ, has two-dimensional Student’s distribution, which density has the form

g(u, v) = \frac{1}{2π\sqrt{1 − ρ^{2}}} (1 + \frac{u^{2} − 2ρuv + v^{2}}{(n − 2)(1 − ρ^{2})})^{−n/2} , u, v ∈ R.

Let α ∈ (0, 1) be a given confidence level and B_{0} ⊂ R^{2} be a Borel set such that

P (\frac{\sqrt{(n − 2)n}s_{t}}{μ} (\frac{θ̂_{1} − θ_{1}}{R_{0}}, \frac{μ(θ̂_{2} − θ_{2})}{R_{0}}) ∈ B_{0}) = α.  (1)

Then the random set

B(x) = (θ̂_{1}, θ̂_{2}) − \frac{μR_{0}}{\sqrt{(n − 2)n}s_{t}}B′_{0},  (2)

where

B′_{0} = {(u, v) : (u, μv) ∈ B_{0}},

is the confidence region (strong) of level α for (θ_{1}, θ_{2}). Furthermore,

R(θ, B) = E_{θ}λ(B(x)) = \frac{μλ(B_{0})}{(n − 2)ns_{t}^{2}}E_{θ}R_{0}^{2} = \frac{σ^{2}μλ(B_{0})}{ns_{t}^{2}}.

So, the set B^{∗}(x) of the form (2) is the optimal confidence region of level α for (θ_{1}, θ_{2}) if the corresponding set B_{0}^{∗} has the minimal area among all the sets satisfying the condition (1).

Now we can apply Basic Lemma to the function g(u, v), which satisfies all it’s conditions. As the set B_{0}^{∗} we can take

B_{0}^{∗} = {(u, v) : g(u, v) ≥ z_{α}},

where z_{α} is defined by the equation

\begin{matrix}∫ & ∫ \\ B_{0}^{∗} \end{matrix} g(u, v)dudv = α.

Hence,

B_{0}^{∗} = {(u, v) : u^{2} − 2ρuv + v^{2} ≤ (n − 2)(1 − ρ^{2})r_{α}^{2}}

= \{(u, v) : u^{2} + 2\frac{t̄}{μ}uv + v^{2} ≤ \frac{(n − 2)s_{t}^{2}r_{α}^{2}}{μ^{2}} \} ,

where

r_{α}^{2} = (2πz_{α}\sqrt{1 − ρ^{2}})^{−2/n} − 1.

It is worth noting that r_{α}^{2} depends only on n and α. Indeed,

α = \begin{matrix}∫ & ∫ \\ B_{0}^{∗} \end{matrix} g(u, v)dudv = \frac{1}{2π\sqrt{1 − ρ^{2}}} \begin{matrix}∫ & ∫ \\ B_{0}^{∗} \end{matrix} (1 + \frac{u^{2} − 2ρuv + v^{2}}{(n − 2)(1 − ρ^{2})})^{−n/2} dudv,

or equivalently,

\begin{matrix}∫ & ∫ \\ B_{0}^{∗} \end{matrix} (1 + \frac{u^{2} − 2ρuv + v^{2}}{(n − 2)(1 − ρ^{2})})^{−n/2} dudv = 2πα\sqrt{1 − ρ^{2}}.

After changing of variables (u, v) → (r, φ) according to formulas

u = \frac{r\sqrt{n−2}}{\sqrt{2}} [\sqrt{1 + ρ} cos φ + \sqrt{1 − ρ} sin φ]
v = \frac{r\sqrt{n−2}}{\sqrt{2}} [\sqrt{1 + ρ} cos φ − \sqrt{1 − ρ} sin φ]

with the Jacobian of transformation |J| = r(n − 2)\sqrt{1 − ρ^{2}} we get

\begin{aligned}∫ & ∫ \\ & B′ \end{aligned} r(1 + r^{2})^{−n/2}drdφ = \frac{2πα}{n − 2},

where

B′ = {(r, φ) : 0 ≤ r ≤ r_{α}, 0 ≤ φ ≤ 2π}.

So,

r_{α} ; ∫ ; 0r(1+r^{2})^{−n/2}dr = \frac{α}{n − 2} ⟺ r_{α}^{2} ; ∫ ; 0(1+z)^{−n/2}dz = \frac{2α}{n − 2} ⟺ 1−(1 + r_{α}^{2})^{−\frac{n−2}{2}} = α.

Therefore,

r_{α}^{2} = (1 − α)^{−2/(n−2)} − 1.

Finally, the optimal confidence region has the form

B^{∗}(x) = \{(u, v) : u = θ̂_{1} − \frac{μR_{0}}{\sqrt{(n − 2)n}s_{t}}z, v = θ̂_{2} − \frac{μR_{0}}{\sqrt{(n − 2)n}s_{t}}w, (z, μw) ∈ B_{0}^{∗}\}

= \{(u, v) : ((θ̂_{1} − u)\frac{\sqrt{(n − 2)n}s_{t}}{μR_{0}}, (θ̂_{2} − v)\frac{\sqrt{(n − 2)n}s_{t}}{R_{0}} ) ∈ B_{0}^{∗}\} ,

or equivalently,

B^{∗}(x) = \{(u, v) : (u − θ̂_{1})^{2} + 2t̄(u − θ̂_{1})(v − θ̂_{2}) + μ^{2}(v − θ̂_{2})^{2} ≤ \frac{r_{α}^{2}R_{0}^{2}}{n}\} .

(3)

It is an ellipse which centre is at the point (θ̂_{1}, θ̂_{2}). Furthermore,

λ(B^{∗}(x)) = \frac{πr_{α}^{2}R_{0}^{2}}{ns_{t}}

and

R(θ, B^{∗}) = \frac{π(n − 2)σ^{2}r_{α}^{2}}{ns_{t}}.

It is interesting to note that the confidence region which we have just obtained is the same as classical one given in statistical books though the classical confidence region is obtained through Fisher-Snedecor distribution.

The values of r_{α} for different α and n are given in the following table.

\begin{matrix}α & n = 10 & n = 20 & n = 30 \\ 0.1 & .1633710378 & .1085150496 & .0869145615 \\ 0.2 & .2395229905 & .1584413077 & .1267538007 \\ 0.3 & .3053933758 & .2010629677 & .1606365120 \\ 0.4 & .3690790782 & .2416610705 & .1927728871 \\ 0.5 & .4349794420 & .2829482974 & .2252923412 \\ 0.6 & .5073789806 & .3273731495 & .2600743105 \\ 0.7 & .5926214264 & .3783322693 & .2996735073 \\ 0.8 & .7038101882 & .4425078248 & .3490392471 \\ 0.9 & .8822014566 & .5399533915 & .4228104008 \\ 0.95 & 1.055813680 & .6284511071 & .4884659333 \\ 0.975 & 1.230799276 & .7117795937 & .5490592764 \\ 0.99 & 1.470468517 & .8173741720 & .6240957411 \end{matrix}

Confidence Interval for Regression Function at Given Point

Here we construct confidence interval [f_{−}(t), f_{+}(t)] for regression function f(t) = θ_{1} + θ_{2}t at given point t basing on the optimal (strong) confidence region B^{∗}(x) of level α for (θ_{1}, θ_{2}) of the form (3).

It is naturally to take

f_{−}(t) = min_{(u,v)∈B^{∗}} (u + vt), f_{+}(t) = max_{(u,v)∈B^{∗}} (u + vt).

Hence, the problem is reduced to that of searching for the extremes of linear function u + vt on B^{∗}.

We utilize Largange method for solving the problem. Let

L(u, v, λ) = u + vt + λ(r^{2} − (u − θ̂_{1})^{2} − 2t̄(u − θ̂_{1})(v − θ̂_{2}) − μ^{2}(v − θ̂_{2})^{2})

be Lagrange function, where r^{2} = n^{−1}r_{α}^{2}R_{0}^{2}. The extremes could be found through solving the system of equations with respect to (u, v)

\{ \begin{matrix}\frac{∂L(u,v,λ)}{∂u} = 0 \\ \frac{∂L(u,v,λ)}{∂v} = 0 \\ (u − θ̂_{1})^{2} + 2t̄(u − θ̂_{1})(v − θ̂_{2}) + μ^{2}(v − θ̂_{2})^{2} = r^{2} \end{matrix} ⟺

\{ \begin{matrix}2λ(u − θ̂_{1}) + 2λt̄(v − θ̂_{2}) = 1 \\ 2λt̄(u − θ̂_{1}) + 2λμ^{2}(v − θ̂_{2}) = t \\ (u − θ̂_{1})^{2} + 2t̄(u − θ̂_{1})(v − θ̂_{2}) + μ^{2}(v − θ̂_{2})^{2} = r^{2} \end{matrix} ⟺

\{ \begin{matrix}v − θ̂_{2} = \frac{1}{2λ} \frac{t−t̄}{s_{t}^{2}} \\ u − θ̂_{1} = \frac{1}{2λ} (1 − \frac{t−t̄}{s_{t}^{2}}t̄) \\ \frac{1}{4λ^{2}} [(1 − \frac{t−t̄}{s_{t}^{2}}t̄)^{2} + 2t̄(1 − \frac{t−t̄}{s_{t}^{2}}t̄) \frac{t−t̄}{s_{t}^{2}} + μ^{2} (\frac{t−t̄}{s_{t}^{2}})^{2}] = r^{2} \end{matrix} ⟺

\{ \begin{matrix}u − θ̂_{1} = \frac{1}{2λ} (1 − \frac{t−t̄}{s_{t}^{2}}t̄) \\ v − θ̂_{2} = \frac{1}{2λ} \frac{t−t̄}{s_{t}^{2}} \\ \frac{1}{2λ} = ±_{\frac{r}{\sqrt{\frac{(t−t̄)^{2}}{s^{2}_{t}} +1}}} \end{matrix} ⟹ \{ \begin{matrix}u = θ̂_{1} ± (1 − \frac{t−t̄}{s_{t}^{2}}t̄) _{\frac{r}{\sqrt{\frac{(t−t̄)^{2}}{s^{2}_{t}} +1}}} \\ v = θ̂_{2} ± \frac{t−t̄}{s_{t}^{2}}_{\frac{r}{\sqrt{\frac{(t−t̄)^{2}}{s^{2}_{t}} +1}}} \end{matrix}

Finally, we get the confidence interval [f_{−}(t), f_{+}(t)] , where

f_{−}(t) = f̂(t) − \frac{r_{α}R_{0}}{\sqrt{n}}\sqrt{\frac{(t − t̄)^{2}}{s_{t}^{2}} + 1},

f_{+}(t) = f̂(t) + \frac{r_{α}R_{0}}{\sqrt{n}}\sqrt{\frac{(t − t̄)^{2}}{s_{t}^{2}} + 1},

and f̂(t) = θ̂_{1} + θ̂_{2}t.

Of course, this confidence interval is not strong since

P_{θ}^{(n)} (f_{−}(t) ≤ f(t) ≤ f_{+}(t)) > α.

Exercise after Lecture 4

Assume that at points t_{i} ∈ [−1, 1] you have observations x_{i}, i = 1, …, 10, namely :

\begin{matrix}t_{i} & −1.0 & −0.8 & −0.6 & −0.4 & −0.2 & 0.2 & 0.4 & 0.6 & 0.8 & 1.0 \\ x_{i} & 1.2 & 2.0 & 0.0 & 0.3 & 2.0 & 1.3 & −0.9 & 1.2 & 2.6 & 3.3 \end{matrix}

You have decided to fit linear regression model to the data. Construct, please :

• least squares estimators θ̂_{1}, θ̂_{2};

• optimal confidence region B^{∗} of level α = 0.95 for (θ_{1}, θ_{2});

• confidence interval [f_{−}(t), f_{+}(t)] for regression function f(t) = θ_{1} + θ_{2}t at point t = 1.5.

Confidence Region for Regression Function in Given Interval

Here we construct the confidence region B for the regression function f(t) = α_{0} + β_{0}t in given interval [t_{−}, t_{+}]. Without loss of generality we suppose that t_{−}t_{+} < 0. We use two methods.

The first one is based on the confidence interval for the regression function in given point. We get

B(x) = {(t, v) : t_{−} ≤ t ≤ t_{+}, f_{−}(t) ≤ v ≤ f_{+}(x)} =
{(t, v) : t_{−} ≤ t ≤ t_{+}, f̂(t) − c_{0} ≤ v ≤ f̂(t) − c_{0}},

where f̂(t) = a + bt,

c_{0} = \frac{r_{α}\sqrt{Q}}{\sqrt{n}}\sqrt{\frac{(t − t̄)^{2}}{s_{t}^{2}} + 1}.

The second method is the following. Let us find such c ∈ R that

P ( sup_{t_{−}≤t≤t_{+}} |f(t) − f̂(t)| ≤ c) = α.

Denote

B′ = {(u, v) : −c ≤ u + vt ≤ c, t ∈ [t_{−}, t_{+}]}.

Then

P ((a − α_{0}, b − β_{0}) ∈ B′) = α.

Finally, the confidence region is written as

B(x) = {(t, v) : t_{−} ≤ t ≤ t_{+}, f̂(t) − c ≤ v ≤ f̂(t) − c}.

Optimal Designs

Now assume that we can vary the design points {t_{1}, …, t_{n}} given n and α. The question is : how to choose the design in an optimal way. This problem is called an optimal design problem.

For construction of the optimal design we should first choose the criterion for it. It is quite naturally to take the value of R((α_{0}, β_{0}), B) = E_{(α_{0},β_{0})}λ(B(x)) as an optimality criterion. In other words, we call the design {t_{1}^{∗}, …, t_{n}^{∗}} optimal if

{t_{1}^{∗}, …, t_{n}^{∗}} = arg min_{{t_{1},...,t_{n}}} R((α_{0}, β_{0}), B).

Since in our case

R((α_{0}, β_{0}), B) = \frac{πσ^{2}r_{α}^{2}(n − 2)}{ns_{t}},

therefore the optimal design is that with the maximal value of s_{t}. It means that, e. g. in the case t_{i} ∈ [−1, 1], i = 1, …, n, and n is even, we should make exactly n/2 observations in each points −1 and 1.