Lecture 1 : Basic Notions. Location-Scale Families. Two Methods for Estimation. Confidence Regions. Equivariant and Invariant Statistics.

Basic Notions

ξ – random variable on the probability space (Ω, F, P), (X , B_{X}) – range space of ξ, X ⊂ R, F_{ξ}(u) = P (ξ ≤ u) – distribution function of ξ, u ∈ R, f_{ξ}(u) = \frac{dF_{ξ}(u)}{du} – density function of ξ, u ∈ R, P_{ξ}(A) = _{A}^{∫} f_{ξ}(u)du – distribution of ξ, A ∈ B_{X} , x = (x_{1}, x_{2}, …, x_{n}) – sample of size n drawn from the distribution P_{ξ}, (X^{n}, B_{X^{n}}) – sample space, X^{n} = X × ⋯ × X ,

P_{ξ}^{(n)}(A) = ∫ … ∫_{A} f_{ξ}(u_{1}) … f_{ξ}(u_{n})du_{1} … du_{n}, A ∈ B_{X^{n}},

(X^{n}, B_{X^{n}}, P_{ξ}^{(n)}) – statistical model (space). ——————————————————————— If P_{ξ} ∈ {P_{θ} : θ ∈ Θ}, Θ ⊂ R^{k}, k ≥ 1, then we call the statistical model parametric and denote it by (X^{n}, B_{X^{n}}, P_{θ}^{(n)}). Saying that we have a parametric model (X^{n}, B_{X^{n}}, P_{θ}^{(n)}) means that we have a sample x of size n drawn from the distribution P_{θ}. If P_{ξ} ∈ {P_{θ} : θ ∈ Θ} then F_{ξ} ∈ {F_{θ} : θ ∈ Θ} and vice versa.

Location-Scale Families

Consider the distribution family {F_{θ} : θ ∈ Θ} such that for any u ∈ R

F_{θ}(u) = F_{0}(u − θ), θ ∈ R,  (1)

where F_{0} is a distribution function called standard. Here Θ = R. It is easily seen that if ξ has standard distribution function then the distribution function of ξ + θ is F_{θ}.

The distribution family with the condition (1) we call location family.

Consider the distribution family {F_{θ} : θ ∈ Θ} such that for any u ∈ R

F_{θ}(u) = F_{1} (\frac{u}{θ}) , θ > 0,  (2)

where F_{1} is a distribution function called standard. Here Θ = (0, ∞). It is easily seen that if ξ has standard distribution function then the distribution function of θξ is F_{θ}.

The distribution family with the condition (2) we call scale family.

Consider the distribution family {F_{θ} : θ ∈ Θ} such that for any u ∈ R

F_{θ}(u) = F_{(0,1)} (\frac{u − θ_{1}}{θ_{2}} ) , θ = (θ_{1}, θ_{2}), θ_{1} ∈ R, θ_{2} > 0,  (3)

where F_{(0,1)} is a distribution function called standard. Here Θ = R × (0, ∞). It is easily seen that if ξ has standard distribution function then the distribution function of θ_{2}ξ + θ_{1} is F_{θ}.

The distribution family with the condition (3) we call location-scale family.

The parametric model (X^{n}, B_{X^{n}}, P_{θ}^{(n)}) with the condition (3) is called the model with location and scale parameters.

Examples.

1. Normal distribution N(μ, σ). The standard distribution function is that of N(0, 1) distribution. It has the form

F_{(0,1)}(u) = \frac{1}{\sqrt{2π}} u ; ∫ ; −∞ e^{−z^{2}/2}dz, u ∈ R,

while the distribution function of N(μ, σ), μ ∈ R, σ > 0, is given as

F_{(μ,σ)}(u) = \frac{1}{\sqrt{2π}σ} u ; ∫ ; −∞ e^{−\frac{(z−μ)^{2}}{2σ^{2}}} dz = F_{(0,1)} (\frac{u − μ}{σ}) , u ∈ R.

Here μ is location parameter and σ is scale parameter.

2. Exponential distribution E(μ, σ). The standard distribution function is that of E(0, 1) distribution. It has the form

F_{(0,1)}(u) = 1 − e^{−u}, u ≥ 0,

while the distribution function of E(μ, σ), μ ∈ R, σ > 0, is given as

F_{(μ,σ)}(u) = 1 − e^{−\frac{u−μ}{σ}} = F_{(0,1)} (\frac{u − μ}{σ}) , u ≥ μ.

Here again μ is location parameter and σ is scale parameter.

3. Gamma distribution G(α, σ). It is not a location-scale family. The distribution function of G(α, σ), α > 0, σ > 0, is written as

F_{(α,σ)}(u) = \frac{1}{σ^{α}Γ(α)} u ; ∫ ; 0 z^{α−1}e^{−z/σ}dz, u ≥ 0,

where

Γ(α) = ∞ ; ∫ ; 0z^{α−1}e^{−z}dz.

Here σ is scale parameter, while α is shape parameter.

Two Methods for Estimation

Let (X^{n}, B_{X^{n}}, P_{θ}^{(n)}), θ ∈ Θ ⊂ R^{k}, be a parametric model. Our goal is to estimate θ basing on the sample x. Definition 1. We call statistic any function^{1} T (x) defined on X^{n}, taking values in R^{m}, m ≥ 1, and such that it does not depend on θ.

Hence, statistic can be regarded as a random vector (variable), which distribution is defined by P_{θ}.

There are two methods for estimation of θ :

• point estimation;

• estimation with the help of confidence regions (intervals).

We call any statistic T (x), which values are in Θ and regarded as approximations of θ, the estimator (point estimator) of θ.

In this case the matrix

R(θ, T) = E_{θ} ((T (x) − θ)^{T} (T (x) − θ))

is often taken to be crucial for choosing the criterion of picking out the estimator.

The second type of estimators are confidence regions (intervals).

1This function should be measurable.

Confidence Regions

Let α ∈ (0, 1) be a given number which we call confidence level. Let B = B(x) be a function from X^{n} to B_{Θ}. Definition 2. We say that a random set B is the confidence region of level α for θ if

P_{θ}^{(n)}(θ ∈ B(x)) ≥ α, ∀θ ∈ Θ.

We call the confidence region B strong if

P_{θ}^{(n)}(θ ∈ B(x)) = α, ∀θ ∈ Θ.

In the particular case k = 1, we speak about the confidence interval (T_{1}(x), T_{2}(x)) of level α for which

P_{θ}^{(n)}(T_{1}(x) ≤ θ ≤ T_{2}(x)) ≥ α (= α),

where T_{1}(x), T_{2}(x) are statistics taking values in Θ.

The quality of the confidence region is reasonable to characterize by the function

R(θ, B) = E_{θ}λ(B(x)),

where λ(B(x)) is a volume of set B(x). Definition 3. We call the confidence region B^{∗} optimal (in B_{Θ}) if

R(θ, B^{∗}) = min_{B∈B_{Θ}} R(θ, B).

Remark 1. The desirable property of confidence regions is connectedness. Remark 2. When k > 1, from the practical point of view it is reasonable also to consider confidence regions which are optimal for chosen class D of subsets from Θ, e. g. for the class of rectangulars

D = {[a_{1}, b_{1}] × [a_{2}, b_{2}] × ⋯ × [a_{k}, b_{k}] : a_{i} < b_{i}, i = 1, …, k}.

Of course, the optimal rectangular confidence region does not necessary be (and usually is not) the optimal one in B_{Θ}.

Equivariant and Invariant Statistics

Let (X^{n}, B_{X^{n}}, P_{θ}^{(n)}) be a parametric model with location parameter μ and scale parameter σ, i. e. θ = (μ, σ). Denote 1 = (1, 1, …, 1). All statistics, which we consider from now on, are real-valued.

Definition 4. Statistic T (x) is equivariant with respect to :

• location parameter if T (x + μ1) = T (x) + μ;

• scale parameter if T (σx) = σT (x);

• location and scale parameters if T (σx + μ1) = σT (x) + μ.

Definition 5. Statistic T (x) is invariant with respect to :

• location parameter if T (x + μ1) = T (x);

• scale parameter if T (σx) = T (x);

• location and scale parameters if T (σx + μ1) = T (x).

Example. Let

T_{1}(x) = x̄ = \frac{1}{n} n ; ∑ ; i=1 x_{i}, T_{2}(x) = s_{x} = \sqrt{\frac{1}{n} n ; ∑ ; i=1(x_{i} − x̄)^{2}}.

We have

T_{1}(σx + μ1) = \frac{1}{n} n ; ∑ ; i=1(σx_{i} + μ) = σx̄ + μ,

T_{2}(σx + μ1) = \sqrt{\frac{1}{n} n ; ∑ ; i=1((σx_{i} + μ) − (σx̄ + μ))^{2}} = σs_{x}.

Hence, statistic T_{1}(x) is equivariant with respect to location and scale parameters, while statistic T_{2}(x) is invariant with respect to location parameter and equivariant with respect to scale parameter.

Lemma 1. Let x be a sample drawn from the distribution P_{(μ,σ)}. If statistic T (x) is invariant with respect to location parameter and equivariant with respect to scale parameter, then for any Borel set A ⊂ R

P_{(μ,σ)}^{(n)} (\frac{T (x)}{σ} ∈ A) = P_{(0,1)}^{(n)} (T (y) ∈ A) = _{A}^{∫} f(u)du,

where f is the density function of T (y) when y drawn from the distribution P_{(0,1)}.

Proof.

P_{(μ,σ)}^{(n)} (\frac{T (x)}{σ} ∈ A) = P_{(0,1)}^{(n)} (\frac{T (σy + μ1)}{σ} ∈ A)

= P_{(0,1)}^{(n)} (\frac{σT (y)}{σ} ∈ A) = P_{(0,1)}^{(n)} (T (y) ∈ A) .

Corollary 1. Under the conditions of Lemma 1 for any Borel set A ⊂ R

P_{(μ,σ)}^{(n)} (\frac{(T (x))^{2}}{σ^{2}} ∈ A) = P_{(0,1)}^{(n)} ((T (y))^{2} ∈ A) = _{A}^{∫} f̃(u)du,

where f̃ is the density function of (T (y))^{2} when y drawn from the distribution P_{(0,1)}.

Corollary 2. Assume that the conditions of Lemma 1 are fulfilled and statistic T (x) is positive. Then for any Borel set A ⊂ R

P_{(μ,σ)}^{(n)} (\frac{σ}{T (x)} ∈ A) = P_{(0,1)}^{(n)} (\frac{1}{T (y)} ∈ A) = _{A}^{∫} \frac{1}{u^{2}}f (\frac{1}{u}) du,

where f is the density function of T (y) when y drawn from the distribution P_{(0,1)}.

Lemma 2. Let x be a sample drawn from the distribution P_{(μ,σ)}. If statistic T_{1}(x) is equivariant with respect to location and scale parameters and positive statistic T_{2}(x) is invariant with respect to location parameter and equivariant with respect to scale parameter and T_{1}(x), T_{2}(x) are independent, then for any Borel set A ⊂ R

P_{(μ,σ)}^{(n)} (\frac{T_{1}(x) − μ}{T_{2}(x)} ∈ A) = P_{(0,1)}^{(n)} (\frac{T_{1}(y)}{T_{2}(y)} ∈ A) = _{A}^{∫} ∞ ; ∫ ; 0vf_{1}(uv)f_{2}(v)dudv,

where f_{i} are density functions of T_{i}(y), i = 1, 2, respectively, when y drawn from the distribution P_{(0,1)}.

Proof.

P_{(μ,σ)}^{(n)} (\frac{T_{1}(x) − μ}{T_{2}(x)} ∈ A) = P_{(0,1)}^{(n)} (\frac{T_{1}(σy + μ1) − μ}{T_{2}(σy + μ1)} ∈ A)

= P_{(0,1)}^{(n)} (\frac{σT_{1}(y) + μ − μ}{σT_{2}(y)} ∈ A) = P_{(0,1)}^{(n)} (\frac{T_{1}(y)}{T_{2}(y)} ∈ A) .

Lemma 3. Under the conditions of Lemma 2 for any Borel set B ⊂ R^{2}

P_{(μ,σ)}^{(n)} ((\frac{T_{1}(x) − μ}{T_{2}(x)}, \frac{σ}{T_{2}(x)}) ∈ B) = P_{(0,1)}^{(n)} ((\frac{T_{1}(y)}{T_{2}(y)}, \frac{1}{T_{2}(y)}) ∈ B)

= \begin{aligned}∫ & ∫ \\ & B \end{aligned} \frac{1}{v^{3}}f_{1} (\frac{u}{v}) f_{2} (\frac{1}{v} ) dudv.

Proof.

P_{(μ,σ)}^{(n)} ((\frac{T_{1}(x) − μ}{T_{2}(x)}, \frac{σ}{T_{2}(x)}) ∈ B) = P_{(0,1)}^{(n)} ((\frac{T_{1}(σy + μ1) − μ}{T_{2}(σy + μ1)}, \frac{σ}{T_{2}(σy + μ1)}) ∈ B)

= P_{(0,1)}^{(n)} ((\frac{σT_{1}(y) + μ − μ}{σT_{2}(y)}, \frac{σ}{σT_{2}(y)}) ∈ B) = P_{(0,1)}^{(n)} ((\frac{T_{1}(y)}{T_{2}(y)}, \frac{1}{T_{2}(y)}) ∈ B) .

Questions and Exercises after Lecture 1

1. Let {F_{θ} : θ ∈ Θ} be a scale distribution family and ξ be a positive random variable having standard distribution function. What could you say about the distribution family of random variables ln(θξ) ?

2. Consider the family of Uniform distributions in the interval (θ_{1} − θ_{2}/2, θ_{1} + θ_{2}/2) , where θ_{1}, θ_{2} are unknown parameters. Is it an example of location-scale distribution family ?

3. Could you give an example of any other location-scale distribution family ?

4. Is there any difference in such two sentences:

i) We call statistic any function of the sample which does not depend on unknown parameters; ii) We call statistic any function of the sample which distribution does not depend on unknown parameters.

5. Check, please, what statistics among given below are invariant or equivariant with respect to location and/or scale parameters:

1) ax̄, a > 0, a ≠ 1,  2) as_{x}, a > 0, a ≠ 1,  3) \frac{x̄}{s_{x}},  4) n ; ∑ ; i=1 |x_{i}|,

5) n ; ∑ ; i=1 x_{i}^{2},  6) x_{(1)},  7) x − x_{(1)},  8) \frac{x − x_{(1)}}{s_{x}},

where x_{(1)} = min{x_{1}, x_{2}, …, x_{n}}.