Lecture 2 : Basic Notions. Location-Scale Families. Two Methods for Estimation. Comparison of Estimators. Unbiased Estimators. Optimal Linear Unbiased Estimators. Maximum Likelihood Estimators.

Basic Notions

ξ – random variable on the probability space (Ω, F, P), (X , B_{X}) – range space of ξ, X ⊂ R, F_{ξ}(u) = P (ξ ≤ u), u ∈ R – distribution function of ξ, f_{ξ}(u) = \frac{dF_{ξ}(u)}{du}, u ∈ R – density function of ξ (absolutely continuous), P_{ξ}(A) = P (ξ ∈ A) = _{A}^{∫} f_{ξ}(u)du, A ∈ B_{X} – distribution of ξ (absolutely continuous), {a_{i}, p_{i}}_{i≥1}, p_{i} = P (ξ = a_{i}) > 0, a_{i} ∈ R, ∑_{i} p_{i} = 1 – distribution of ξ (discrete), x = (x_{1}, x_{2}, …, x_{n}) – sample of size n drawn from the distribution P_{ξ}, x_{(1)} ≤ x_{(2)} ≤ … ≤ x_{(n)} – ordered statistics, (X^{n}, B_{X^{n}}) – sample space, X^{n} = X × ⋯ × X ,

P_{ξ}^{(n)}(A) = ∫ … ∫_{A} f_{ξ}(u_{1}) … f_{ξ}(u_{n})du_{1} … du_{n}, A ∈ B_{X^{n}},

(X^{n}, B_{X^{n}}, P_{ξ}^{(n)}) – statistical model (space). If P_{ξ} ∈ {P_{θ} : θ ∈ Θ}, Θ ⊂ R^{k}, k ≥ 1, then we call the statistical model parametric and denote it by (X^{n}, B_{X^{n}}, P_{θ}^{(n)}). Saying that we have a parametric model (X^{n}, B_{X^{n}}, P_{θ}^{(n)}) means that we have a sample x of size n drawn from the distribution P_{θ}. P_{ξ} ∈ {P_{θ} : θ ∈ Θ} ⟺ F_{ξ} ∈ {F_{θ} : θ ∈ Θ}

Location-Scale Families

Consider the distribution family {F_{θ} : θ ∈ Θ = R} such that

F_{θ}(u) = F_{0}(u − θ), u ∈ R,  (1)

where F_{0} is a distribution function called standard. It is easily seen that if ξ has a standard distribution function then the distribution function of ξ + θ is F_{θ}.

The distribution family with (1) is called the location family.

Consider the distribution family {F_{θ} : θ ∈ Θ = (0, ∞)} such that

F_{θ}(u) = F_{1} (\frac{u}{θ}) , u ∈ R,  (2)

where F_{1} is a distribution function called standard. It is easily seen that if ξ has a standard distribution function then the distribution function of θξ is F_{θ}.

The distribution family with (2) is called the scale family.

Consider the distribution family {F_{θ} : θ ∈ Θ = R × (0, ∞)} such that

F_{θ}(u) = F_{(0,1)} (\frac{u − θ_{1}}{θ_{2}} ) , u ∈ R, θ = (θ_{1}, θ_{2}),  (3)

where F_{(0,1)} is a distribution function called standard. It is easily seen that if ξ has a standard distribution function then the distribution function of θ_{2}ξ + θ_{1} is F_{θ}.

The distribution family with (3) is called the location-scale family.

The parametric model (X^{n}, B_{X^{n}}, P_{θ}^{(n)}) with the condition (1), (2) or (3) is called the model with location, scale or location and scale parameters, respectively.

Examples.

1. Normal location-scale family N(μ, σ), μ ∈ R, σ > 0. The standard distribution function is that of N(0, 1) distribution :

F_{(0,1)}(u) = \frac{1}{\sqrt{2π}} u ; ∫ ; −∞ e^{−\frac{z^{2}}{2}} dz, u ∈ R.

The distribution function of N(μ, σ) is given as

F_{(μ,σ)}(u) = \frac{1}{\sqrt{2π}σ} u ; ∫ ; −∞ e^{−\frac{(z−μ)^{2}}{2σ^{2}}} dz = F_{(0,1)} (\frac{u − μ}{σ}) , u ∈ R.

Hence, μ is a location parameter = the expectation and σ is a scale parameter = the standard deviation.

2. Exponential location-scale family E(μ, σ), μ ∈ R, σ > 0. The standard distribution function is that of E(0, 1) distribution :

F_{(0,1)}(u) = 1 − e^{−u}, u ≥ 0.

The distribution function of E(μ, σ) is given as

F_{(μ,σ)}(u) = 1 − e^{−\frac{u−μ}{σ}} = F_{(0,1)} (\frac{u − μ}{σ}) , u ≥ μ.

Here again μ is a location parameter and σ is a scale parameter.

3. Gamma distribution G(α, σ), α > 0, σ > 0. The distribution function of G(α, σ) is written as

F_{(α,σ)}(u) = \frac{1}{σ^{α}Γ(α)} u ; ∫ ; 0 z^{α−1}e^{−\frac{z}{σ}} dz, u ≥ 0,

where

Γ(α) = ∞ ; ∫ ; 0z^{α−1}e^{−z}dz.

Here σ is a scale parameter, while α is a shape parameter. It is not a location-scale family.

Two Methods for Estimation

Let (X^{n}, B_{X^{n}}, P_{θ}^{(n)}), θ ∈ Θ ⊂ R^{k}, be a parametric model. Our goal is to estimate θ basing on the sample x. Definition 1. A function^{1} t(x) defined on X^{n} and taking values in R^{k} is said to be a statistic if it does not depend on θ.

A statistic can be regarded as a random vector (variable), which distribution is defined by P_{θ}. Examples.

1. x̄ = \frac{x_{1} + x_{2} + … + x_{n}}{n} is a statistic,

2. θ_{1}x̄ is not a statistic.

There are two methods for estimation of θ :

• point estimation;

• confidence set (interval) estimation.

We call any statistic θ̂(x), which values are in Θ and regarded as approximations of θ, the estimator (point estimator) of θ. How can we evaluate the quality of an estimator ? How to pick out the best estimator ?

Define a risk function as

R(θ̂(x), θ) = E_{θ}(l(θ̂(x), θ)),

where l(·, ·) is the so-called loss function. The risk function is often taken to be crucial for picking out the estimator. We take

l(θ̂, θ) = ∥θ̂ − θ∥^{2}.

1This function should be measurable.

Comparison of Estimators. Unbiased Estimators

Let θ̂(x) and θ̃(x) be two estimators of θ. Definition 2. We say that the estimator θ̂ of θ is better than θ̃ if

R(θ̂, θ) ≤ R(θ̃, θ) ∀θ ∈ Θ

and there exists θ for which the above inequality is sharp.

Is it a good definition ? Example. Let x = (x_{1}, x_{2}, …, x_{n}) ∼ N(θ, 1). Consider θ̂(x) = x̄, θ̃(x) = 5. Then

R(θ̂, θ) = \frac{1}{n}, R(θ̃, θ) = (θ − 5)^{2}.

The estimators are uncomparable !

Let bθ̂(θ) = E_{θ}(θ̂ − θ) be the bias of the estimator θ̂. Then it is easy to show that

R(θ̂, θ) = k ; ∑ ; i=1 V ar_{θ}(θ̂_{i}) + ∥bθ̂(θ)∥^{2}.

In the above example, R(θ̂, θ) = V ar_{θ}(θ̂), R(θ̃, θ) = b2 ; θ̃(θ). Definition 3. We say that the estimator θ̂(x) of θ is unbiased if

E_{θ}(θ̂(x)) = θ ∀θ ∈ Θ.

Example. x = (x_{1}, x_{2}, …, x_{n}) ∼ N(μ, σ). The estimator μ̂ = x̄ of μ is unbiased. The estimator σ̂^{2} = \frac{1}{n} ∑_{i=1}^{n}(x_{i} − x̄)^{2} of σ^{2} is biased (Exercise 5).

Linear Unbiased Estimators

Location parameter. x = (x_{1}, x_{2}, …, x_{n}) ∼ F_{0}(u − θ), θ ∈ R.

Consider linear estimators of the form θ̂(x) = ∑_{j=1}^{n} c_{j}x_{j}. When do they be unbiased ? Denote a = E_{0}(x_{1}). Then E_{θ}(x_{1}) = a + θ. We have

E_{θ}(θ̂(x)) = n ; ∑ ; j=1 c_{j}(a + θ) = θ ∀θ ∈ R ⟺ n ; ∑ ; j=1 c_{j} = 1, a = 0.

Hence, they can not be unbiased at all if a ≠ 0 ! But we can modify them a little to make them unbiased. Indeed, the estimators

θ̂(x) = n ; ∑ ; j=1 c_{j}(x_{j} − a) = n ; ∑ ; j=1 c_{j}x_{j} − a with n ; ∑ ; j=1 c_{j} = 1  (4)

are unbiased.

Scale parameter. x = (x_{1}, x_{2}, …, x_{n}) ∼ F_{1}(\frac{u}{θ}), θ > 0.

Consider θ̂(x) = ∑_{j=1}^{n} c_{j}x_{j}. Denote a = E_{1}(x_{1}). Then E_{θ}(x_{1}) = aθ and

E_{θ}(θ̂(x)) = n ; ∑ ; j=1 c_{j}aθ = θ ∀θ > 0 ⟺ n ; ∑ ; j=1 c_{j} = \frac{1}{a}, a ≠ 0.

Hence, they can not be unbiased if a = 0. But if a ≠ 0 then the estimators

θ̂(x) = n ; ∑ ; j=1 c_{j}x_{j}  with n ; ∑ ; j=1 c_{j} = \frac{1}{a}  (5)

are unbiased.

Location and scale parameter. x = (x_{1}, x_{2}, …, x_{n}) ∼ F_{(0,1)}(\frac{u−θ_{1}}{θ_{2}}), θ_{1} ∈ R, θ_{2} > 0. Denote a = E_{(0,1)}(x_{1}). Then E_{θ}(x_{1}) = θ_{2}a + θ_{1}. Linear estimators can not be unbiased :

E_{θ} (θ̂(x)) = E_{θ} (( \begin{matrix}∑_{j=1}^{n} c_{j}x_{j} + c \\ ∑_{j=1}^{n} d_{j}x_{j} + d \end{matrix} )) = ( \begin{matrix}θ_{1} \\ θ_{2} \end{matrix} ) ∀θ_{1} ∈ R, θ_{2} > 0 ⟺

n ; ∑ ; j=1 c_{j} = 1, c = −aθ_{2}, n ; ∑ ; j=1 d_{j} = \frac{1}{a}, d = −\frac{θ_{1}}{a}, a ≠ 0.

Optimal Linear Unbiased Estimators

Scale parameter. x = (x_{1}, x_{2}, …, x_{n}) ∼ F_{1}(\frac{u}{θ}), θ > 0, a = E_{1}(x_{1}) ≠ 0.

R(θ̂(x), θ) = V ar_{θ}(θ̂(x)) = V ar_{θ}( n ; ∑ ; j=1 c_{j}x_{j}) → min ; (c_{1},...,c_{n}):∑ c_{j}=\frac{1}{a} ∀θ > 0.

We have

V ar_{θ}(θ̂(x)) = E_{θ}( n ; ∑ ; j=1 c_{j}x_{j}−θ)^{2} = E_{θ}( n ; ∑ ; j=1 c_{j}(x_{j}−aθ))^{2}

= n ; ∑ ; i=1 n ; ∑ ; j=1 c_{j}c_{i}E_{θ}((x_{i}−aθ)(x_{j}−aθ)) = n ; ∑ ; j=1 c_{j}^{2}E_{θ}(x_{j}−E_{θ}x_{j})^{2} = V ar_{θ}(x_{1})· n ; ∑ ; j=1 c_{j}^{2}.

Consider the Lagrangian function

L(c_{1}, c_{2}, …, c_{n}, λ) = n ; ∑ ; j=1 c_{j}^{2} + λ ( n ; ∑ ; j=1 c_{j} − \frac{1}{a}) .

Due to the necessary condition for an extremum,

\{ \begin{matrix}2c_{i} + λ & = & 0, & i = 1, 2, …, n \\ ∑_{j=1}^{n} c_{j} & = & \frac{1}{a} \end{matrix} .

The solution is : c_{1}^{∗} = … = c_{n}^{∗} = \frac{1}{na}, λ^{∗} = −\frac{2}{na}. Since Hess L_{cc}(c^{∗}, λ^{∗}) = 2I_{n} > 0, the point c^{∗} = (\frac{1}{na}, …, \frac{1}{na}) is really that of minimum. The optimal estimator is θ̂(x) = \frac{x̄}{a}.

Location parameter. x = (x_{1}, x_{2}, …, x_{n}) ∼ F_{0}(u − θ), θ ∈ R, a = E_{0}(x_{1}). Similarly,

R(θ̂(x), θ) = V ar_{θ} ( n ; ∑ ; j=1 c_{j}x_{j} − a) = V ar_{θ}(x_{1})· n ; ∑ ; j=1 c_{j}^{2} → min ; (c_{1},...,c_{n}):∑ c_{j}=1 ∀θ ∈ R

and the optimal estimator is θ̂(x) = x̄ − a.

Optimal Linear Unbiased Estimators (continuation)

Examples.

1. Normal location family N(μ, 1), μ ∈ R. Here a = E_{0}(x_{1}) = 0. The optimal linear unbiased estimator for μ is μ̂(x) = x̄.

2. Exponential location family E(μ, 1), μ ∈ R. Here a = E_{0}(x_{1}) = 1. The optimal linear unbiased estimator for μ is μ̂(x) = x̄ − 1.

3. Normal scale family N(0, σ), σ > 0. Here a = E_{1}(x_{1}) = 0. Hence, there does not exist linear unbiased estimator for σ.

4. Exponential scale family E(0, σ), σ > 0. Here a = E_{1}(x_{1}) = 1. The optimal linear unbiased estimator for σ is σ̂(x) = x̄.

Maximum Likelihood Estimators

Let x = (x_{1}, x_{2}, …, x_{n}) be a sample of size n drawn from P_{θ}. Maximum likelihood function :

p(x, θ) = n ; ∏ ; j=1 f_{θ}(x_{j}) − if distribution is absolutely continuous ,

p(x, θ) = n ; ∏ ; j=1 P_{θ}(ξ = x_{j}) − if distribution is discrete .

Definition 4. Any point θ^{∗}(x) from the set arg max_{θ∈Θ} p(x, θ) = arg max_{θ∈Θ} ln p(x, θ)

is said to be the maximum likelihood estimator of θ.

Examples.

1. Normal distribution N(μ, σ^{2}), μ ∈ R, σ > 0.

p(x, μ, σ^{2}) = \frac{1}{(2πσ^{2})^{\frac{n}{2}}} e^{−\frac{1}{2σ^{2}} ∑_{j=1}^{n}(x_{j}−μ)^{2}} ⟹

ln p(x, μ, σ^{2}) = −\frac{n}{2} ln(2π) − \frac{n}{2} ln σ^{2} − \frac{1}{2σ^{2}} n ; ∑ ; j=1(x_{j} − μ)^{2}.

Due to the necessary condition for an extremum,

\{ \begin{matrix}\frac{∂ ln p(x,μ,σ^{2})}{∂μ} & = & 0 \\ \frac{∂ ln p(x,μ,σ^{2})}{∂σ^{2}} & = & 0 \end{matrix} ⟺ \{ \begin{matrix}\frac{1}{σ^{2}} ∑_{j=1}^{n}(x_{j} − μ) & = & 0 \\ −\frac{n}{2σ^{2}} + \frac{1}{2σ^{4}} ∑_{j=1}^{n}(x_{j} − μ)^{2} & = & 0 \end{matrix}

⟺  \{ \begin{matrix}μ^{∗} & = & x̄ \\ (σ^{2})^{∗} & = & \frac{1}{n} ∑_{j=1}^{n}(x_{j} − x̄)^{2} \end{matrix} .

Now we use the sufficient condition to verify whether (μ^{∗}, (σ^{2})^{∗}) is really the point of maximum :

Hess ln p(x, μ, σ^{2}) = ( \begin{matrix}−\frac{n}{σ^{2}} & −\frac{1}{σ^{4}} n ; ∑ ; j=1(x_{j} − μ) \\ −\frac{1}{σ^{4}} n ; ∑ ; j=1(x_{j} − μ) & \frac{n}{2σ^{4}} − \frac{1}{σ^{6}} n ; ∑ ; j=1(x_{j} − μ)^{2} \end{matrix} ) ⟹

Hess ln p(x, μ^{∗}, (σ^{2})^{∗}) = ( \begin{matrix}−\frac{n^{2}}{∑_{j=1}^{n}(x_{i} − x̄)^{2}} & 0 \\ 0 & −\frac{n^{3}}{2(∑_{j=1}^{n}(x_{i} − x̄)^{2})^{2}} \end{matrix} ) < 0.

As we know, the estimator μ^{∗} of μ is unbiased, while the estimator (σ^{2})^{∗} of σ is biased.

2. Uniform distribution U(θ_{1}, θ_{2}), θ_{1} < θ_{2}. In this case

p(x, θ_{1}, θ_{2}) = \frac{1}{(θ_{2} − θ_{1})^{n}}, θ_{1} ≤ x_{(1)}, θ_{2} ≥ x_{(n)}.

Clearly, p(x, θ_{1}, θ_{2}) is an increasing function of θ_{1} ∈ (−∞, x_{(1)}] and a decreasing function of θ_{2} ∈ [x_{(n)}, ∞). Therefore, its maximum with respect to θ_{1} is attained at θ_{1}^{∗} = x_{(1)} while that with respect to θ_{2} is attained at θ_{2}^{∗} = x_{(n)}.

Questions and Exercises after Lecture 2

1. Let {F_{θ} : θ ∈ Θ} be a scale distribution family and ξ be a positive random variable having the standard distribution function. What could you say about the distribution family of random variables ln(θξ) ?

2. Consider the family of Uniform distributions in the interval (θ_{1} − θ_{2}/2, θ_{1} + θ_{2}/2) , where θ_{1} ∈ R, θ_{2} ∈ (0, ∞) are unknown parameters. Is it an example of the location-scale distribution family ?

3. Could you give any other example of the location-scale distribution family ?

4. Is there any difference in such two sentences:

i) We call statistic any function of the sample which does not depend on the unknown parameters; ii) We call statistic any function of the sample which distribution does not depend on the unknown parameters.

5. Calculate E_{(a,σ^{2})}σ̂^{2} for the sample (x_{1}, x_{2}, …, x_{n}) ∼ N(μ, σ). How to make this estimator of σ^{2} unbiased ?

6. What is the relation between the quantile of the distribution F_{θ} and that of a standard distribution within the location-scale family ?

Construct an unbiased estimator of the quantile in the case of N(μ, σ) distribution.

7. Let (x_{1}, x_{2}, …, x_{n}) ∼ E(μ, σ). Calculate E_{(μ,σ)}x_{(1)}. Is it possible to construct an unbiased estimator for μ basing on x_{(1)} ?

8. Utilize estimators ∑_{j=1}^{n} c_{j}x_{(j)} instead of ∑_{j=1}^{n} c_{j}x_{j} for construction of unbiased estimators for location parameter, scale parameter, location and scale parameters together. Do they exist ? Which estimators are optimal ?

9. Do there exist linear unbiased estimators for location parameter, scale parameter, location and scale parameters together in the case of Uniform location-scale family (see Exercise 2) ? If yes then which estimators are optimal ?

10. Construct the maximum likelihood estimators for the unknown parameters of

(a) the exponential distribution E(μ, σ), μ ∈ R, σ > 0;

(b) the Poisson distribution P (ξ = k) = \frac{λ^{k}e^{−λ}}{k!}, λ > 0, k = 0, 1, …;

(c) the geometric distribution P (ξ = k) = p(1 − p)^{k}, 0 < p < 1, k = 0, 1, ….