Lecture 3 : Second-Order Stochastic Processes. Covariance Functions. Gaussian Processes. Stationary Processes in the Wide Sense and Strictly Stationary Processes. Herglotz’s Theorem. Bochner’s Theorem.

Second-Order Stochastic Processes. Covariance Functions

Definition 1. A stochastic process {X(t), t ∈ T } is called a second-order stochastic process if E(X(t))^{2} < ∞ for every t ∈ T.

As we already know the family of all finite-dimensional distributions completely characterize a stochastic process. However, in many cases, some more compressed characteristics of a stochastic process are of interest. In the case of second-order stochastic processes, the most important characteristics are the mean function and the covariance function.

Definition 2. Covariance function of a second-order stochastic process {X(t), t ∈ T } is given by

K(s, t) = Cov (X(s), X(t)) = E(X(s) − EX(s))(X(t) − EX(t))
= E(X(s) − m(s))(X(t) − m(t)) = EX(s)X(t) − m(s)m(t), (s, t) ∈ T × T,

where m(t) = EX(t) is the mean function of the stochastic process.

Examples.

1. Brownian Motion {W (t), t ≥ 0} is a second-order stochastic process since E(W (t))^{2} = t < ∞ for every t ≥ 0. As we already know, m(t) = 0, K(s, t) = min{s, t}.

2. Poisson Process {Y (t), t ≥ 0} is a second-order stochastic process since E(Y (t))^{2} = μ^{2}t^{2} + μt < ∞ for every t ≥ 0. As we already know, m(t) = μt, K(s, t) = μ min{s, t}.

3. Let {X(t), t ≥ 0} be a second-order stochastic process having independent increments and let X(0) = 0. What can we say about the covariance function ? Set B(t) = E((X(t) − m(t))^{2}. If 0 ≤ s ≤ t then

K(s, t) = B(s) + E (X(t) − m(t) − (X(s) − m(s))) (X(s) − m(s)) = B(s)+

E(X(t)−X(s))(X(s)−m(s)) = B(s)+E(X(t)−X(s))X(s)−(m(t)−m(s))m(s) = B(s).

In the same way it can be shown that for 0 ≤ t ≤ s we have K(s, t) = B(t) ⟹ K(s, t) = B(min{s, t}).

Some Properties of Covariance Functions.

1. K(t, t) = E((X(t) − m(t))^{2} = V ar X(t) ≥ 0 for every t ∈ T, and the equality is possible only in the case when X(t) = m(t) with probability 1, i.e. when X(t), actually, is not random.

2. K(s, t) = K(t, s) for every (s, t) ∈ T × T (symmetricity).

3. (K(s, t))^{2} ≤ K(s, s)K(t, t) for every (s, t) ∈ T × T (follows from the Cauchy-Schwartz inequality).

4. For any n, any t_{1}, …, t_{n} ∈ T and any real numbers z_{1}, …, z_{n},

∑_{k,i=1}^{n} z_{k}z_{i}K(t_{k}, t_{i}) ≥ 0 (nonnegative definiteness).

Indeed,

∑_{k,i=1}^{n} z_{k}z_{i}K(t_{k}, t_{i}) = E ( ∑_{k,i=1}^{n} z_{k}z_{i}(X(t_{k}) − m(t_{k}))(X(t_{i}) − m(t_{i})))

= E (∑_{k=1}^{n} z_{k}(X(t_{k}) − m(t_{k})))^{2} = V ar (∑_{k=1}^{n} z_{k}X(t_{k})) ≥ 0.

If a stochastic process {X(t), t ∈ T } has the covariance function K(s, t) and the mean function m(t) and f is any function f : T → R, then the stochastic process {X(t) + f(t), t ∈ T } has the same covariance function K(s, t) but the mean function m(t) + f(t). Thus, changing the mean function does not change the covariance, and we can suppose further on dealing with covariance functions, without loss of generality, that m(t) = 0. It also means that the covariance function has no influence on sample paths of the process !

It is worth also to note that the second property (symmetricity) together with fourth (nonnegative definiteness) are the fundamental properties of covariance functions. The first property immediately follows from the nonnegative definiteness if we take n = 1. On the other hand, the third property follows from the symmetricity and nonegative definiteness (see Exercise 2).

Gaussian Processes. Characterization of Covariance Function.

Definition 3. A stochastic process {X(t), t ∈ T } is said to be a Gaussian process iff for each t_{1}, …, t_{n} ∈ T, n = 1, 2, …, the random vector (X(t_{1}), …, X(t_{n})) is jointly Gaussian.

Of course, a Gaussian process is a second-order process.

Of course, the Brownian motion is a Gaussian process (see Exercise 5).

Lemma 1. The function K(s, t), (s, t) ∈ T × T, is a covariance function of a secondorder stochastic process iff it is symmetric and nonnegative definite.

Proof. As we already know, if K(s, t), (s, t) ∈ T × T, is a covariance function of a second-order stochastic process then it is symmetric and nonnegative definite. Therefore, we need only to prove that if K(s, t), (s, t) ∈ T ×T, is symmetric and nonnegative definite then there exists a second-order stochastic process whose covariance function is precisely K. It is easy to show even more, namely, that there exists a Gaussian process {X(t), t ∈ T } whose covariance function is precisely K.

Indeed, let t_{1}, …, t_{n} ∈ T, t_{1} < … < t_{n}, be given. We specify a finite-dimensional distribution of the process {X(t), t ∈ T } as such that the vector (X(t_{1}), …, X(t_{n})) is Gaussian with the covariance matrix ∥K(t_{k}, t_{i})∥, k, i = 1, …, n. Then, it is known, that any subvector (X(t_{i_{1}}), …, X(t_{i_{j}})) is also Gaussian with the covariance matrix ∥K(t_{i_{r}}, t_{i_{s}})∥, r, s = 1, …, j. It now remains to apply the Kolmogorov theorem and construct a process with such specified finite-dimensional distributions.

Stationary Processes

Definition 4. A second-order stochastic process {X(t), t ∈ T } is said to be stationary in the wide sense iff its mean function m(t) = const for all t ∈ T, while the covariance function possesses the property K(s, t) = K(s + h, t + h) for all s, t, h such that s, t, s + h, t + h ∈ T.

In other words, if T contains zero then the covariance function K(s, t) depends only on the difference between s and t, so that K(s + t, s) may be expressed as a function of t alone. Abusing the notation slightly, we write K(s + t, s) = K(t); that is

K(t) = Cov (X(s + t), X(s)) = E(X(s + t) − m)(X(s) − m).

It is very useful to rewrite all properties of covariance functions given on page 3 for covariance functions of stationary in the wide sense stochastic processes.

Corollary 1. The function K(t) = Cov (X(s + t), X(s)), t ∈ T, is a covariance function of a stationary in the wide sense stochastic process iff it is symmetric, i.e. K(−t) = K(t) for every t ∈ T, and nonnegative definite, i.e. ∑_{k,i=1}^{n} z_{k}z_{i}K(t_{k} − t_{i}) ≥ 0 for any n, any t_{1}, …, t_{n} ∈ T and any real numbers z_{1}, …, z_{n}.

Definition 5. An arbitrary stochastic process {X(t), t ∈ T } is called strictly stationary iff all its finite-dimensional distribution functions F_{t_{1},...,t_{n}}(x_{1}, …, x_{n}) = P (X(t_{1}) ≤ x_{1}, …, X(t_{n}) ≤ x_{n}) have the property

F_{t_{1},...,t_{n}}(x_{1}, …, x_{n}) = F_{t_{1}+h,...,t_{n}+h}(x_{1}, …, x_{n}) ∀(x_{1}, …, x_{n}) ∈ R^{n}

for all n and all t_{1}, …, t_{n}, h such that t_{1} < … < t_{n} and all t_{i}, t_{i} + h ∈ T.

In other words, the joint distribution function of X(t_{1}), …, X(t_{n}) depends only on the differencies t_{2} − t_{1}, …, t_{n} − t_{n−1}. Thus, if the ”observation points” t_{1}, …, t_{n} are translated rigidly, the joint distribution is not changed.

Dependence Between Two Types of Stationarities and Examples

It is worth to note that strict stationarity implies wide sense stationarity for secondorder stochastic processes. Indeed (for simplicity suppose that all distribution functions F_{t_{1},...,t_{n}} possess densities f_{t_{1},...,t_{n}}),

E(X(s)X(t)) = ∫ ∫ xyf_{s,t}(x, y)dxdy = ∫ ∫ xyf_{s+h,t+h}(x, y)dxdy = E(X(s+h)X(t+h))

and similarly EX(t) = EX(t + h).

The converse is not true however. For example, let T be the set of all integers and X(t), t ∈ T, be independent random variables with means zero and variances equal to 1. Then m(t) = EX(t) = 0, while

K(s + t, s) = E(X(s + t)X(s)) = \{ \begin{matrix}1, & if t = 0 \\ 0, & if t ≠ 0 \end{matrix} .

Hence it is a stationary in the wide sense stochastic process. But the process need not be strictly stationary. For example, we might take X(t) to be normally distributed for nonnegative integers and uniformly distributed in the interval [−\sqrt{3}, \sqrt{3}] for negative integers. Thus F_{t}(x) = P (X(t) ≤ x) may depend on t.

Examples.

1. Gaussian process is not necessary a stationary in the wide sense process. On the other hand, for a Gaussian process the covariance function completely determines all finite-dimensional distributions, hence stationarity in the wide sense is equivalent to strict stationarity.

2. Brownian motion can never be a stationary in the wide sense process and therefore can never be a strictly stationary (see Exercise 6) ! But it is interesting to note that the stochastic process {W (t + 1) − W (t), t ≥ 0}, where {W (t), t ≥ 0} is a Brownian motion, is already stationary in the wide sense and strictly stationary stochastic process (see Exercise 7).

3. The same conclusion as for Brownian motion could be made also for Poisson process.

4. Let Z_{1}, Z_{2} be independent identically distributed random variables having normal N(0, σ^{2}) distribution. Define the stochastic process {X(t), t ∈ R} as X(t) = Z_{1} cos(λt) + Z_{2} sin(λt), where λ is a fixed real number.

Clearly, its mean function m(t) = 0, t ∈ R, and the covariance function

K(s, t) = E(X(s)X(t)) = E (Z_{1} cos(λs) + Z_{2} sin(λs)) (Z_{1} cos(λt) + Z_{2} sin(λt)) =

EZ_{1}^{2} cos(λs) cos(λt) + EZ_{2}^{2} sin(λs) sin(λt) = σ^{2} cos(λ(t − s)), s, t ∈ R.

Therefore, it is a second order stationary in the wide sense stochastic process.

It is also a Gaussian process since

( X(t_{1}) ; ⋮ ; X(t_{n}) ) = ( \begin{matrix}cos(λt_{1}) & sin(λt_{1}) \\ . . . . . . & . . . . . . \\ cos(λt_{n}) & sin(λt_{n}) \end{matrix} ) ( \begin{matrix}Z_{1} \\ Z_{2} \end{matrix} ) .

Hence the process is also a strictly stationary process.

5. Let T be the set of all integers, while X_{t}, t ∈ T, be independent identically distributed random variables having normal N(0, 1) distribution. Define a stochastic process {X(t), t ∈ T } as X(t) = X_{t}, t ∈ T. Such a process is called the white noise stochastic process.

It is easy to see that m(t) = 0, t ∈ T, while

K(s + t, s) = \{ \begin{matrix}1, & if t = 0 \\ 0, & if t ≠ 0 \end{matrix} .

Hence, the white noise is a stationary in the wide sense stochastic process and since it is also Gaussian, it is a strictly stationary stochastic process.

6. Take the same T and X_{t}, t ∈ T, as in the previous example. Define a stochastic process {Y (t), t ∈ T } as Y (t) = c_{0}X_{t} + … + c_{r}X_{t−r}, t ∈ T, where {c_{k}} are given real numbers and r is a given integer. Such a process is called the moving average stochastic process.

Of course, it is a Gaussian process. Moreover, m(t) = 0, t ∈ T,

K(s + t, s) = \{ \begin{matrix}c_{0}c_{|t|} + c_{1}c_{|t|+1} + ⋯ + c_{r−|t|}c_{r}, & if |t| ≤ r \\ 0, & if |t| > r \end{matrix} .

Hence, the process is a stationary in the wide sense and a strictly stationary stochastic process.

Herglotz’s Theorem

Theorem 1. (Herglotz) A function K(t), t ∈ T – the set of all integers, is the covariance function of a stationary in the wide sense stochastic process iff there exists a random variable ξ taking values in [−π, π] and having a symmetrical about the origin distribution such that

K(t) = K(0)E cos(tξ)  ∀t ∈ T.  (1)

The distribution of ξ is called the spectral distribution of the process.

Proof. We consider only the case when K(0) > 0 otherwise the statement of the theorem is trivial (see property 1 on page 3). (⟸) If (1) holds then it is clear that K is symmetric. It is also nonnegative definite since for any n, any t_{1}, …, t_{n} and any real numbers z_{1}, …, z_{n},

∑_{k,i=1}^{n} z_{k}z_{i}K(t_{k} − t_{i}) = K(0)E ( ∑_{k,i=1}^{n} z_{k}z_{i} cos ((t_{k} − t_{i})ξ)) =

K(0)E ( ∑_{k,i=1}^{n} z_{k}z_{i} cos(t_{k}ξ) cos(t_{i}ξ) + ∑_{k,i=1}^{n} z_{k}z_{i} sin(t_{k}ξ) sin(t_{i}ξ)) =

K(0)E ((∑_{k=1}^{n} z_{k} cos(t_{k}ξ))^{2} + (∑_{k=1}^{n} z_{k} sin(t_{k}ξ))^{2}) ≥ 0.

The desired statement follows now from Corollary 1. (⟹) For x ∈ [−π, π] define the function

g_{N}(x) = \frac{1}{2πNK(0)} ∑_{k,i=1}^{N} cos((k − i)x)K(k − i).

It is symmetrical about the origin.

Due to Corollary 1, K(t) is nonnegative definite. Hence, g_{N}(x) ≥ 0. Since the number of values (k, i) in the sum such that k − i = s is (N − |s|), s = −(N − 1), …, 0, …, (N − 1), we can rewrite

g_{N}(x) = \frac{1}{2πNK(0)} ∑_{s=−(N−1)}^{N−1}(N − |s|)K(s) cos(sx).

Since

∫_{−π}^{π} g_{N}(x)dx = \frac{1}{2πNK(0)} ∑_{s=−(N−1)}^{N−1}(N − |s|)K(s) ∫_{−π}^{π} cos(sx)dx = 1,

g_{N}(x) is a density function of some random variable ξ_{N} taking values in [−π, π] and having symmetrical about the origin distribution.

It is possible to show that for the sequence of random variables {ξ_{N}}_{N≥1} there exists a random variable ξ also taking values in [−π, π] and having a symmetrical distribution such that the distribution function of ξ_{N_{k}} for some subsequence {N_{k}}_{k≥1} tends to that of ξ at any point of continuity as k → ∞. Since

E(cos(tξ_{N})) = \frac{1}{2πNK(0)} ∑_{s=−(N−1)}^{N−1}(N − |s|)K(s) ∫_{−π}^{π} cos(tx) cos(sx)dx =

\{ \begin{matrix}(1 − |t|/N)K(t)/K(0), & if |t| < N \\ 0, & if |t| ≥ N \end{matrix} ,

letting N approach infinity along the sequence N_{1}, N_{2}, … in both sides of the equality, we get (1).

Examples.

1. Let ξ from Herglotz’s theorem be such that P (ξ = 0) = 1. What is the correspondent covariance function and how to construct an example of stochastic process having such a spectral distribution ?

The corresponding covariance function is K(t) = K(0) cos(t · 0) = K(0), i.e. it is a function which is constant for all integers. Let X be any random variable with zero mean and variance σ^{2}. Define a process X(t) = X, t ∈ T. For such a process K(s + t, s) = EX^{2} = σ^{2}.

2. Let ξ from Herglotz’s theorem be uniformly distributed in [−π, π]. What is the correspondent covariance function and how to construct an example of stochastic process having such a spectral distribution ?

The correspondent covariance function is: K(t) = 0, t ≠ 0 and K(0) > 0. An example of the stochastic process having such a spectral density: the white noise.

Bochner’s Theorem

Theorem 2. (Bochner) A function K(t), t ∈ R, that is continuous at the origin is the covariance function of a stationary in the wide sense stochastic process iff there exists a random variable ξ having a symmetrical about the origin distribution such that

K(t) = K(0)E cos(tξ)  ∀t ∈ R.  (2)

Remark. Bochner’s theorem is a generalization of Herglotz’s theorem since if T is the set of all integers, the condition of continuity at the origin is automatically satisfied.

Summary. The following statements about the function K(t), t ∈ R, are equivalent:

(i) K is symmetric, nonnegative definite and continuous at the origin;

(ii) K is the covariance function of a stationary in the wide sense second-order stochastic process and K is continuous at the origin;

(iii) K has a representation (2).

Questions and Exercises

1. Let {X(t), t ≥ 0} be a second-order stochastic process having independent stationary increments and let X(0) = 0. Show that m(t_{1} + t_{2}) = m(t_{1}) + m(t_{2}), B(t_{1} + t_{2}) = B(t_{1}) + B(t_{2}).

2. Show that if the function K(s, t) on T × T is symmetric and nonnegative definite then (K(s, t))^{2} ≤ K(s, s)K(t, t) for every (s, t) ∈ T × T. (Prompt. Consider two cases K(s, t) = 0 and K(s, t) ≠ 0. In the second case use the definition of nonnegative definiteness with n = 2, z_{1} = 1, z_{2} = −x/K(s, t), t_{1} = s, t_{2} = t, where x is an arbitrary real number).

3. Show that the function min{s, t}, s, t ≥ 0, is nonegative definite.

4. If both the functions K_{1}(s, t) and K_{2}(s, t) are covariances (symmetric and nonnegative definite), is it true that the following are also covariances:

(i) K_{1}(s, t) + K_{2}(s, t); (ii) K_{1}(s, t)K_{2}(s, t).

5. Show that the Brownian motion is a Gaussian process.

6. Taking the definition of strictly stationarity and n = 2 prove that the Brownian motion is not a strictly stationary process.

7. Show that the stochastic process {W (t + 1) − W (t), t ≥ 0}, where {W (t), t ≥ 0} is a Brownian motion, is stationary in the wide sense stochastic process. What about strictly stationarity ?

8. Let {W (t), t ≥ 0} be a Brownian motion. Calculate the mean function and the covariance function of the following stochastic process {W (t) − tW (1), t ∈ [0, 1]} (Brownian bridge). Is it a stationary in the wide sense and strictly stationary stochastic process ?

9. Let {X(t), t ∈ R} be a second order stationary in the wide sense stochastic process. Define another process {Y (t), t ∈ R} as Y (t) = X(t+1)−X(t). Show that this process is also stationary in the wide sense.

10. Let Z_{1}, Z_{2} be independent identically distributed discrete random variables having the distribution P (Z_{i} = ±1) = 1/2. Show that X(t) = Z_{1} cos(λt)+Z_{2} sin(λt), t ∈ R, λ is a fixed real number, is a second order stationary in the wide sense stochastic process which is not strictly stationary.

11. Let ξ from Herglotz’s theorem be such that P (ξ = ±π) = 1/2. Calculate the correspondent covariance function and construct an example of stochastic process having such a spectral distribution.