This work was supported by the National Natural Science Foundation of China and the Swedish Research Council for Engineering Sciences (TFR).

Necessary and Sucient Conditions for Stability of LMS

Lei Guo y

Lennart Ljung z

and Guan-Jun Wang x

First Version: November 9, 1995 Revised Version: August 28, 1996

Abstract . In a recent work 7], some general results on exponential stabil- ity of random linear equations are established, which can be applied directly to the performance analysis of a wide class of adaptive algorithms including the basic LMS ones, without requiring stationarity, independency and boundedness assumptions of the system signals. The current paper attempts to give a complete characterization of the exponential stability of the LMS algorithms, by providing a necessary and sucient condition for such a stability in the case of possibly unbounded, nonstationary and non- -mixing signals. The results of this paper can be applied to a very large class of signals including those generated from, e.g., a Gaussian process via a time-varying linear lter. As an application, sev- eral novel and extended results on convergence and tracking performance of LMS are derived under various assumptions. Neither stationarity nor Markov chain assumptions are necessarily required in the paper.

This work was supported by the National Natural Science Foundation of China and the Swedish Research Council for Engineering Sciences (TFR).

y

Institute of Systems Science, Chinese Academy of Sciences, Beijing, 100080, P.R. China.

Email: Lguo@iss03.iss.ac.cn.

z

Department of Electrical Engineering, Linkoping University, S-581 83 Linkoping, Sweden.

Email: Ljung@isy.liu.se.

x

Department of Mathematics, The Central University for Nationalities, Beijing 100081, P. R. China.

1

1 Introduction

1.1 The Contribution

The well-known least mean squares (LMS) algorithm, aiming at tracking the

\best linear t" of an observed (or desired) signal

y k

based on a measured d -dimensional (input) signal

k

, is dened recursively by

x k+1 = x k +  k ( y k

k x k )  x 0

d  (1) where  > 0 is a step-size.

Due to its simplicity, robustness and ease of implementation, the LMS algo- rithm is known to be one of the most basic adaptive algorithms in many areas including adaptive signal processing, system identication and adaptive control, and it has received considerable attention in both theory and applications over the past several decades (see, among many others, the books 20], 19] and 2], the survey 14], and the references therein). Also, it has been found recently that the LMS is H

-optimal in the sense that it minimizes the energy gain from the disturbances to the predicted errors, and it is also risk sensitive optimal and minimizes a certain exponential cost function (see 11]).

In many situations, it is desirable to know at least the answers to the following questions:



Is the LMS stable in the mean squares sense?



Does the LMS have good tracking ability?



How to calculate and to minimize the tracking errors?

Now, for a given sequence

k

, (1) is a linear, time-varying dierence equa- tion. The properties of this equation are essentially determined by the homoge- neous equation:

x k+1 = ( I

 _{k k} ) x k (2)

with fundamental matrix

  ( tk ) =

^t

j=k ( I

 _{j j} ) (3)

2

The expression for tracking errors will then be of the form

t

X

k=1   ( tk ) v ( k ) (4)

where

v ( k )

describes the error sources (measurement noise, parameter varia- tions etc). As elaborated in, e.g., 8] and 6], the essential key to the analysis of (4) is to prove exponential stability of (3). This was also the motivation behind the work of 1]. We shall establish such exponential stability in the sense that for any p

1 there exist positive constants M ,  and 

such that

E

  ( tk )

^p ] ^1=p

M (1

 ) ^t

^k 

t

k



(0 

] : (5) The expectation E here is with respect to the sequence

k

.

Clearly, the property (5) is a property of the sequence

k

only. We shall here establish (5) under very general conditions on

k

. These are of the kind (precise conditions are given in Theorem 2):



Restrictions on the dependence among the k : This takes the form that k

is formed by possibly time varying, but uniformly stable ltering of a noise source " j which is mixing and obeys an additional condition on the rate of decay of dependence.



Restrictions on the tail of the distribution of k . This takes the form that Eexp( 

" k

2 )] < C 

k (6) for some  > 0 and some constant C . Here " k is the \source" from which

k was formed.

Both these restrictions are very mild, and allow for example the Gaussian, de- pendent case (unlike most previous treatments). Now, for sequences k subject to these two restrictions the necessary and sucient condition for (5) to hold is

that _k+h

X

i=k+1 E  _{i i} ]

I 

k

0  (7)

for some h > 0 and > 0. This is the \persistence of excitation" or \full rank"

condition on k .

This result is the main contribution of this paper. Furthermore, several direct applications of the stability result to adaptive tracking will be given under various noise assumptions, which in particular, yield more general results on LMS than those established recently in 8].

3

Most of the existing work related to exponential stability of (2) is concerned with the case where the signals

k

are independent or M -dependent (cf., e.g., 20], 19], 4], 1],2]). This independence assumption can be relaxed considerably if we assume that the signals

k

are bounded as in, e.g., 6],18] and 12].

Note that the boundedness assumption is suitable for the study of the so called normalized LMS algorithms (cf. 19], 6] and 15]), since the normalized signals are automatically bounded. In this case, some general results together with a very week (probably the weakest ever known) excitation condition for guaranteeing the exponential stability of LMS can be found in 6]. Moreover, in the bounded - mixing case, a complete characterization of the exponential stability can also be given. Indeed, in that case it has been shown in 6] that (7) is the necessary and sucient condition for (2) to be exponentially stable.

For general unbounded and correlated random signals, the stability analysis for the standard LMS algorithm (1), becomes more complex as to have deed complete solution for over 30 years. Recently, some general stability results ap- plicable to unbounded nonstationary dependent signals are established in 7], and based on which a number of results on the tracking performance of the LMS al- gorithms can be derived (see 8]). In particular, the result of 7] can be applied to a typical situation where the signal process is generated from a white noise sequence through a stable linear lter :

k =

j=

A j " k

j +  k 

j=

A j

<

 (8)

where

" k

is an independent sequence satisfying

sup _k E exp( 

" k

 )] <

 for some  > 0  > 2  (9) and

 k

is a bounded deterministic process.

It is obvious that the expression (8) has a similar form as the well-known Wold decomposition for wide-sense stationary processes. Note, however, that the signal process

k

dened by (8) need not be a stationary process nor a Markov chain in general.

Unfortunately, the condition (9) with > 2 excludes the case where

" k

is a

4

Gaussian process, since such signals could only satisfy a weaker condition : sup _k E exp( 

" k

2 )] <

 for some  > 0 : (10) The motivation of this paper has thus been to relax the moment condition (9) so that, at least, the signal process

k

dened by (8) and (10) can be included.

This will be done in a more general setting based on a relaxation of the moment condition used in Theorem 3.2 of 7].

2 The Main results

2.1 Notations

Here we adopt the following notations introduced in 7].

a) . The maximum eigenvalue of a matrix X is denoted by  max ( X ), and the Euclidean norm of X is dened as its maximum singular value, i.e.,

k

X

=

 max ( XX ^ )

 and the L p -norm of a random matrix X is dened as

k

X

p =

E (

X

^p )

 p

1 

b) . For any square random matrix sequence F =

F k

, and real numbers p

1 

(0  1), the L p -exponentially stable family

p is dened by

S

p ( 

) =

8

<

:

F :

^k

j=i+1 ( I

F j )

p

M (1

 ) ^k

ⁱ 

8



(0 

] 

k

i

0  for some M > 0and 

(0  1)

)

Likewise, the averaged exponentially stable family

is dened by

S

( 

) =

8

<

:

F :

^k

j=i+1 ( I

E  F j ])

M (1

 ) ^k

ⁱ 

8



(0 

] 

k

i

0  for some M > 0  and 

(0  1)

)

In what follows , it will be convenient to set

S

p =



(01)

S

p ( 

) 

=



(01)

S

( 

)  (11)

5

c) . Let p

1  F =

F i

. Set

M

p =

F : sup _i

S i ^(T)

p = o ( T )  as T

(12) where

S _i ^(T) = ^(i+1)T

¹

j=iT ( F j

E  F j ]) (13)

The denition of

p is reminiscent of the law of large numbers. As shown by Lemma 3 of 9], it includes a large class of random processes.

2.2 The Main Results

We rst present a preliminary theorem.

Theorem 1 ^{. Let}

^{F k}

be a random matrix process. Then

f

F k

=

F k

p 

p

1  provided that the following two conditions are satised:

(i). There exist positive constants "M and K such that for any n

1, E

"

exp

"

ⁿ

i=1

F j

!#



M exp( Kn ) holds for any integer sequence 0

j 1 < j 2 ::::: < j n .

(ii). There exists a constant M and a nondecreasing function g ( T ) with g ( T ) = o ( T ), as T

, such that for any xed T , all small  > 0 and any n

i

0,

E

8

<

:

exp

0

@



ⁿ

j=i+1

S ^(T) _j

1

A 9

=





M exp

 g ( T ) + o (  )]( n

i )

where S j ^(T) is dened by (13).

The proof is given in Section 4.

Remark 1 . The form of Theorem 1 is similar to that of Theorem 3.2 in 7].

The key dierence lies in the condition (i). This condition was introduced in 5], p.112 and is, in a certain sense, a relaxation of the corresponding condition used in Theorem 3.2 of 7]. Such a relaxation enables us to include Gaussian signals as a special case, when the LMS algorithms are in consideration, as will be shown shortly.

6

Based on Theorem 1 we may prove that for a large class of unbounded non- stationary signals including (8), the condition (7) is also necessary and sucient for the exponential stability of LMS.

Let us start with the following decomposition which is more general than that in (8):

k =

j=

A ( kj ) " k

j +  k 

j=

sup _k

A ( kj )

<

 (14) where

 k

is a d -dimensional bounded deterministic process, and

" k

is now a general m -dimensional -mixing sequence. The weighting matrices A ( kj )

R

d

m are assumed to be deterministic.

We remark that the summability condition in (14) is precisely the standard denition for uniform stability of time-varying linear lters (cf., e.g., 13]). Also, recall that a random sequence

" k

is called -mixing if there exists a non- increasing function ( m ) (called the mixing rate) with ( m )

0  1],

m

0 and ( m )

0 as m

such that

A

sup B

P ( B

A )

P ( B )

( m ) 

m

0 k

(



)

where by denition

i ^j ,

i

j

, is the -algebra generated by

" k i

k

j

.

The -mixing concept is a standard one in the literature for describing weakly dependent random processes. As is well-known, the -mixing property is satised by, for example, any M-dependent sequences, sequences generated from bounded white noises via a stable linear lter, and stationary aperiodic Markov chains which are Markov ergodic and satisfy Doeblin's condition (cf. 3]).

The main result of this paper is then stated as follows.

Theorem 2 . Consider the random linear equation (2). Let the signal process

f

k

be generated by (14) where

 k

is a bounded deterministic sequence, and

f

" k

is a -mixing process which satises for any n

1 and any integer sequence j 1 < j 2 ::::: < j n

E

"

exp



ⁿ

i=1

" j

2

!#



M exp( Kn ) (15) where M and K are positive constants. Then for any p

1, there exist constants 

> 0, M > 0 and 

(0  1), such that for all 

(0 

]

2

4

E

^t

j=k+1 ( I

 _{j j} )

^p

3

5

1=p



M (1

 ) ^t

^k 

t

k

0 (16)

7

if and only if there exists an integer h > 0 and a constant > 0 such that

k+h

i=k+1 E  _{i i} ]

I 

k

0 : (17)

The proof is also given in Section 4.

Remark 2 . By taking A ( k 0) = IA ( kj ) = 0 

k

j

= 0 and  k = 0 

k in (14), we see that

k

coincides with

" k

, which means that Theorem 2 is applicable to any -mixing sequences. Furthermore, if

" k

is bounded, then (15) is automatically satised. This shows that Theorem 2 may include the corresponding result in 6] as a special case.

Note, however, that a linearly ltered -mixing process like (14) will no longer be a -mixing sequence in general (because of the possible unboundedness of

f

" k

). In fact, Theorem 2 is applicable also to a quite large class of processes other than -mixing, as shown by the following corollary.

Corollary 1 . Let the signal process

k

be generated by (14) where

 k

is a bounded deterministic sequence, and

" k

is an independent sequence satisfying condition (10). Then

_{k k}

p for all p

1 if and only if there exists an integer h > 0 and a constant > 0 such that (17) holds.

Proof . By Theorem 2, we need only to show that condition (15) is true. This is obvious since

" k

is an independent sequence satisfying (10).

Remark 3 . Corollary 1 continues to hold if the independence assumption of

f

" k

is weakened to M-dependence. Moreover, the moment condition (10) used in Corollary 1 may also be further relaxed if additional conditions are imposed.

This is the case when, for example,

k

is a stationary process generated by a stable nite dimensional linear state space model with the innovation process

f

" k

being an i.i.d. sequence (see, 16]).

3 Performance of Adaptive tracking

Let us now assume that

y k

and

k

are related by a linear regression

y k = _k x

_k + v k (18)

where

x

_k

is the true or \ctitious" time-varying parameter process, and

v k

represents the disturbance or unmodeled dynamics.

8

The objective of the LMS algorithm (1) is then to track the time-varying un- known parameter process

x

_k

. The tracking error will depend on the parameter variation process

 k

dened by

 k = x

_k

x

_k

₁ (19)

through the following error equation obtained by substituting (18)-(19) into (1):

x ~ k+1 = ( I

 _{k k} )~ x k +  k v k

 k+1 (20) where ~ x k =

x k

x

_k .

Obviously, the quality of tracking will essentially depend on properties of

f

k   k v k

. The homogeneous part of (20) is exactly the equation (2), and can be dealt with by Theorem 2. Hence, we need only to consider the non- homogeneous terms in (20). Dierent assumptions on

 k v k

will give dierent tracking error bounds or expressions, and we shall treat three cases separately in the following.

3.1 First Performance Analysis

By this, we mean that the tracking performance analysis is carried out under a \worst case" situation, i.e., the parameter variations and the disturbances are only assumed to be bounded in an averaging sense. To be specic, let us make the following assumption:

A1) . There exists r > 2 such that

= sup

_k

v k

r <

and  = sup

_k

 k

r <

Note that this condition includes any \unknown but bounded" deterministic disturbances and parameter variations as a special case.

Theorem 3 . Consider the LMS algorithm (1) applied to (18). Let condition A1) be satised. Also, let

k

be as in Theorem 2 with (17) satised. Then for all t

1 and all small  > 0

E

x t

x

_t

² = O ( ² +  ²

 ^{2 ) +} O (1

 ] ^t ) where 

(0  1) is a constant.

9

This result follows immediately from Theorem 2, (20) and the Holder inequal- ity. We remark that various such \worst case" results for other commonly used algorithms(e.g., RLS and KF) may be found in 6]. The main implication of Theorem 3 is that the tracking error will be small if both the parameter variation (  ) and the disturbance ( ) are small.

3.2 Second Performance Analysis

By this, we mean that the tracking performance analysis is carried out for zero mean random parameter variations and disturbances which may be correlated processes in general. To be specic, we introduce the following set for r

1,

N

r =

8

<

:

w : sup _k

^k+n

i=k+1 w i

r

c _wr

n

n

1

9

=



(21)

where c _wr is a constant depending on r and the distribution of

w i

only.

Obviously,

r is a subset of

r dened by (12). It is known (see 9]) that martingale dierence, zero mean

and 

mixing sequences can all be included in

r . Also, from the proof of Lemma 3 in 9], it is known that the constant c _wr can be dominated by sup _k

w k

r in the rst two cases, and by sup _k

w k

r+  ( > 0), in the last case.

Moreover, it is interesting to note that

r is invariant under linear transfor- mations. This means that if

k

and

" k

are related by (8) with  k

0, then

f

" k

r implies that

k

r . This can be easily seen from the following inequality:

k

k+n

i=k+1 i

r =

j=

A j k+n

i=k+1 " i

j

r

 1

X

j=

A j

k+n

i=k+1 " i

j

r

Thus, random processes generated from martingale dierences, or

or 

mixing sequences via an innite order linear lter can all be included in

r .

Now, we are in a position to introduce the following condition for the second performance analysis.

A2) ^{. For some r > 2,}

^k

^{r and}

^{k v k}

^{r :}

Theorem 4 . Consider the LMS algorithm (1) applied to the model (18). Let

f

k

be dened as in Theorem 2 with (17) satised, and let the condition A2)

10

hold for a certain r . Then for all t

1 and all small  > 0, E

x t

x

t

2 = O

 ( c ^v r ) ² + ( c _r ) ²



!

+ O (1

 ] ^t )

where c ^v _r and c _r are the constants dened in (21), and which depend on the distributions of

k v k

and

 k

respectively. Moreover,  is the same constant as in Theorem 3.

Proof . By Lemma A.2 of 8] and Theorem 2, it is easy to see from (20) that

the desired result is true.

Note that the upper bound in Theorem 4 signicantly improves the \crude"

bound given in Theorem 3 for small  , and it roughly indicates the familiar trade-o between noise sensitivity and tracking ability.

Theorem 4 can be applied directly to the convergence analysis of some stan- dard ltering problems (cf. 20],4] and 2]). For example, let

y k

and

k

be two stationary processes, and assume that our purpose is to track the least mean squares solution

x

=  E ( _{k k} )]

¹ E ( k y k )

of min _x E ( y k

x  k ) ²

recursively based on real-time measurements

y i  i i

k

. Now, dene

v k

by

y k = _k x

+ v k

It is then obvious that E k v k = 0. Furthermore, in many standard situations it can be veried that

k v k

r for some r > 2. Thus, Theorem 4 applied to the above linear regression, gives

E

x t

x

² = O (  ) + O (1

 ] ^t )  which tends to zero as t

and 

0.

Apparently, Theorem 4 is also applicable to nonstationary signals

y k

and

f

k

.

3.3 Third Performance Analysis

By this, we mean that the analysis is purposed to get an explicit (approxi- mate) expression for the tracking performance rather than just getting an upper

11

bound as in the previous two cases. This is usually carried out under white noise assumptions on

 k v k

. Roughly speaking, the parameter process in this case will behave like a random walk, and some detailed interpretations of this param- eter model may be found in 14] and 8]. We make the following assumptions:

A3 . The regressor process is generated by a time-varying causal lter

k =

j=0 A ( kj ) " k

j +  k 

j=0 sup _k

A ( kj )

<

(22) where

 k

is a bounded deterministic sequence, and

" k   k v k

1

is a -mixing process with mixing rate denoted by ( m ). Assume also that (15) and (17) hold.

A4 . The process

 k v k

satises the following conditions:

( i ) : E  v k

k ] = 0  E  k+1

k ] = E  k+1 v k

k ] = 0 ( ii ) : E  v _2k

k ] = R v ( k )  E  k  _k ] = Q ( k )

( iii ) : sup _k E 

v k

r

jF

k ]

M  = sup

_k

 k

r <



where r > 2 and M > 0 are constants, and

k denotes the -algebra generated by

" i   i v i

1 i

k

.

Theorem 5 . Consider the LMS algorithm (1) applied to the model (18). Let conditions A3) and A4) be satised. Then the tracking error covariance matrix has the following expansion for all t

1 and all small  > 0

E ~ x t x ~ _t ] = ! t + O

(  )  +  ²

 ^{+ (1}

 ) ^t ]

!

where the function (  )

0 as 

0, and ! t is recursively dened by

! t+1 = ( I

S t )! t ( I

S t ) ^ +  ² R v ( t ) S t + Q ( t + 1) with S t = E  _{t t} ] and R v ( t ) and Q ( t ) being dened as in condition A4).

This theorem relaxes and unies the conditions used in Theorem 5.1 of 8].

The proof is given in Section 4. The expression for the function (  ) may be found from the proof, and from the related formula in Theorem 4.1 of 8]. (See (45)).

Note that in the (wide-sense) stationary case, S t

SR v ( t )

R v Q ( t )

Q , and ! t will converge to a matrix ! dened by the Lyapunov equation(cf.8])

S ! + ! S = R v S + Q



12

In this case, the trace of the matrix !, which represents the dominating part of the tracking error E

x ~ t

2 for small  and large t , can be expressed as

tr _{(!) = 12} R v d + tr ( S

¹ Q )

 ^]

where d =

dim ( k ). Minimizing tr (!) with respect to  , one obtain the following formula for the step-size  :

 =

s

tr ( S

¹ Q ) R v d :

4 Proof of Theorems 1, 2 and 5

Proof of Theorem 1 ^.

By the proof of Lemma 5.2 in 7] we know that Theorem 1 will be true if (32) in 7] can be established. However, by (34) in 7] and condition (ii), it is easy to see that we need only to show that for any xed c

1, t

1 and T > 1, and for all small  > 0,

k

n

Y

j=i+1 (1 +  ² c

H j

)

t

M

1 + O ( 

)

ⁿ

ⁱ 

n > i (23) where M > 0 is a constant and

 ² H j =  ² H j (2) +  ³ H j (3) +

+  ^T H j ( T ) + O (  ² )

with H j ( k ) =

jT

j

<j

<

<j

(j+1)T

1 F j

F j

 k = 2 

T:

Now, let us set

f j = exp



^(j+1)T

¹

s=jT

F s

Then for any 2

k

T and jT

j 1 < ::: < j k

( j + 1) T

1, by using the inequalities k

³ ₂ + ^k ₄ and x

exp ( x ), we have for 

(0  1)

 ^k

F j

:::F j





( 

F j

) ::: ( 

F j

)





exp



(

F j

+ ::: +

F j

)





f j

13

Consequently,

(1 +  ² c

H j

)



T

Y

k=2 (1 +  ^k c

H j ( k )

)(1 + O (  ² ))



T

Y

k=2

Y

iT

j

<j

<j

(i+1)T

1 (1 +  ^k c

F j

F j

)(1 + O (  ² ))



(1 + 

cf j ) ²

(1 + O (  ² )) (24) Note that

n

Y

j=i+1 (1 + 

cf j ) = ⁿ

ⁱ

k=0 ( 

c ) ^k

i+1

j

<:::<j

n f j

:::f j

Now, applying the Minkowski inequality to the above identity, noting the disjoint property of the sets

j i T < j < ( j i + 1) T

1

, i = 1  2 ::: , for j 1 < j 2 <

::: , taking  small enough so that 2 ^T t

" and using Condition (i) it is evident that

k

n

Y

j=i+1 (1 + 

cf j )

2

t



n

i

X

k=0 ( 

c ) ^k

i+1

j

<:::<j

n M

exp

( KT 2 ^T t ⁾ k



M

1 + c

exp( KT 2 ^T t ⁾



n

i

Finally, from this and (24), we have for any n > i

k

n

Y

j=i+1 (1 +  ² c

H j

)

t















n

Y

j=i+1 (1 + 

cf j )













2

2

t 1 + O (  ² )] ⁿ

ⁱ



M

8

<

:

1 + c

exp( KT 2 ^T t ⁾

2



n

i

1 + O (  ² )] ⁿ

ⁱ



M 1 + O ( 

)] ⁿ

ⁱ  for all small  > 0  which is (23). This completes the proof of Theorem 1.

2

14

The proof of Theorem 2 is rather involved, and so it is divided (prefaced) with several lemmas.

For the analysis to follow, it is convenient to rewrite (14) as

k =

j=

a j " ( kj ) +  k 

j=

a j <

 (25)

where by denition

a j = sup

_k

A ( kj )

 " ( kj ) =

a

_j ¹ A ( kj ) " k

j (26) (We set " ( kj ) = 0 

k , if a j = 0 for some j ).

The new process

" ( kj )

has the following simple properties:

(i). For any k and j ,

" ( kj )

" k

j

(ii). For any xed j , the process

" ( kj )

is -mixing with the same mixing rate as

" k

(iii). For any k and j , " ( kj ) is

" k

j

-measurable.

These three properties will be frequently used in the sequel without further explanations.

Lemma 1 ^{. Let}

^F t

be a -mixing d

d dimensional matrix process with mixing rate

( m )

. Then

sup _i

S _i ^(T)

2

2 cd

(

T ^T

¹

m=0

q

( m )

) 1

2



T

1  where S _i ^(T) is dened by (13) and c is dened by c = sup

_i

F i

EF i

2 .

Proof ^{. Denote} G k = F k

EF k . Then by Theorem A.6 in 10](p.278) we have

k

E  G j G _k ]

2 dc ²

(

j

k

) 

jk Consequently, by using the inequality

j

trF

d

F



F

^d

^d We get

k

S i ^(T)

2 2 = E

^(i+1)T

¹

jk=iT G j G _k

15



tr

^(i+1)T

¹

jk=iT EG j G _k



d ^(i+1)T

¹

jk=iT

k

EG j G _k



2 c ² d ^{2 (i+1)T}

¹

jk=iT

q

(

j

k

)



4 c ² d ² T ^T

¹

m=0

q

( m )

This gives the desired result.

Lemma 2 ^{. Let F} k = _{k k} , where

k

is dened by (14) with sup _k

" k

4 <

. Then

F k

2 where

2 is dened by (12).

Proof . First of all, we may assume that the process

" k

is of zero mean (otherwise, the mean can be included in  k ). Then by (25),

k

S _i ^(T)

2 =

^(i+1)T

¹

t=iT  _{t t}

E _{t t} ]

2

 1

X

kj=

a k a j

(i+1)T

1

X

t=iT  " ( tk ) " ( tj ) ^

E" ( tk ) " ( tj ) ^ ]

2

+2

j=

a j

(i+1)T

1

t=iT " ( tj )  _t

2 (27)

Note that for any xed k and j , both the processes

" ( tk ) " ( tj ) ^

and

f

" ( tj )

are -mixing with mixing rate ( m

k

j

) and ( m ) respectively (where by denition, ( m ) = 1



m < 0) .

By Lemma 1, it is easy to see that the last term in (27) is of order o ( T ). For dealing with the second last term, we denote

f kj ( T ) = 2 cd

(

T ^T

¹

m=0

q

( m

k

j

)

) 1

2

: (28)

where c is dened as in Lemma 1. Consequently, by ( m )

1,

m , it is not dicult to see that

sup _kj f kj ( T )

2 cdT (29)

and sup

j

k

j

<

T f kj ( T ) = o ( T ) : (30)

16

Now, by the summability of

a j

,

X

j

k

j

T a k a j

0  as T

Hence by (29)

X

j

k

j

T a k a j f kj ( T ) = o ( T ) (31) and by (30)

X

j

k

j

<

T a k a j f kj ( T ) = o ( T ) : (32) Combining (31) and (32) gives

1

X

kj=

a k a j f kj ( T ) = o ( T ) : (33) By this and Lemma 1, we know that the second last term in (27) is also of the order o ( T ) uniformly in i . Hence,

F k

2 by the denition (12).

Lemma 3 ^{. Let sup} _k E

k

2 <

. Then

_{k k}

if and only if condition (17) holds, where

is dened in (11).

Proof. Let us rst assume that (17) is true. Take 

= (1 + sup _k E

k

2 )

¹ . Then applying Theorem 2.1 in 6] to the deterministic sequence A k = E  _{k k} ] for any 

(0 

], it is easy to see that

_{k k}

( 

).

Conversely, if

_{k k}

, then there exists 

(0  (1+sup _k E

k

2 )

¹ ] such that

_{k k}

( 

). Now, applying Theorem 2.2 in 6] to the deterministic sequence A k = 

E  _{k k} ], it is easy to see that (17) holds. This completes the

proof.

Lemma 4 ^{. Let F} k = _{k k} , where

k

is dened by (14) with (15) satised.

Then

F k

satises Condition (i) of Theorem 1.

Proof . Without loss of generality assume that  k

0. Let us denote A = ^

j=

a j (34)

where

a j

is dened by (26). Then by the Schwarz inequality from (25) we have

k

k

2



A

j=

a j

" k

j

2

17

Consequently, by the Holder inequality and (15) we have for "

A

² E exp

"

ⁿ

i=1

F j



E exp

"A

j=

a j

n i=1

k

" j

j

2

g

= E

j=

exp

"Aa j

n i=1

k

" j

j

2

g

 1

Y

j=

E exp

"A ²

ⁿ

i=1

k

" j

j

2

g

!ajA

 1

Y

j=

( M exp

Kn

)

= M exp

Kn

:

This completes the proof.

The following lemma was originally proved in 5] (p.113).

Lemma 5 . Let

z k

be a nonnegative random sequence such that for some a > 0 b > 0 and for all i 1 < i 2 < :::::: < i n 

n

1,

E exp

ⁿ

k=1 z i

exp

an + b

: (35)

Then for any L > 0 and any n

i

0, E exp

1

2

n

X

j=i+1 z j I ( z j

L )

exp

e ^a

( n

i ) + b

where I (

) is the indicator function.

Proof ^{. Denote}

f j = exp(12 z j ) I ( z j

L ) :

Then by rst applying the simple inequality I ( x

L )

e

=e

and then using (35), we have for any subsequence j 1 < j 2 :::::: < j k

E  f j

::::::f j

]

= E _exp(12

^k

i=1 z j

)

^k

i=1 I ( z j

L )



E exp(

^k

i=1 z j

) = exp( kL 2 )



exp

( a

L

2 ) k + b

18

By this we have

E exp

ⁿ

j=i+1

1 2 z j I ( z j

L )

= E

ⁿ

j=i+1 exp

1

2 z j I ( z j

L )



E

ⁿ

j=i+1

1 + exp(12 z j ) I ( z j

L )

= E

ⁿ

j=i+1

1 + f j

= E

8

<

:

n

i

X

k=0

X

i+1

j

<:::<j

n f j

:::f j

9

=





e ^b

8

<

:

n

i

X

k=0

X

i+1

j

<:::<j

n exp

( a

L 2 ) k

9

=



= e ^{b n}

j=i+1

f

1 + exp( a

L 2 )



exp

( n

i )exp( a

L 2 ) + b

This completes the proof of Lemma 5.

Lemma 6 ^{. Let F} k = _{k k} , where

k

is dened by (14) with (15) satised.

Then

F k

satises Condition (ii) of Theorem 1.

Proof . Set for any xed k and l , z j =

z j ( kl ) =

^(j+1)T

¹

t=jT  " ( tk ) " ( tl ) ^

E" ( tk ) " ( tl ) ^ ]

Then, similar to (27) from (25) we have

n

X

j=i+1

k

S _j ^(T)

kl=

a k a l

n j=i+1 z j + +2

k=

a k

n j=i+1

(j+1)T

1

t=jT " ( tk )  _t

: (36) We rst consider the second last term in (36). By the Holder inequality,

E exp

8

<

:



kl=

a k a l

n j=i+1 z j

9

=



19