Experimental Evaluation of some Thresholding Methods for Estimating Time-Delays in Open-Loop

Experimental Evaluation of some Thresholding

Methods for Estimating Time-Delays in

Open-Loop

Svante Bj¨orklund Control & Communication Department of Electrical Engineering

Link¨opings universitet, SE-581 83 Link¨oping, Sweden WWW: http://www.control.isy.liu.se

E-mail: svabj@isy.liu.se 14th July 2003

AUTOMATIC CONTROL

COMM_{UNICATION SYSTE}MS LINKÖPING

Sammanfattning Abstract Rapporttyp Report: category Licentiatavhandling C-uppsats D-uppsats Övrig rapport Språk Language Svenska/Swedish Engelska/English ISBN

Serietitel och serienummer

Title of series, numbering

URL för elektronisk version

Titel Title Författare Author Datum Date Avdelning, Institution Division, department

Automatic Control

X

LiTH-ISY-R-1400-3902

Department of Electrical Engineering

2525

Experimental Evaluation of some Thresholding Methods for Estimating Time-Delays

in Open-Loop

In this report we study estimation of time-delays in linear dynamical systems with additive

noise. Estimating time-delays is a common engineering problem, e.g. in automatic control,

sys-tem identification and signal processing.

The purpose with this work is to test and evaluate a certain class of methods for time-delay

esti-mation, especially with automatic control applications in mind. Particularly interesting it is to

determine the best method. Is one method best in all situations or should different methods be

used for different situations? The tested class of methods consists essentially of thresholding the

cross correlation between the output and input signals. This is a very common method for

time-delay estimation. The methods are tested and evaluated experimentally with the aid of

simula-tions and plots of RMS error, bias and confidence intervals.

The results are: The methods often miss to detect because the threshold is too high. The

thresh-old has nevertheless been selected to give the best result. All methods over-estimate the

time-de-lay. Nearly the whole RMS error is due to the bias. None of the tested methods is always best.

Which method is best depends on the system and what is done when missing detections. Some

form of averaging of the cross correlation, e.g. integration to step response or CUSUM, is

advan-tageous. Fast systems are easiest. White noise input signal is easiest and steps is hardest. The

RMS-errors are high in average (approximately greater than 6 sampling intervals). The error is

lower for fast system or for high SNR.

X

2003-07-15

Contents

5 Discussion and conclusions 16 5.1 Discussion . . . 16

5.2 Conclusions . . . 16

5.3 Recommendations . . . 16

5.4 Future work . . . 17

References 1

1

Introduction

The problem we address in this report is estimating time-delays in linear dynamical sys-tems with additive noise. A synonym for time delay is dead-time. Estimating time-delays is a common engineering problem, e.g. in control performance monitoring of industrial processes [Hor00, Swa99], in design and tuning of controllers, in range estimation in radar [KQ92] and in direction estimation by time-delay of arrival in signal intelligence [HR97, Wik02]. Dead-time estimation is also a necessary part in all system identification [Lju99].

The purpose with this work is to test and evaluate a certain class of methods for time-delay estimation, especially with automatic control applications in mind. Particularly interesting is it to determine the best method. Is one method best in all situations or should different methods be used for different situations? The tested class of methods consists essentially of thresholding the cross correlation between the output and input signals. This is a very common method for time-delay estimation. The methods are tested and evaluated experimentally with the aid of simulations and plots of RMS error, bias and confidence intervals.

In the next chapter, the thresholding methods are briefly described. Then, Chapter 3 is about the simulation setup. After that, in Chapter 4 the analysis of the simulations is conducted. Following, Chapter 5 contains discussion, conclusions and suggestions for further work. After a literature reference list, Appendix A contains validation of required prerequisites for the analysis.

2

Thresholding methods

Thresholding methods is a subgroup of cross correlation methods. The steps of the methods are:

1. Estimate the impulse response and estimate the uncertainty of the impulse response estimate. The estimated impulse response is in principle the cross correlation be-tween the output and input signals of the system [Bj¨o03].

2. Optionally, integrate to step response. 3. Thresholding.

If the number zero is outside a certain confidence interval, then we consider the impulse (step) response to have started and this point of time is the time-delay estimate. The thresholding can be either

• Direct thresholding [Bj¨o03] or.

• Cumulative sum (CUSUM) thresholding [Bj¨o03]. For information about CUSUM see also [Gus00, GLM01].

There are some method parameters to choose. The most important are

• The relative threshold hstd and relative drift νstd [Bj¨o03].

The methods that we have used in this report are:

• idimp5. Direct thresholding of impulse response with prewhitening and hstd = 5.

This would give a confidence interval with a confidence level of 0.999999713 if the impulse response estimates are Gaussian distributed (which is a good assumption, see [Lju99]) and the estimate of the uncertainty in the confidence interval estimate is accurate.

• idstep5. Direct thresholding of step response without prewhitening and hstd= 5.

• idimpCusum4. CUSUM thresholding of impulse response with prewhitening, hstd =

3, νstd= 1 .

• idstepCusum4. CUSUM thresholding of step response with prewhitening, hstd = 1,

νstd= 6 .

The choices of hstd and νstdare the result of a statistical analysis in [Bj¨o03]. Note the

opposite relation between hstd and νstdfor idimpCusum4 and idstepCusum4. For more on

the used methods in this report see [Bj¨o03].

In signal processing applications the system often consists of a pure time delay, maybe with an amplitude change. Then the right thing to do is to estimate the peak of the impulse response (cross correlation) instead.

3

Simulation setup

The setup for the simulations is the same as in [Bj¨o03] but with different time-delay estimation methods. Since the estimates are very non-Gaussian, so many as 4096 trials were simulated. From this 4 estimates of RMS error was calculated for the plots with confidence intervals. In this report we are studying properties in average.

There were many missed detections because the threshold was often not crossed. This complicates the analysis. It is not obvious what to do when missing. One possibility is to assign a value to the time-delay estimate. If this value is very incorrect, then we give a penalty because of the miss. Another possibility is to give an alarm. A third is to find the highest peak of the impulse response. This latter possibility will give a bias in the estimate. It, unfortunately, turns out that which method is the best depends on what we do when the detection is missed.

In this report, when a method missed to detect a uniform distributed random number in the range 20 to 30 was delivered as the time-delay estimate (as in Section 7.7-7.9 in [Bj¨o03]). The reason for this special choice of was to come closer to the required prerequisites for the confidence interval calculations. See appendix A and reference [Bj¨o03] for more on this subject.

Three environment factors were varied during the simulations: The system, the input signal type and the SNR [Bj¨o03]. The SNR was either 1 or 100. See [Bj¨o03] for the definition of the SNR. The impulse responses of the four used systems are depicted in Figure 3.1. Note that for all the systems the time delay will be 10 after the sampling. More information about the systems can be found in [Bj¨o03].

Figures 3.2-3.4 show the used input signals in the time and frequency domains. More information about the input signals can be found in [Bj¨o03].

Time (sec.) Amplitude Impulse response G1 (t130g) 0 14 28 42 56 70 0 0.02 0.04 0.06 0.08 From: U(1) To: Y(1) Time (sec.) Amplitude Impulse response G2 (t130g) 0 14 28 42 56 70 0 0.2 0.4 0.6 0.8 From: U(1) To: Y(1) Time (sec.) Amplitude Impulse response G5 (t130g) 0 14 28 42 56 70 0 0.02 0.04 0.06 0.08 From: U(1) To: Y(1) Time (sec.) Amplitude Impulse response G6 (t130g) 0 14 28 42 56 70 −0.01 0 0.01 0.02 0.03 0.04 0.05 From: U(1) To: Y(1)

Figure 3.1: Impulse response of system G1-G2 and G5-G6. True time-delay after sampling

Td= 10. 0 50 100 150 200 250 300 350 400 450 500 −1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1 samples Time signal RBS10−30% 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 10−2 10−1 100 101 Normalized frequency Power Power spectruml RBS10−30%

Figure 3.2: Time signal (left) and frequency spectrum (right) for a realization of the input signal type RBS 10-30%.

0 50 100 150 200 250 300 350 400 450 500 −1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1 samples Time signal RBS0−100% 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 10−1 100 101 Normalized frequency Power Power spectruml RBS0−100%

Figure 3.3: Time signal (left) and frequency spectrum (right) for a realization of the input signal type RBS 0-100%. 0 50 100 150 200 250 300 350 400 450 500 −1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1 samples Time signal steps

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 10−4 10−3 10−2 10−1 100 101 102 Normalized frequency Power

Power spectruml steps

Figure 3.4: Time signal (left) and frequency spectrum (right) for a realization of the input signal type Steps.

4

Analysis

In this chapter we compare the tested methods and see in which cases they could be used. In Section 4.1 we start by a simple analysis that can give some feeling about the problems and how the methods work. Section 4.2 then has a statistical analysis, from which it is possible to draw conclusions.

4.1 Simple analysis

Figure 4.1 displays estimated impulse response for one of the systems but for different input signals and different SNRs. We see that in some cases the estimate is very inaccurate. This makes the time-delay estimation a hard job in these cases.

0 10 20 30 40 50 60 70 −0.02 0 0.02 0.04 0.06 0.08 Time [s] Impulse response ,10−30%, SNR=100, G1 0 10 20 30 40 50 60 70 −0.1 −0.05 0 0.05 0.1 0.15 Time [s] Impulse response ,10−30%, SNR=1, G1 0 10 20 30 40 50 60 70 −0.02 0 0.02 0.04 0.06 0.08 Time [s] Impulse response ,0−100%, SNR=100, G1 0 10 20 30 40 50 60 70 −0.05 0 0.05 0.1 0.15 Time [s] Impulse response ,0−100%, SNR=1, G1 0 10 20 30 40 50 60 70 −0.2 −0.1 0 0.1 0.2 0.3 Time [s]

Impulse response ,steps, SNR=100, G1

0 10 20 30 40 50 60 70 −2 −1 0 1 2 Time [s]

Impulse response ,steps, SNR=1, G1

Figure 4.1: Impulse response estimate of the system G1 by the function idimp4 for different

input signal types and different SNRs [Bj¨o03]. The solid line is the true impulse response. The circles are the estimated impulse response and the triangles mark ±two estimated standard deviations. Note the different ranges of the vertical axes. (t130f1.m)

Direct thresholding of the impulse response estimate is illustrated in Figure 4.2 and CUSUM thresholding in Figure 4.3. Note that in these figures, the relative threshold hstd

and relative drift νstdwere not the same as in Chapter 2 and in the statistical analysis in

0 10 20 30 40 50 60 70 −0.08 −0.06 −0.04 −0.02 0 0.02 0.04 0.06 0.08 0.1 0.12 Time [s] Impulse response 0 10 20 30 40 50 60 70 0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 Threshold test Time [samples]

Figure 4.2: Left: Estimated impulse response with uncertainty. Right: Estimated impulse response and threshold. Simulated input signal of type RBS 10-30%. SNR was 1. System G1.

The estimated time delay with idimp4 (hstd= 3) was ˆTd= 11. (t134d1.m)

0 10 20 30 40 50 60 70 −0.06 −0.04 −0.02 0 0.02 0.04 0.06 0.08 0.1 0.12 Time [s] Impulse response G1 0 10 20 30 40 50 60 70 0 0.01 0.02 0.03 0.04 0.05 0.06 0.07

CUSUM test statistics G₁

Time [samples]

Test statistics threshold drift

Figure 4.3: Left: Impulse response with uncertainty. Right: Test statistics g(t), threshold h and drift ν for CUSUM on impulse response. Simulated input signal of type RBS 10-30%. SNR was 1. System G1. The estimated time delay with idimpCusum3 (hstd= 2 and νstd= 1)

4.2 Statistical analysis

In this section we will perform a statistical analysis. Bear in mind that:

• All results are in average. Certain special cases can give a different result.

• Even if there is a statistically significant difference, it is not sure that the difference has any practical importance.

The statistical analysis is conducted in the same way as in Chapter 7 in [Bj¨o03]. The transformation was (RMS error)ˆ(0.73028) (See [Bj¨o03]). This means “The lower the better”. Figure 4.4 shows confidence intervals for pair-wise comparisons of methods. We see that:

• Step response is significantly better (not overlapping confidence intervals) than im-pulse response in average.

• Step response: no significant difference (overlapping confidence intervals) between direct and CUSUM thresholding.

• Impulse response: significant difference between direct and CUSUM thresholding and CUSUM is better.

3.5 4 4.5 4 3 2 1 t149b5:Method

3 groups have population marginal means significantly different from Group 1 1:idimp5 2:idstep5 3:idimpCusum4 4:idstepCusum4

Figure 4.4: Confidence intervals (the lines in the circles) for pair-wise comparisons (95% simultaneous confidence level) for different thresholding methods. Positive transformation: (RMS error)ˆ(0.730028) => ”The lower the better”. (t149b1.m)

Figure 4.5 shows the average RMS error and bias for the different methods. Wee see that the RMS error ' 6 in average. It is high because:

• We are punishing missed detections. • The case with SNR=1 is difficult difficult.

• The methods are overestimating the time delay. Nearly the whole RMS error is due to the bias. − idimp5 idstep5 idimpCusum4 idstepCusum4 0 2 4 6 8 10 8.67 . MAX 6.06 . MIN Method t149b5:030505 17:30 rms: rms, data(:,m,m,m,:,m,m,m,m) 7.36 6.32 − idimp5 idstep5 idimpCusum4 idstepCusum4 0 2 4 6 8 10 8.39 . MAX 4.47 . MIN Method t149b5:030505 17:30 bias: bias, data(:,m,m,m,:,m,m,m,m)

6.69

5.42

Figure 4.5: Average RMS error (left) and bias (right) of time-delay estimates for different thresholding methods.

In Figure 4.6 it is obvious there is a large difference in performance between low and high SNR. Using step response estimates, the mean RMS error is 2.0 for high SNR but 10.4 for low SNR. Nearly the whole RMS error is due to the bias.

In Figure 4.7 wee see how good the methods are for different input signal types. We notice:

• No method is always significantly best but step response is often better than impulse response.

• Wideband random signal easiest for all methods. • Steps signal hardest for all methods.

Figure 4.8 shows the average RMS error and bias for different methods and input signal types. Wee see that:

idimp5 idstep5 idimpCusum4 idstepCusum4 100 1 0 2 4 6 8 10 12 14 4.46 Method 2.18 12.9 . MAX 2.9 t149b5:030505 17:30 rms: rms, data(:,m,m,:,m,m,m,m,m) 9.94 1.78 . MIN 11.8 SNR 10.9 idimp5 idstep5 idimpCusum4 idstepCusum4 100 1 0 2 4 6 8 10 12 14 4.27 Method 1.1 . MIN 12.5 . MAX 2.46

t149b5:030505 17:30 bias: bias, data(:,m,m,:,m,m,m,m,m)

7.84

1.51 10.9

SNR 9.33

Figure 4.6: Average RMS error (left) and bias (right) of time-delay estimates for different thresholding methods and SNRs.

2 2.5 3 3.5 4 4.5 5 5.5 6 6.5 12:idstepCusum4*steps 11:idimpCusum4*steps 10:idstep5*steps 9:idimp5*steps 8:idstepCusum4*RBS0−100% 7:idimpCusum4*RBS0−100% 6:idstep5*RBS0−100% 5:idimp5*RBS0−100% 4:idstepCusum4*RBS10−30% 3:idimpCusum4*RBS10−30% 2:idstep5*RBS10−30% 1:idimp5*RBS10−30% t149b5:Method*InType

11 groups have population marginal means significantly different from Group 1

Figure 4.7: Confidence intervals (the lines in the circles) for pair-wise comparisons (95% simultaneous confidence level) for different thresholding methods and input signals. Positive transformation: (RMS error)ˆ(0.730028) => ”The lower the better”. (t149b1.m)

idimp5 idstep5 idimpCusum4 idstepCusum4 RBS10−30% RBS0−100% steps 0 5 10 15 5.87 InType 6.06 3.77 5.02 4.8 9.32 t149b5:030505 17:30 rms: rms, data(:,:,m,m,m,m,m,m,m) 6.55 3.39 . MIN 11.2 6.02 Method 9.76 13.4 . MAX idimp5 idstep5 idimpCusum4 idstepCusum4 RBS10−30% RBS0−100% steps 0 5 10 15 4.31 InType 5.63 2.81 3.15 3.73 9.12

t149b5:030505 17:30 bias: bias, data(:,:,m,m,m,m,m,m,m)

6.2 0.887 . MIN 10.7 5.85 Method 9.37 13.1 . MAX

Figure 4.8: Average RMS error (left) and bias (right) of time-delay estimates for different thresholding methods and input signals.

• Impulse response significantly best for the fast system. • Step response significantly best for the other systems. • The fast system is the easiest for all methods.

Figure 4.10 shows the average RMS error and bias for different methods and systems. We see that:

• Fast system: RMS error ≈ 3.5 (step response), 2.9 (impulse response).

• High order system with complex poles: RMS error ≈ 7.8 (step response), 10.5 (im-pulse response).

Figures 4.11 and 4.12 depict the RMS error and bias of the methods for all environment factors. We observe that:

• The bias is most often positive . It is often large. The few cases with negative bias has a very small bias. This indicate that we need a better estimation of the change time point in the impulse and step response.

1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 16:idstepCusum4*cplx4 15:idimpCusum4*cplx4 14:idstep5*cplx4 13:idimp5*cplx4 12:idstepCusum4*real4 11:idimpCusum4*real4 10:idstep5*real4 9:idimp5*real4 8:idstepCusum4*fast2 7:idimpCusum4*fast2 6:idstep5*fast2 5:idimp5*fast2 4:idstepCusum4*slow2 3:idimpCusum4*slow2 2:idstep5*slow2 1:idimp5*slow2 t149b5:Method*Sys

14 groups have population marginal means significantly different from Group 1

Figure 4.9: Confidence intervals (the lines in the circles) for pair-wise comparisons (95% simultaneous confidence level) for different thresholding methods and systems. Positive trans-formation: (RMS error)ˆ(0.730028) => ”The lower the better”. (t149b1.m)

idimp5 idstep5 idimpCusum4 idstepCusum4 slow2 fast2 real4 cplx4 0 5 10 15 6.56 7.64 3.27 Sys 6.28 2.82 . MIN 7.35 10.4 3.67 t149b5:030505 17:30 rms: rms, data(:,m,m,m,m,m,m,m,:) 8.79 8.11 2.88 6.71 10.2 10.6 Method 7.57 10.8 . MAX idimp5 idstep5 idimpCusum4 idstepCusum4 slow2 fast2 real4 cplx4 0 5 10 15 5.55 6.75 2.59 Sys 4.55 2.44 6.33 10.1 2.29 . MIN

t149b5:030505 17:30 bias: bias, data(:,m,m,m,m,m,m,m,:)

7.97 7.2 2.52 5.07 9.59 10.3 Method 5.97 10.6 . MAX

Figure 4.10: Average RMS error (left) and bias (right) of time-delay estimates for different thresholding methods and systems.

idimp5*100 idimp5*1 idstep5*100 idstep5*1 idimpCusum4*100 idimpCusum4*1 idstepCusum4*100 idstepCusum4*1 slow2*RBS10−30% slow2*RBS0−100% slow2*steps fast2*RBS10−30% fast2*RBS0−100% fast2*steps real4*RBS10−30% real4*RBS0−100% real4*steps cplx4*RBS10−30% cplx4*RBS0−100% cplx4*steps 0 10 20 12.2 6.9 0.906 15.3 0.88 14.5 2.49 3.09 9.62 0.573 0.636 0.323 15.3 0.595 10.6 15.5 0.637 0.268 5.19 4.56 1.13 Sys*InType 13.7 0 0.465 0.23 16.3 1.96 15.3 8.46 1.29 15.2 0.541 1.17 3.16 15.3 0.829 15.4 1.13 15 0.241 1.94 0.79 15.3 0.15 14.2 4.15 11.6 0.981 16.2 1.93 1.42 15.3 t149b5:030505 17:30 rms: rms, data(:,:,m,:,m,m,m,m,:) 10 1.81 15.4 0.839 11.1 0 0.109 0. MIN 15.4 1.48 15.6 9.02 5.65 1.32 15.3 0 5.66 13.7 1.37 16 2.04 15.3 0.458 15.4 1.09 12.4 4.19 15.2 1.45 14.1 6.89 1.72 15.3 0.923 16.5. MAX 2.14 15.4 15.3 Method*SNR 5.74 15.3 2.68 15.3 1.22 15.2

Figure 4.11: Average RMS error of time-delay estimates for different thresholding methods and different environment factors.

idimp5*100 idimp5*1 idstep5*100 idstep5*1 idimpCusum4*100 idimpCusum4*1 idstepCusum4*100 idstepCusum4*1 slow2*RBS10−30% slow2*RBS0−100% slow2*steps fast2*RBS10−30% fast2*RBS0−100% fast2*steps real4*RBS10−30% real4*RBS0−100% real4*steps cplx4*RBS10−30% cplx4*RBS0−100% cplx4*steps −200 20 8.86 5.35 0.661 15 0.377 13.6 0.541 3.04 7.18 0.3 −0.0186 −0.0403 15 −0.0522 6.68 15.1 −0.104 0.000977 4.53 2.22 0.309 Sys*InType 10.4 0 −0.0361 −0.00708 15.5 −0.585 15.1 6.69 1.05 14.7 −0.0471 −0.0388 3.11 15 0.687 15 0.735 14.2 −0.00732 −0.771. MIN −0.197 15.1 0.0225 11.5 4.11 9.39 0.89 15.5 −0.753 0.134 15

t149b5:030505 17:30 bias: bias, data(:,:,m,:,m,m,m,m,:)

8.37 1.55 15 0.38 7.58 0 −0.00171 0 15.1 1.11 14.9 7.97 2.98 0.644 15 0 5.61 12.2 1.26 15.3 −0.255 15 0.0117 15 0.837 9.12 4.14 14.9 1.24 13.5 4.28 1.07 15 0.802 15.7. MAX −0.0266 15 15 Method*SNR 5.69 15 2.42 15 1.14 15

Figure 4.12: Average bias of time-delay estimates for different thresholding methods and different environment factors.

5

Discussion and conclusions

5.1 Discussion

There is a very high confidence level for the direct thresholding. This implies that we over-estimate the time delay. All methods over-over-estimate the time-delay. The used thresholds have nevertheless been selected to give the best result [Bj¨o03]. A better estimation of the change time than simple thresholding is needed. In [KG81] a more sophisticated estimation method of the change time is used. This method has not been tested in this report.

5.2 Conclusions

We draw the following conclusions from the work in this report:

• The methods often miss to detect. This is a problem in the analysis.

• No method is always best. Which is best depends on the system and what is done when missing detections.

• In the tested cases (the used penalty when missing), using step response was best in average. It was also best in most combinations of factor environments.

• In the tested cases, using impulse response was best in average for fast systems. • No significant difference between direct and CUSUM thresholding for step response. • CUSUM significantly better than direct thresholding for impulse response except for

fast systems.

• Some form of averaging, e.g. integration to step response or CUSUM, of the impulse response enhances the time-delay estimate (except for fast systems). Integrating to step response and CUSUM are two ways to smooth the impulse response estimate. Using CUSUM on the step response estimate does not further improve the estimates in most cases.

• The wideband random input signal is easiest. Input signal with steps is hardest. • Fast system is easiest.

• All methods over-estimate the time-delay. A better estimation of the change time is needed.

• Nearly the whole RMS error is due to the bias.

• The RMS-errors are high in average (& 6 sampling intervals). This is too high to useful. The error is lower for fast system or for high SNR.

5.3 Recommendations

• Use thresholding methods only for fast systems or when the SNR is high. • For step input the SNR must be high.

• Integration to step response and/or CUSUM thresholding enhances the performance for the impulse response estimate, except for fast systems..

5.4 Future work

Some possible future work is:

• Estimate better the change point in the impulse and step responses to avoid over-estimating the time-delay.

• Are there really no better thresholds and drifts?

• Test logarithmic cross spectrum scale (cepstrum), see [Bj¨o03].

• Test the thresholding methods on random systems and make a statistical analysis. • Test other methods in the same way as in this report.

References

[Bj¨o03] Svante Bj¨orklund. Experimental evaluation of some cross correlation meth-ods for time-delay estimation in linear systems. Technical Report LiTH-ISY-R-2513, Department of Electrical Engineering, Link¨oping University, SE-581 83 Link¨oping, Sweden, April 2003.

[GLM01] F. Gustafsson, L. Ljung, and M. Millnert. Signalbehandling. Studentlitteratur, Lund, Sweden, 2001. In Swedish.

[Gus00] F. Gustafsson. Adaptive Filtering and Change Detection. Wiley, 2000. ISBN 0-471-49287-6.

[Hor00] A. Horch. Condition Monitoring of Control Loops. Phd thesis TRITA-S3-REG-0002, Department of Signals, Sensors and Systems, Royal Institute of Technol-ogy, Stockholm, Sweden, 2000.

[HR97] A. W. Houghton and C. D. Reeve. Direction finding on spread spectrum signals using the time-domain filtered cross spectral density. IEE Proceedings - Radar, Sonar and Navigation, 144(6):315–320, December 1997.

[KG81] H. Kurz and W. Goedecke. Digital parameter-adaptive control of processes with unknown dead time. Automatica, 17:245–252, January 1981.

[KQ92] Simon Kingsley and Shaun Quegan. Understanding Radar Systems. McGraw-Hill, 1992. ISBN 0-07-707426-2.

[Lju99] Lennart Ljung. System Identification: Theory for the User. Prentice-Hall, Upper Saddle River, N.J. USA, 2nd edition, 1999.

[Mat01] The MathWorks, Inc. MATLAB Statistics Toolbox. User’s Guide. Version 3, 2001.

[Mon97] D. C. Montgomery. Design and Analysis of Experiments. Wiley, 1997. ISBN 0-471-15746-5.

[Swa99] Anthony Paul Swanda. PID Controller Performance Assessment Based on Closed-Loop Response Data. Phd thesis, University of California, Santa Bar-bara, California, USA, June 1999.

[Wik02] Maria Wikstr¨om. Utveckling och implementering av ett audiopejlsystem baserat p˚a tidsdifferensm¨atning. Master’s thesis LiTH-ISY-EX-3277-2002, Link¨opings universitet, Link¨oping, Sweden, October 2002. In swedish.

A

Validation of confidence intervals

This appendix comments on the use and applicability of the confidence intervals in Sec-tion 4.2.

The 4 RMS estimates (Chapter 3) have first gone through a transformation to make the variance more constant [Bj¨o03]. The transformation became (RMS error)ˆ(0.73028) (Figure A.1). Then ANOVA [Mon97] was executed and confidence intervals for pair-wise comparisons [Mat01] plotted . The traditional ANOVA table is given in Table A.1. Since all p-values (the column Prob>F) are very small, all factors and interactions have effect with a very high confidence level. The confidence intervals are plotted in Sec-tion 4.2. For the ANOVA and confidence intervals to be valid some prerequisites must be fulfilled [Mon97, Bj¨o03]. These are usually tested by studying some validation graphs [Mon97, Bj¨o03] (Figures A.2-A.3) . We see in these graphs that the prerequisites are not completely fulfilled so we must be somewhat careful in the interpretation of the ANOVA and confidence interval. See [Mon97, Bj¨o03] for more information on this.

Source Sum Sq. d.f. Mean Sq. F Prob>F Method 50.4689 3 16.823 1910.2693 0 InType 564.3458 2 282.1729 32041.1107 0 SNR 1421.6913 1 1421.6913 161434.944 0 Sys 527.049 3 175.683 19949.0399 0 Method*InType 21.1096 6 3.5183 399.5042 0 Method*SNR 5.9101 3 1.97 223.6997 0 Method*Sys 48.857 9 5.4286 616.42 0 InType*SNR 21.597 2 10.7985 1226.1859 0 InType*Sys 13.6246 6 2.2708 257.8478 0 SNR*Sys 45.817 3 15.2723 1734.1939 0 Method*InType*SNR 127.9938 6 21.3323 2422.3104 0 Method*InType*Sys 11.3337 18 0.62965 71.4977 0 Method*SNR*Sys 4.8604 9 0.54004 61.3222 0 InType*SNR*Sys 327.4812 6 54.5802 6197.6538 0 Method*InType*SNR*Sys 44.8829 18 2.4935 283.1397 0 Error 2.5363 288 0.0088066 Total 3239.5586 383

Table A.1: Analysis of Variance table for all methods. Constrained (Type III) sums of squares. Positive transformation: (RMS error)ˆ(0.73028).

−16 −14 −12 −10 −8 −6 −4 −2 0 2 −16 −14 −12 −10 −8 −6 −4 −2 0 log10(cellAbsMeans(:)) log10(cellStd(:))

Before transformation.: Cell std vs. mean

Figure A.1: Plot (without transform) for choosing a variance-stabilizing transform [Mon97] for ANOVA. The transformation is chosen by fitting a straight line to the data points by the least squares method. The outlier in the lower left corner is ignored when calculating the transformation. It is due to zero variance (always the same time-delay estimate) for one factor level combination.

−0.4 −0.2 0 0.2 0.4 0.001 0.0030.01 0.02 0.05 0.10 0.25 0.50 0.75 0.90 0.95 0.98 0.99 0.997 0.999 Data Probability

t149b5: Normal plot of residuals

−0.40 −0.2 0 0.2 0.4 0.6 5 10 15 20 25 30 t149b5: Histogram of residuals No resids=384 0 100 200 300 400 −0.4 −0.2 0 0.2 0.4 0.6 t149b5: Residuals vs. time Time Residuals 0 2 4 6 8 −0.4 −0.2 0 0.2 0.4

0.6 t149b5: Residuals vs. fitted value

Fitted value

Residuals

Figure A.2: Residual analysis for ANOVA and confidence intervals. Ideally the residuals should be Gaussian and the residuals vs. time and fitted value should be within a horizontal band and be structureless [Mon97, Bj¨o03].

1 1.5 2 2.5 3 3.5 4 0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 t149b5: Residuals vs. Method Method Residuals 1 1.2 1.4 1.6 1.8 2 2.2 2.4 2.6 2.8 3 0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.1 t149b5: Residuals vs. InType InType Residuals 1 1.2 1.4 1.6 1.8 2 0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 t149b5: Residuals vs. SNR SNR Residuals 1 1.5 2 2.5 3 3.5 4 0 0.02 0.04 0.06 0.08 0.1 0.12 t149b5: Residuals vs. Sys Sys Residuals

Figure A.3: Standard deviation of residuals versus factor levels for ANOVA and confidence intervals. Ideally the standard deviation for all factor levels should be equal[Mon97, Bj¨o03].