Probabilistic Fault Isolation in Embedded Systems Using Training Data

Probabilistic Fault Isolation in Embedded Systems Using Training Data

AXEL CORNELL

Masters’ Degree Project

Stockholm, Sweden April 2008

Intheheavyvehicleindustrycustomers,lawsandincreasinglycomplexpro-

cessesdemand methods of supervisingeveryaspectof atruck. Faultisolation

systems are introduced to do just that. In order to assure a sustainable de-

velopment new types of isolation systems are investigated to substitute the

consistencybasedisolationsystemsoftoday.

Inthis thesisan application of aprobabilisticisolation method that ranks

possiblefaults ontheirlikelinessofbeingafaultintheprocessisimplemented

andevaluatedasapossiblefuturereplacementoftoday'ssystem. Thismethod

basestheisolationontrainingdatacollectedfrommeasurementsontheprocess

andanobservationoftheprocess.

The probabilistic isolation method is evaluated on how it performs under

dierent circumstancessuch astheeect ofdierent amountsof trainingdata

and how well it performs if the tests and observations of the process are of

varyingquality.

Solutiontoseveralproblemsthatarisewhenthismethodisimplementedare

alsoinvestigatedsuchashowthesystemhandlescaseswhereseveralfaultsoccur

at the sametime, what happens ifthere are missing datain theobservations

ofthesystemandhowtosolveproblemsthatinvolveexecutiontimeswhich is

importantinembeddedsystems.

The results that are derived show that this probabilistic isolation system

performswellontheprocessasitistodayandthatthisisagoodsubstitutewhen

developingforfutureprocesses. Thereishoweveraneedforfurtherdevelopment

of thesystemsuchas improvedisolationwhen there areseveral faults present

in theprocessandquestions onhowto collectandstorethetrainingdata still

remain to be answered. A full scale implementation would allow for better

comparisonwiththecurrentsystemandgivemoreinformationonruntimeand

storageproblems.

This masterthesishasbeencarriedoutatScania CVABandin particulathe

PowertrainControlSystemDevelopmentDiagnosisgroup. Thankstoeverybody

foracceptingme intothegroupandforallsupportthroughouttheproject.

SpecialthankstoDan HallgrenandAnna Pernestålatthediagnosisgroup

forsharingtheirknowledgeandbeingpatientasmentorstomeduringthework

onthisthesis.

AcknowledgmentstoProfessorBoWahlbergattheAutomaticControlgroup

at the Royal Institute of Technology in Stockholm for his work and help as

examinerofthethesis.

Last, but denitely least, I would like to thank my coworker Anders Sel-

hammerforhisendlesspatience,hardworkandfriendshipwhileworkingonhis

parallelthesis.

Abstract 1

Acknowledgements 1

Contents 2

Abbreviations 5

List ofFigures 7

List ofTables 9

1 Introduction 10

1.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

1.2 Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

2 Fault Isolation 12 2.1 FaultDiagnosisProblem . . . . . . . . . . . . . . . . . . . . . . . 12

2.2 ComponentsandBehaviorModes. . . . . . . . . . . . . . . . . . 13

2.3 Observations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

2.4 DecisionStructure . . . . . . . . . . . . . . . . . . . . . . . . . . 14

2.5 ConsistencyBasedDiagnosisSystems . . . . . . . . . . . . . . . 14

2.6 AProbabilisticApproachtoFaultIsolation . . . . . . . . . . . . 15

2.7 TrainingData . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

2.8 ComputationsofProbabilities . . . . . . . . . . . . . . . . . . . . 16

2.8.1 Priorprobability . . . . . . . . . . . . . . . . . . . . . . . 17

2.8.2 TheLikelihood . . . . . . . . . . . . . . . . . . . . . . . . 17

2.8.3 NormalizationFactor. . . . . . . . . . . . . . . . . . . . . 18

2.8.4 PosteriorProbability . . . . . . . . . . . . . . . . . . . . . 18

3 SolvingReal Problems 19 3.1 MethodsforHandling IncompleteObservations . . . . . . . . . . 19

3.1.1 AssumptionofaValueasaSolutiontoLostData . . . . 19

3.1.2 Marginalization. . . . . . . . . . . . . . . . . . . . . . . . 21

3.1.3 EliminationofMissingData. . . . . . . . . . . . . . . . . 23

3.1.4 TreatingMissingValuesAsaThirdValue . . . . . . . . . 24

servations . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

3.1.6 ACloserComparisonofMarginalizationandElimination ofLostData . . . . . . . . . . . . . . . . . . . . . . . . . 26

3.2 TheUseofSubsystems. . . . . . . . . . . . . . . . . . . . . . . . 28

3.2.1 TheAdvantagesofSubsystems ForStorageandIsolation 29 3.2.2 The Advantages of Subsystems For Collecting Training Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

3.3 HandlingMultipleModesIntheProcessUnderDiagnosis . . . . 29

3.3.1 MultipleModesInOneSubsystem . . . . . . . . . . . . . 30

3.3.2 KnownMultipleModesInaSubsystem . . . . . . . . . . 31

4 PerformanceMeasures 33 4.1 ExpectedCorrectness . . . . . . . . . . . . . . . . . . . . . . . . 34

4.2 ExpectedProbabilityofCorrectClassication. . . . . . . . . . . 35

4.3 ExpectedRatiooftheUnderlyingProbabilityandtheMostProb- ableWrongEstimation. . . . . . . . . . . . . . . . . . . . . . . . 35

5 Evaluationof the DiagnosisSystem 37 5.1 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . 37

5.2 AmountofTrainingData . . . . . . . . . . . . . . . . . . . . . . 39

5.3 Robustness to Probabilities of False Positive Observations and False NegativeObservations . . . . . . . . . . . . . . . . . . . . . 43

5.4 RobustnessWith Respectto thePriors. . . . . . . . . . . . . . . 49

5.5 EvaluationoftheMethods forHandling MissingData . . . . . . 56

5.6 WhatHappensIfThere IsaDoubleFault . . . . . . . . . . . . . 62

6 Taking Prior Information Into Account 67 6.1 WeighingtheTwoMethods . . . . . . . . . . . . . . . . . . . . . 67

6.2 EvaluationoftheCombinedMethod . . . . . . . . . . . . . . . . 69

6.3 TrainingDataCompensatesforWrongModels . . . . . . . . . . 71

7 Evaluationof the PracticalAspects ofan Implementation 76 7.1 ImportantParametersandData . . . . . . . . . . . . . . . . . . 76

7.1.1 TrainingData. . . . . . . . . . . . . . . . . . . . . . . . . 76

7.1.2 PriorProbabilities . . . . . . . . . . . . . . . . . . . . . . 77

7.1.3 Decision Structure and Assumed Probabilities of False PositiveandNegative . . . . . . . . . . . . . . . . . . . . 77

7.2 RisksandDiculties . . . . . . . . . . . . . . . . . . . . . . . . . 77

7.2.1 WrongModel . . . . . . . . . . . . . . . . . . . . . . . . . 78

7.2.2 DoubleFaults. . . . . . . . . . . . . . . . . . . . . . . . . 78

7.2.3 CollectingTrainingData . . . . . . . . . . . . . . . . . . 78

7.2.4 StoringTrainingData . . . . . . . . . . . . . . . . . . . . 79

7.2.5 Runtime . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79

8.1 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80

8.2 Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . 81

8.3 FutureWork . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81

A Variablesand Notation 82

B Probability Rules 83

B.1 BasicRules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83

B.2 BayesRule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83

B.3 Marginalization . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83

Bibliography 84

ECU ElectronicControlUnit

CDF CumulativeDistributionFunction

CCDF ComplementaryCumulativeDistributionFunction

FTC FaultTolerantControl

Theusedvariablesaredened inAppendix A

2.1 Overviewofthediagnosissystem . . . . . . . . . . . . . . . . . . 12

3.1 Resultstothesameisolationwithdierentmethodsforhandling

missingdatain theobservation . . . . . . . . . . . . . . . . . . . 25

5.1 Plotof

µ ¯

^and

ν ¯

^{forVaryingAmountsofTrainingDataNforEach}

Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

5.2 PlotoftheProbabilitythat theCorrectModehasaProbability

GreaterthanxforVaryingAmountsofTrainingDataNforEach

Mode,AlsoCalledtheComplementaryCumulativeDistribution

Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

5.3 Plot of

µ ¯

^and

ν ¯

^{for Dierent} Probabilities of False Alarms and MissedAlarms . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

5.4 PlotoftheCCDFof theCorrectUnderlying ModesProbability

forDierentProbabilitiesofFalseAlarmsandMissedAlarms . . 45

5.5 Plot of

µ ¯

^and

ν ¯

^{for Dierent} Probabilities of False Alarms and MissedAlarms . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

5.6 PlotoftheCCDFof theCorrectUnderlying ModesProbability

forDierentProbabilitiesofFalseAlarmsandMissedAlarms . . 48

5.7 Plotof

µ ¯

^and

ν ¯

^{forDierentPrior}Probabilities . . . . . . . . . . 50 5.8 PlotoftheCCDFof theCorrectUnderlying ModesProbability

forDierentPriorProbabilities . . . . . . . . . . . . . . . . . . . 51

5.9 Plotof

µ ¯

^and

ν ¯

^{forDierentPrior}Probabilities . . . . . . . . . . 52 5.10 PlotoftheCCDFof theCorrectUnderlying ModesProbability

forDierentPriorProbabilities . . . . . . . . . . . . . . . . . . . 53

5.11 Plotof

µ

^and

ν

^forPriorProbabilities

p 1

^and

p 10

^{forSystem1 . 54}

5.12 Plotof

µ

^and

ν

^forPriorProbabilities

p 1

^and

p 10

^{forSystem1 . 54}

5.13 Plot of

µ

^and

ν

^{for Isolating} Observations with Missing Data UsingMarginalizationand EliminationofMissingData . . . . . 57

5.14 PlotoftheCCDFof theCorrectUnderlying ModesProbability

forIsolatingObservationswithMissingDataUsingMarginaliza-

tionandEliminationofMissingData. . . . . . . . . . . . . . . . 58

5.15 Plot of

µ

^and

ν

^{for Isolating} Observations with Missing Data AssumingtheMissingValuestoEither0or1 . . . . . . . . . . . 59

ity for Isolating Observations with Missing Data Assuming the

MissingValuestoEither0or1 . . . . . . . . . . . . . . . . . . . 60

5.17 Plot of

µ ¯

^{for System One and Two for Isolating} Observations with Double Faults andDierentProbabilitiesof False Positive

andNegativeObservations. . . . . . . . . . . . . . . . . . . . . . 63

5.18 Plot of

ν ¯

^{for System One and Two for Isolating} Observations with Double Faults andDierentProbabilitiesof False Positive

andNegativeObservations. . . . . . . . . . . . . . . . . . . . . . 63

5.19 Plot of the CCDF of the Correct Underlying Modes, the Most

LikelyUnderlyingModesandtheLeastLikelyUnderlyingModes

ProbabilitiesforIsolatingObservationswithDoubleFaultsWith

p f p = 0.01

^and

p f n = 0.3

^{. . . . . . . . . . . . . . . . . . . . . . . 64}

5.20 Plot of the CCDF of the Correct Underlying Modes Probabil-

ity for Isolating Observations with Double Faults for Dierent

ProbabilitiesofFalsePositiveandNegativeAlarms . . . . . . . . 65

6.1 Plotof

n ^m _X /N ^m

^Against

N ^m

^{forFourDierent}Observationsand theSameBehaviorMode . . . . . . . . . . . . . . . . . . . . . . 68

6.2 Plotof

µ ¯

^and

ν ¯

^fortheCombinationwiththeThesis[2] . . . . . 70 6.3 PlotoftheCCDFof theCorrectUnderlying ModesProbability

fortheCombinationwiththeMethodin [2] . . . . . . . . . . . . 70

6.4 Plot of

µ ¯

^and

¯ ν

^{Showing How Training Data Corrects Faulty}

Knowledge. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

6.5 PlotoftheCCDFof theCorrectUnderlying ModesProbability

ShowingHowTrainingDataCorrectsFaultyKnowledge . . . . . 75

2.1 Atypicaldecisionstructure . . . . . . . . . . . . . . . . . . . . . 14

2.2 AnExampleof TypicalTrainingData . . . . . . . . . . . . . . . 16

3.1 Anexampleofalargerdecisionstructurethatcanbepartitioned intotwosmallersubmatrices . . . . . . . . . . . . . . . . . . . . 28

3.2 Thedecisionstructurefortheprocess . . . . . . . . . . . . . . . 31

5.1 TheGeneralSetup fortheExperimentsin Chapter5 . . . . . . . 38

5.2 TheFirstoftheTwoDecisionStructuresUsedin theEvaluation 38 5.3 TheSecondoftheTwoDecisionStructuresUsedintheEvaluation 38 5.4 ResultstoSimulationswithVaryingAmountofTrainingData . 39 5.5 Resultsfrom Figure5.2 ShowingtheProbabilitythat theProb- abilityoftheCorrectModesisGreaterThan70% . . . . . . . . 42

5.6 TheProbabilitiesofFalse PositiveandFalseNegativeThat Are EvaluatedFor. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

5.7 TheProbabilitiesofFalse PositiveandFalseNegativeThat Are EvaluatedFor. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

5.8 ResultstoSimulationswithDierentProbabilitiesforFalseAlarm

p f p

^{and MissedAlarm}

p f n

^{. . . . . . . . . . . . . . . . . . . . . . 44}

5.9 Resultsfrom Figure5.4 ShowingtheProbabilitythat theProb- abilityoftheCorrectModesisGreaterThan70% . . . . . . . . 46

5.10 TheDierentPriorProbabilitiesThatAreSimulatedfor. . . . . 49

5.11 ResultstoSimulationswithDierentPriorProbabilities . . . . . 50

5.12 ResultstoSimulationswithDierentPriorProbabilities . . . . . 51

5.13 ResultstoSimulationswithDierentMethodsForHandlingMiss- ing Data in TheObservation for Both

l = 1

^and

l = 2

^Missing Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

5.14 TheProbabilitiesof FalsePositiveand FalseNegativeObserva- tionsThatAreEvaluatedFor . . . . . . . . . . . . . . . . . . . . 62

6.1 InformationAbouttheFourDierentObservationsintheExper- imentontheAmountof NeededTrainingData . . . . . . . . . . 69

6.2 Resultsfrom Figure6.3 Showingthe Probabilitythat theProb- abilityoftheCorrectModesisGreaterThan70% . . . . . . . . 71

WrongModel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

6.4 The SecondDecision Structures Modied for Experiment With

WrongModel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

A.1 Variablesasweusethemin thisthesis . . . . . . . . . . . . . . . 82

Introduction

1.1 Background

The heavy truck industryis driven mainly by customer demands. Customers

demandmoreandmoreuptimeandlongerlifecyclesoftheirheavytruckeets.

Any defects must be found and breakdowns quickly repaired, either by fault

tolerantcontrol,FTC,orbyreparationsbyamechanic.

Not only customers demand better performances but law requirements on

bothanationalandinternationalbasis. Requirementsoflowerandloweremis-

sions force the industryto spend large amounts of resourceson research and

development. Not onlytomaintainlowemissionsbut alsotondfaultsin the

trucksothat emissionstandardscanbemetandto assuresafetyontheroads.

Todays generation of isolation systemsin trucksallow manufacturersto meet

thestandardsoftodaybutnewerandmorepreciseisolationsystemswillallow

asustainabledevelopmentandtomeetfuturerequirements.

Asenginesbecomemoreandmorecomplexprocessesitbecomesmoredif-

culttomonitorexactlywhat goesonintheengine. Asaneectnewmethods

ofisolatingfaultsin heavytrucksarecontinuouslydeveloped.

Inthisthesisanapplicationofaprobabilisticfaultisolationthatrankspos-

siblefaults ontheirlikelinessofbeingpresentfaultsisimplementedandinves-

tigated asapossiblesubstitute to todaysisolationsystem. The method bases

the isolationon collectedtraining datasothat the model that theisolation is

based on is creatednot byengineers but aself-generated model createdfrom

collecteddataoftherealprocess.

Researchhasbeenconducted onthesubjectandthisthesisis primarilyan

applicationofworkin[1].