Classification and Localization of Vehicle Occupants Using 3D Range Images

(1)

Using 3D Range Images

PANDURANGA RAO DEVARAKOTA

Do toralThesis inSignal Pro essing

(2)

TRITA-EE 2008-003

ISSN 1653-5146

ISBN 978-91-7178-850-4

SignalPro essingLaboratory

SE-10044Sto kholm

SWEDEN

AkademiskavhandlingsommedtillståndavKunglTekniskahögskolanframlägges

tilloentliggranskningföravläggandeavteknologiedoktorsexamenisignalbehan-

dlingmåndagenden28januari2008klo kan14:00iamphitheatre,BS004,Campus

Limpertsberg,UniversityofLuxembourg,Luxembourg.

©PanduRangaRaoDevarakota,januari2008

(3)

Abstra t

Thisthesisdealswiththeproblemof lassifyingautomotivevehi leo u-

pantsandestimatingtheirposition. Thisinformationis riti alindesigning

futuresmart airbag systemsproviding maximumprote tion for passengers.

A ordingtothe Ameri anNationalHighwayTra Safety Administration

(NHTSA),sin e1990,intheUSA,227deathshavebeenattributedtoairbags

deployedinlow-speed rasheswhi hin luded119 hildren,and22infants. In

these ases,intelligent deployment oftheairbag, basedonthe typeand po-

sition of o upant ould have avoided these fatalitie s. Current ommer ial

lassi ation systemsbased ontraditionalsensors, su h as pressure sensors

arenotabletodete tthepositionofo upants. Vision-basedsystemsaread-

vantageousoverpressuresensorbasedsystems,asthey anprovideadditional

fun tionalitieslike dynami o upantpositionanalysis or hildseatorienta-

tiondete tion. Onthe other hand,vision-based systemshave to opewith

several hallenges, su has, illumination onditions, temperature,humidity,

largevariationofs enes, ost,and omputationalaspe ts.

This thesis presents new pattern re ognition te hniques for lassifying,

lo aliz ing and tra king vehi le o upants using a low-resolution 3-D opti-

altime-of-ight range amera. Thissensor is apable ofprovidingdire tly

adense rangeimage,independentof theillumination onditionsand obje t

textures.Basedonthiste hnology,IEES.A.ispresentlydevelopinga amera

systemfortheappli ationofo upant lassi ation. Aprototypeofthis am-

erahasbeenthebasisforthisstudy. Therstpartofthethesispresentsthe

problemof o upant lassi ation. Herein, weinvestigate geometri feature

extra tionmethodstodis riminatebetweendierento upanttypes. Wede-

velopfeaturesthatareinvariantunderrotationandtranslation. Amethodfor

redu ingthe sizeof thefeaturesetisanalyzedwithemphasisonrobustness

and low omputational omplexity while maintaining highly dis riminative

information. Inaddition, several lassi ation methods are studied in lud-

ingBayesquadrati lassier,GaussianMixtureModel(GMM) lassierand

polynomial lassier. Weproposetheuseofa lusterbasedlinearregressio n

lassierusingapolynomialkernelwhi hisparti ularlywellsuitedto oping

withlargevariationswithinea h lass. Fulls aleexperimentshavebeen on-

du tedwhi hdemonstratethata lassi ationreliabilityofalmost100% an

bea hievedwiththeredu edfeaturesetin ombinationwitha lusterbased

lassier.

Inthis safety riti al appli ation, it isequally important to addressthe

problemofreliabilityestimationforthesystem. State-of-the-artmethodsto

estimatethereliability ofthe lassi ationare basedeitheron lassi ation

outputorbasedondensityestimation. These ondpart ofthethesis treats

estimation of the reliability of the pattern lassi ation system. Herein, a

novelreliabilitymeasureisproposedfor lassi ationoutputwhi htakesinto

a ountthelo aldensityoftrainingdata. Experimentsverifythatthisreli-

abilitymeasureoutperformsstate-of-the-artmethodsinmany ases.

Lastly,theproblemofdynami allydete tingout-of-positiono upantsis

addressed in the thirdpart of the thesis. This task requires dete ting and

(4)

methods,su h asdete ting head-likeobje ts intheimage by analyzing the

lo alshapesarenotrobustwiththe urrentsensor. Manyregionsinas ene

su hastheshoulderortheelbowoftheo upant anbein orre tlydete ted

asthehead. Inorder to opewiththese hallenges, weexploit topologyin-

formationin the range image. A modied Reeb graph te hnique has been

developed thatextra ts atopologi alskeletonof the3D ontourthat is in-

variantunderrotationand translations. Resultsverifythat theReebgraph

dete tssu essfully theheadi.e., thehead always orrespondstooneofthe

end points of the skeleton. Subsequently, a data asso iation algorithm to

sele t the orre t head andidate out of the Reeb graph andidates is pre-

sented. Results show that the resulting head dete tion algorithm basedon

(5)

First,Iwouldliketore ordmysin eregratitudetoProfessorBj

o ¨

^rn^Ottersten,^my

supervisoratKTH,fortheopportunitytopursuemydo toralstudiesinhisgroup.

I am greatly indebted to him for his immense guidan e, patien e, and support

throughout my do toralresear h. His stimulating suggestionsand ideas, sharing

hisknowledge,anden ouragementmademyresear hreallyenjoyable. IfeelBj

o ¨

^rn

is an ideal adviser. I would like to thank Dr. Bruno Mirba h, my supervisor at

IEE,whokeptaneyeontheprogressofmyworkandwasalwaysavailablewhenI

neededhisadvi es. Iamverythankfultohimwhohelpedmetolearnandprovided

te hni al and resear h insight to the proje t. Parti ularly, I am very grateful to

himfor hisstimulations and ontributions to theresear h. Bj

¨ o

^rn ^and^Bruno, ^the

helpre eivedfromyoubothprofessionallyandpersonallywasimmense,andI am

gratefulto youboth ineverypossiblewayandhopeto keepupour ollaboration

inthefuture.

Thisproje twas arriedoutintheframeworkofLIASIT,UniversityofLuxem-

bourg, and in ollaboration withIEE S.A.of Luxembourg. I would liketo thank

allthe people at LIASIT for their support in every possibleway throughout the

proje t. I onveymy a knowledgement to MagaliMartin, LIASIT,for the indis-

pensable help dealing with travel funds, administration during my stay and my

ommutebetweenSto kholm,soI ouldoptimally arryoutmyresear handtrav-

els. Thanksto JoelGrotzformanydis ussionsduringthe oursework,anditwas

greattoworkwithyou. Aspe ialthankstoKarinDeminandAnnikaAugustsson,

KTH,formaking alladministrativeissuesrunningverysmoothly. Colle tiveand

individual a knowledgments are also owed to my olleagues at signal pro essing

group,KTH.

I wouldliketothankallmy oauthorsatIEE;Marta Castillo-Fran o,Dr. Ro-

mualdGinhoux,SergeKater,withouttheir onstantsupportduringdierentphases

oftheproje t,inparti ular,intheexperimentaldesignanddata olle tion,aswell

asintheimplementationofprepro essingandfeatureextra tionroutines,thiswork

would not have been ompleted in time. I would like to thankAlain Garand for

his ex ellent support during the re ordings. I also want to thank Benoit Ries,

Dr.Frederi Grandidier,ThomasSoligna fortheirlivelyworkingatmosphereand

interestingdis ussions. The time I spent with youall at IEE was rewardingand

unforgettable. I amverygrateful toDr. AloyseS hoos,IEE, whois instrumental

(6)

for LIASITproje tsatIEE.I would liketothankDr. LaurentFederspiel, Ra hel

Papa,andthepeopleintheHumanResour edepartmentatIEE,parti ularly,Mr.

Mi hael Hartmann, Cindy Hubert, and Alexandra Pereirafor their onstantsup-

portandhelpdealingwithadministrationandbureau rati mattersduringmystay

in Luxembourg.

Iwouldliketothankmythesisexamination ommitteemembers;ProfessorArne

Leijon fromKTH,Dr. MartinFritzs hefromDaimlerChryslerResear handTe h-

nology, Ulm, Professor ThomasEngel from University of Luxembourg, Professor

ChristophMeinelfromUniversityofPotsdam,Germany,andProfessorDavidBasin

from ETH, Zuri h. Myspe ial thanksto Professor HamidKrim, North Carolina

State University, the United States, for introdu ing me to Reeb graphte hnique

whi histhemaintopi ofPartIIIofthethesis,andforhistimetoa tasopponent

forthisthesis.

I am very thankful to my friends Dr. Rajesh L.V.V.L and Dr. Phani K

Yalavarthy for the wonderful time we spent on Skype, for their advi es and for

sharing many personal things. Words fail me to express my appre iation to my

wife Rekha whose dedi ation, patien e,loveand persistent onden eon me, has

takentheloadomyshoulder. I oweherforbeingunselshlyletherintelligen e,

passionsandambitions ollidewithmine. Finally,Iwouldliketothankmyparents

withoutwhomnoneofthiswouldhavebeenpossible. Theyareunbelievableinspi-

rationtome. Ithankthemforinstillingtheloveoflearninginmeandfortea hing

meaboutintegrity,dignity,andrespe t. Thisthesisisdedi atedtomyparents.

Pandu RangaRaoDevarakota

(7)

Contents vii

1 Introdu tion 1

1.1 Ba kground . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.2 ExistingTe hnologiesforO upantSensing . . . . . . . . . . . . . . 3

1.3 A Novel3DRangeCamera . . . . . . . . . . . . . . . . . . . . . . . 5

1.4 Obje tives,Challenges,andMethodology . . . . . . . . . . . . . . . 10

1.5 TheExperimental Approa h: Measurements. . . . . . . . . . . . . . 17

1.6 OutlineandContributions . . . . . . . . . . . . . . . . . . . . . . . . 19

I O upant Classi ation 27 2 Introdu tion to O upant Classi ation 29 2.1 Introdu tion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

2.2 RelatedWork . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

2.3 OrganizationofPartI . . . . . . . . . . . . . . . . . . . . . . . . . . 31

3 Prepro essing 33 4 Feature Computation 39 4.1 An OverviewofFeatureExtra tionMethods . . . . . . . . . . . . . 39

4.2 FeatureExtra tionforRangeImages. . . . . . . . . . . . . . . . . . 40

4.3 Prin iple ComponentAnalysis(Eigenimages) . . . . . . . . . . . . . 45

4.4 FeatureSubsetSele tion . . . . . . . . . . . . . . . . . . . . . . . . . 46

4.A CurvaturesoftheParaboloid . . . . . . . . . . . . . . . . . . . . . . 46

4.B Des riptionofAllFeatures . . . . . . . . . . . . . . . . . . . . . . . 47

5 Classi ation Methods 51 5.1 Introdu tion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

5.2 Notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

5.3 BayesQuadrati Classier . . . . . . . . . . . . . . . . . . . . . . . . 52

5.4 GMMClassier . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

(8)

5.5 PolynomialClassier (PC). . . . . . . . . . . . . . . . . . . . . . . . 53

5.6 PolynomialClusterClassier(PCC) . . . . . . . . . . . . . . . . . . 55

6 ExperimentalResults 57 6.1 Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

6.2 ValidationCriterionandPerforman eMeasure . . . . . . . . . . . . 58

6.3 Experiments. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

6.4 Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

6.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

II Reliability Estimation of Pattern Classi ation Methods 67 7 Introdu tion to Reliability Estimation 69 7.1 Ba kground . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

7.2 OrganizationofPartII. . . . . . . . . . . . . . . . . . . . . . . . . . 71

8 Data Model 73 8.1 TheLo alDensityofTrainingPatternsAroundaTest Pattern . . . 73

8.2 ApproximationUsingGaussianMixtureModel . . . . . . . . . . . . 75

8.3 Un ertaintyBasedontheLo alDensityofTrainingData . . . . . . 76

8.4 TheChoi eofParameters

C

^and

r

^. ^. ^. ^. ^. ^. ^. ^. ^. ^. ^. ^. ^. ^. ^. ^. ^. ^. ^. ⁷⁸

8.5 Reje tMethods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78

8.A Derivationof Lo alDensityoftheTrainingPatterns . . . . . . . . . 80

9 ExperimentalResults 83 9.1 ATwo- lassSyntheti Example . . . . . . . . . . . . . . . . . . . . . 83

9.2 ExperimentsonReal-dataSets . . . . . . . . . . . . . . . . . . . . . 86

9.3 ExperimentsonO upantDataSet. . . . . . . . . . . . . . . . . . . 93

9.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95

III 3D Skeleton-based Head Dete tion and Tra king 97 10 Introdu tion to Head Dete tion 99 10.1 Ba kground . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99

10.2 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99

10.3 Dete tionApproa h . . . . . . . . . . . . . . . . . . . . . . . . . . . 101

10.4 OrganizationofPartIII . . . . . . . . . . . . . . . . . . . . . . . . . 102

11 TopologyCodingof 3-DObje ts 105 11.1 Ba kground . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105

11.2 MorseTheory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105

(9)

12MethodforHead Dete tion from the Reeb Graph 109

12.1 ReebGraphExtra tiononRangeData . . . . . . . . . . . . . . . . 109

12.2 VoxelNeighborhood . . . . . . . . . . . . . . . . . . . . . . . . . . . 111

12.3 TheReebGraphExtra tionAlgorithmforHeadDete tion . . . . . 112

12.4 TheMultipleHypothesisTra king . . . . . . . . . . . . . . . . . . . 116

13ExperimentalResults 119 13.1 EvaluationMethodologyandDatabase. . . . . . . . . . . . . . . . . 119

13.2 TopologyofExtra tedReebGraphs . . . . . . . . . . . . . . . . . . 120

13.3 GroundTruthInformation. . . . . . . . . . . . . . . . . . . . . . . . 124

13.4 EvaluationofHeadDete tionMethod . . . . . . . . . . . . . . . . . 128

13.5 EvaluationofMultipleHypothesisHeadTra kingAlgorithm . . . . 131

13.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135

IV Epilogue 137

14ThesisCon lusions 139

(10)

(11)

Introdu tion

This thesis des ribes new re ognition te hniques for lassifying, lo alizing, and

tra kingtheautomotivevehi leo upantsusingalow-resolution3Dopti al time-

of-ightrange amerafordesigningso- alledsmart airbags. This hapterprovides

anoverview of urrentairbagte hnologyand pointsto theriskof airbagindu ed

injurybe auseofunne essarydeployment. Existingte hnologiestoprovidevehi le

o upant lassi ationandlo alizationaredis ussed. Theformulationofthethesis

andthesummaryof ontributionsarepresented. The hapteralsogivesanoutline

ofthethesis,andpresentsthepubli ationsforegoingthisthesis.

1.1 Ba kground

Passengersafety has be ome one of the mostimportant topi s in modern ar in-

dustry. Inthis dire tion,theadditionof airbagsto passengerrestraint systemsis

arelativelynew initiative. Airbags aredesigned as a supplement to seat-beltre-

straintsystemstoensurepassengersafetyin aseoffrontal ara idents. Airbags

wereinventedin1953andtheirdevelopmentbeganinthelate1980's. Intheauto-

motiveindustry, thein orporation ofairbagsin the presentdesign was startedin

the1990's. In aseof afrontal ar rash,airbagsbeginto inateanddeployvery

qui kly (see Figure 1.1). Thereby, airbagsredu e the han e that the o upant's

headandupperbodywillstrikesomepartofthevehi le'sinterior. Whiletheseat

beltsaretheprimaryrestraints,airbagsoersupplementaryprote tionandredu e

theriskofseriousheadinjury. Areportpublishedin2004showsthat,throughout

1987-2003,airbagssavedapproximately16,905livesintheUnitedStates[Adm04℄.

Thisstudy has shown that urrent airbagshavebeenhighly ee tive in redu ing

overallfatalities.

There have been instan es, where the o upants an also beseriouslyinjured

or killed notbe auseof afrontal rash but due to the(unneeded and unwanted)

deploymentofairbags. Forinstan e,whenanairbagdeploysonarear-fa inginfant

seat,it an ausesevereor evenfatalinjuriestotheinfant. Thesameisthe ase,

(12)

Figure1.1: Airbagdeploymentonthefrontseatpassenger.

if a passenger is too lose to the airbag while it is inating. A ording to the

Ameri an National Highway Tra Safety Administration (NHTSA), sin e1990,

in theUnited States,227deaths havebeenattributed to airbagsdeployed inlow-

speed rasheswhi hin luded119 hildrenbetweentheages1and11,and22infants

[Adm04℄.

To this end, the demand for the saferdeploymentof airbags is a knowledged

withthein reasinglyfrequentappli ationofairbagsinautomobiles. Carmanufa -

turersandvehi lesafetyadministrationsputin reasingweightonthedevelopment

of intelligent passengersafety systems, in order to redu e the in iden e of airbag

indu ed injury. A ording to FMVSS208, a safety standard was introdu ed by

NHTSA, and hasbeenfully ee tive sin e2006. A ordingto this standard, the

urrent airbag te hnology requires an o upant lassi ation system, that dea -

tivates an airbag in ase of an infant seat. The same should be the ase for a

non-o upied seat for redu ing repair osts. In addition to these safety require-

ments, it would be bene ial if thesystem oulddetermine the orientation of an

infantseatandthepositionof anadult,as this ouldallowtheadjustmentof the

airbagdeploymenttoboththetypeandposition oftheo upant. Theautomati

adjustmentofthedeploymentoftheairbagbasedono upanttypeandposition,

(13)

Figure 1.2: A pressuresensitivematrix sensor OC ® sensor) developedbyIEE for

1.2 Existing Te hnologies for O upant Sensing

Pressure Sensitive Sensors

Therstsystemthat enteredthemarketin 2003, omplyingwiththenewregula-

tionsforo upant lassi ation,usesseatbasedsensorsthat lassifytheo upan y

a ordingtoweight. OneofthesesystemsistheO upantClassi ationsensorsys-

temOC ®sensordevelopedbyIEES.A.([IEE℄). Theo upant lassi ationsystem

onsistsofapressuresensitivematrixsensor (seeFigure1.2),whi hmeasures the

two-dimensionalpressureprolein theseatarea. A patternre ognitionalgorithm

isusedasabasisforo upant lassi ationdepending onthemeasuredpressured

prole. Thissystemallowsthesuppressionofanairbag inthepresen eofa hild

withor withouta hildrestraintsystem,while allowingafull airbagdeployment

in the presen e of anadult person. However,in ontrast to the existing airbag

system,futuresystemswillbemoreadvan edredu ingtheriskandmagnitude of

injuriesbyautomati allyadaptingtheairbagandseatbeltpretentionto thedriv-

ingstatusofthevehi le,itso upants,and the rashseverity. Su h sophisti ated

airbagdeploymentstrategiesrequireinformationaboutthesizeoftheo upantand

his/herposition,relativeto theairbag module in order to provideea h o upant

withoptimalprote tionandsafety. Theexistingte hnologiesbasedonthepressure