Geometric, Topological and
Optimization Methods
Per Enqvist
DoctoralThesis
Stockholm, 2001
Optimization andSystems Theory
Department of Mathematics
RoyalInstitute of Technology
TRITA-MAT-01-OS-03
ISSN1401-2294
ISRNKTH/OPTSYST/DA01/02{SE
Abstract
Thisthesisconsistsoffourpapersdealingwithvariousaspectsofspectral
estimationandthestochasticrealizationproblem.
In Paper A a robust algorithm for solvingthe Rational Covariance
Extension Problem with degree constraint (RCEP) is presented. This
algorithm improves onthe current stateof art that is basedon convex
optimization. The new algorithm is based on a continuation method,
and uses a change of variables to avoid spectral factorizations and the
numericalill-conditioning in the original formulationoccuring forsome
parametervalues.
InPaperBaparameterizationoftheRCEPisdescribedinthecontext
ofcepstralanalysisandhomomorphic ltering. Further,itisshownthat
thereisanaturalextensionoftheoptimizationproblemmentionedabove
to incorporate cepstral parametersasaparameterizationof zeros. The
extended optimization problem is also convexand, in fact, it is shown
thatawindowofcovariancesandcepstrallagsformlocalcoordinatesfor
ARMAmodelsofordern.
InPaperCthegeometryofshaping ltersisanalyzedbyconsidering
parameterizationsusing variouscombinationsofpoles, zeros,covariance
lags,cepstrallagsandMarkovparameters. Inparticular, thecovariance
andcepstralinterpolationproblemisstudiedusingdi erentialgeometry
anddualitytheory. Assumingthereisanunderlyingsystemthatisstable
and minimum phase, it is shown in this paper that there is a one-to-
onecorrespondencebetweenMarkovparametersandcepstralcoeÆcients.
An approach based on simultaneous Markov and covarianceparameter
interpolationhasbeenstudiedbySkeltonet.al. Inthispaperitisstudied
fromaglobalanalysispointofview.
Paper D deals with a regularization of two lter design methods,
namelythecovarianceandcepstralmatchingARMAdesignmethodand
covariance matching for MA lters. Both methods are posed as opti-
mizationproblems,andabarriertermisintroducedtoachieveastrictly
minimumphase solution. As aresultof the regularization,exactinter-
polationis tradedforagainin entropy, andthemapfromdata to lter
de nedbytheoptimizationproblemsisturnedintoadi eomorphism.
Cepstralanalysis,Markovparameters,Globalanalysis,Convexoptimiza-
tion,Continuationmethods,Entropymaximization.
MathematicsSubject Classi cation(1991): 93E12,60G10,42A70,
Acknowledgments
AlthoughmuchofmytimeasaPh.D.studenthasbeenalonewolfrace,
I could nothavemade it withoutthe help and support of anumber of
keypersons.
My rst and warmest thanks go to my advisor Professor Anders
Lindquist, who despiteaheavy workload hasmanaged to beavailable
fordiscussingmyspuriousideasandforallhishelpwithformalizingand
carryingout theseideas. Thesecond most importantresearcherI have
had the privilege to work with is Professor Chris Byrnes. During his
intensevisitsatKTHhehasmanagedtoboost myresearchwithhiscu-
riosityandgoodmathematicalintuition. Especially,ithasbeenverynice
tobeacoauthoroftwopaperstogetherwiththedynamicduo;Professor
AndersLindquistandProfessorChrisByrnes.
I am also grateful toProfessor Sergei Gusev and ProfessorTryphon
Georgiouforencouraging discussionsthat directlyor indirectlyinspired
toresultspresentedinPaperA ofthisthesis.
Further,IthankProfessorClydeF.Martinforsupervisingmydiploma
workatTexasTechandalltheteachersIhavecomeacross,tonameafew,
ProfessorTomas Bjork,DocentKristerSvanberg, andProfessor Anders
Lindquistforinspiringandinterestingmein graduatestudies.
MycolleaguesAndersDahlen,RyozoNagamune,JorgeMariandUlf
Jonssonhasformed avaluablediscussion panelalwaysready to bounce
any new ideas. The stimulating social environment at the Division of
OptimizationandSystemsTheoryhasbeenanimportantfactormaking
itajoytogettowork. Especially,attimeswhenyouneedsomediversion
fromtheresearch,suchasdoingsports(thankyouHenrikandPetter)or
justhaveachat. InparticularIwouldliketothankmytworoom-mates
during these years: Camilla Landen and Torvald Ersson. Camilla and
I sharedtheinitial confusionasbeginnerPh.D. studentsand wehelped
eachotherthroughthe rstcourses. Ihavealsoappreciatedthecompany
ofTorvaldwhosharemyinterestinsports.
Finally,Iwouldliketothankmyfamily. Withtheirsolidsupportand
encouragementin theback,nothingseemstoodiÆcult.
1 Introduction 1
1 LinearSystemModels . . . . . . . . . . . . . . . . . . . . 1
2 RealizationTheory . . . . . . . . . . . . . . . . . . . . . . 2
2.1 DeterministicRealization Theory . . . . . . . . . . 3
2.2 StochasticRealizationTheory. . . . . . . . . . . . 4
2.3 ConnectionsbetweenStochasticandDeterministic Realization . . . . . . . . . . . . . . . . . . . . . . 6
3 TheRationalCovarianceExtensionProblem . . . . . . . 9
3.1 TheMaximumEntropyModel . . . . . . . . . . . 10
3.2 ParameterizationsofRelaxedVersionsof theRCEP 13 4 ClassicalSpeechModeling . . . . . . . . . . . . . . . . . . 15
4.1 Acoustic TubeModeling . . . . . . . . . . . . . . . 18
4.2 LosslessTubeEquations . . . . . . . . . . . . . . . 19
4.3 TheE ectsofNasalCoupling. . . . . . . . . . . . 21
5 Foliations,TransversalityandLocal Coordinates . . . . . 22
6 SummaryofthePapers . . . . . . . . . . . . . . . . . . . 27
6.1 PaperA . . . . . . . . . . . . . . . . . . . . . . . . 28
6.2 PaperB . . . . . . . . . . . . . . . . . . . . . . . . 28
6.3 PaperC . . . . . . . . . . . . . . . . . . . . . . . . 28
6.4 PaperD . . . . . . . . . . . . . . . . . . . . . . . . 29
A AHomotopyApproachtoRationalCovariance Extension with Degree Constraint 35 1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . 37
2 TheOriginal OptimizationProblem . . . . . . . . . . . . 39
3 ANewFormulationoftheOptimizationProblem . . . . . 43
4 Homotopyapproach . . . . . . . . . . . . . . . . . . . . . 55
4.2 Predictor-Correctormethod . . . . . . . . . . . . . 58
5 TheAlgorithm . . . . . . . . . . . . . . . . . . . . . . . . 59
5.1 Adaptivesteplengthprocedure . . . . . . . . . . . 59
5.2 HowtochoosetheinitialstepsizeÆ. . . . . . . . 62
5.3 Apracticalalgorithm . . . . . . . . . . . . . . . . 64
6 Convergenceoftheproposedalgorithm. . . . . . . . . . . 64
7 Acknowledgments. . . . . . . . . . . . . . . . . . . . . . . 67
B CepstralcoeÆcients, covariance lags and pole-zero mod- elsfor nite datastrings 73 1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . 75
2 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . 78
2.1 Analysisbasedonin nitedata . . . . . . . . . . . 78
2.2 LPC lters . . . . . . . . . . . . . . . . . . . . . . 80
2.3 CepstralmaximizationandLPC lters. . . . . . . 81
3 Homomorphic lteringandgeneralizationsofLPC ltering 83 3.1 Cepstral and covariancewindows as local coordi- natesforpole-zeromodels . . . . . . . . . . . . . . 83
3.2 CepstralmaximizationandageneralizationofLPC design . . . . . . . . . . . . . . . . . . . . . . . . . 92
4 Realizationalgorithmsforlattice-laddernotch (LLN) lters100 4.1 Selectingthepositivepseudo-polynomial. . . . . . 103
4.2 Thealgorithm . . . . . . . . . . . . . . . . . . . . 105
4.3 Examples . . . . . . . . . . . . . . . . . . . . . . . 107
5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . 110
C Identi abilityandwell-posednessofshaping- lterparam- eterizations: Aglobal analysisapproach 119 1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . 121
2 Somegeometricrepresentationsof classesofmodels . . . 126
3 MainResults . . . . . . . . . . . . . . . . . . . . . . . . . 129
4 GlobalanalysisonP n . . . . . . . . . . . . . . . . . . . . 132
5 Identi abilityofshaping lters . . . . . . . . . . . . . . . 140
6 Thesimultaneouspartialrealizationproblem . . . . . . . 147
7 Zeroassignabilityvs. cepstralassignability . . . . . . . . 158
A Divisorsandpolynomials . . . . . . . . . . . . . . . . . . 162
B CalculationofcepstralcoeÆcients . . . . . . . . . . . . . 163
C ConnectivityofP (c) . . . . . . . . . . . . . . . . . . . . 164
D A convex optimization approach to ARMA(n,m) model
design from covariance and cepstrumdata 173
1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . 175
2 LocalcoordinatesforARMAmodels . . . . . . . . . . . . 179
3 Optimizationproblemsforcepstrumandcovarianceinter- polation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
4 RegularizationofProblem (P). . . . . . . . . . . . . . . . 188
5 RegularizationofProblem (M) . . . . . . . . . . . . . . . 196
6 Simulations . . . . . . . . . . . . . . . . . . . . . . . . . . 200
6.1 MAFilter Design. . . . . . . . . . . . . . . . . . . 200
6.2 ARMA FilterDesign . . . . . . . . . . . . . . . . . 203
This introduction is intended to provide some backround material for
understanding the four paperson stochastic realization theory forming
themainbodyofthisthesis. Evenforthereaderswhoarefamiliar with
stochastic realization theory, the introduction will be useful for setting
notationandforintroducingconceptstobeusedlaterfromrelated elds,
such asspeechprocessing,informationtheoryanddi erentialgeometry.
1 Linear System Models
Before stochastic realization theory can be studied, some basic linear
systemtheoryis presented. Throughoutthis thesisthemodelsarecon-
sidered to be linear, and this class of models are described next. A
linearmodel canbe seenasalinear mappingw:U!Y from aninput
space U to an output space Y, bothof which are real vector spaces of
scalar sequences. If f:::;u
1
;u
0
;u
1
;:::g 2 U is a sequence of inputs,
and f:::;y
1
;y
0
;y
1
;:::g2 Y is the corresponding sequence of outputs,
andthesystemisassumedtobecausalandtime-invariant,
y
m
= 1
X
k =0 w
k u
m k
= 1
X
k =0 w
k z
k
u
m
=w(z)u
m
; (1)
forsomeparametersw
k
,where z denotesaforwardshiftoperator. The
parametersw
k
arecalledtheMarkovparametersofthesystemandw(z)
thetransferfunction. Theinput-outputmapcanbedescribedbyablack
boxrepresentationasdepictedin Figure1.
Inthisthesisitisassumedthatthetransferfunctionw(z)belongsto
u
m
-
w(z)
-
y
m
Figure1: Blackboxmodel
asaratioofpolynomials:
w(z)=
(z)
a(z)
= 1
X
k =0 w
k z
k
; (2)
where(z)anda(z)arepolynomialsof niteorderngivenby
(z) 4
=
0 z
n
+
1 z
n 1
+:::+
n
;
j
2R; j=0;1;:::;n; (3)
a(z) 4
= a
0 z
n
+a
1 z
n 1
+:::+a
n
; a
j
2R; j =0;1;:::;n: (4)
Forthemainpartofthis thesis,wewill furtherassumethatw(z)is
proper i.e., the order of (z) is less than orequal to the order of
a(z),
stable i.e., the polynomial a(z) has all its roots in the open unit
disc,
andforthemostpartalsothatw(z)is
minimum phase i.e., the polynomial (z) has all its roots in the
openunitdisc.
2 Realization Theory
Thetheoretical frameworks usedformodelbuilding, basedonparame-
tersobtainedbyobservingasystem,iscalledrealizationtheory. Givena
(linear) system,a number ofdi erentcharacterizing parameterscanbe
determined from the system, such asthe Markovparameters. Realiza-
tiontheorydealwiththeinverseproblem,namelythesynthesisofsystems
basedon aset ofcharacterizing parameters. There are twobranchesof
realization theory, namely deterministic and stochastic realization the-
sincethisdetermineswhichtypeofcharacterizingparametersthatcanbe
estimatedfrom observingthesystem, thecharacterizingparametersare
di erentinthetwotheories. Roughlyspeaking,deterministicrealization
theoryisbasedonMarkovparametersandstochasticrealization theory
isbasedoncovariancelags.
Next, themain ideasof thetwotheoriesare explainedand thecon-
nectionbetweenthemisdiscussed.
2.1 Deterministic Realization Theory
Indeterministicrealization theory, theinput isassumed tobeaknown
control signal and the output signal is observed as the system evolves
overtime. Althoughtherealizationtheoryisindependentoftheparticu-
larapplication,itisoftenusefultoconsiderexamplestoguidethestudies
at thehigher abstraction level. Realization of systemsis used in many
di erent elds of applications, e.g. mechanicalengineering,biology and
physics. Tonameafew,ineconomics,itisoftenassumedthatthein a-
tionratecanbemodeledasasystemcontrolledbytheinterestrate,and
inchemicalengineering,concentrationlevelsarecontrolledbydilutingor
addingsubstances.
Onewayofbuilding amodel forasystemisto rstestimate awin-
dowof Markovparametersw
0
;:::;w
N
from observedinputand output
sequences, and then to determine a realization w(z) based on this pa-
rameterset. Forastable linearsystemtheMarkovparametersform an
exponentiallydecreasingsequence,andifthesequenceistruncatedafter
w
N
therelation(1) appliedform=0;1;:::;N givesthesystem
2
6
6
6
4 y
0
y
1
.
.
.
y
N 3
7
7
7
5
= 2
6
6
6
6
4 u
0 u
1
::: u
N
u
1 u
0 .
.
. .
.
.
.
.
. .
.
. .
.
.
u
1
u
N
::: u
1 u
0 3
7
7
7
7
5 2
6
6
6
4 w
0
w
1
.
.
.
w
N 3
7
7
7
5
: (5)
Assumingthe input provides afull rank systemthe Markovparameter
estimatesfw
j g
N
j=0
canbedetermined.
TherealizationstepdescribednextisbasedontheMarkovparameters
w
0
;:::;w
2n
and theassumption that (2)holds for asystemof order n.
As described in e.g. [20, 6, 1] the coeÆcients of and a are uniquely
determinedbythissetofMarkovparameters. Fromhereandon,wewill
without loss of generality consider the normalizedMarkov parameters,
wherew :=w =w ,sothatinparticularw =1. Thenitcanbeassumed
thatbotha(z)and(z)aremonic,i.e. a
0
=1and
0
=1. Multiplication
of both sides of (2) by a(z), and identi cation of the coeÆcients for
z 1
;:::;z n
leadtotheequationsystem
2
6
6
6
4 w
n+1
w
n+2
.
.
.
w
2n 3
7
7
7
5
= 2
6
6
6
6
4 w
1 w
2
::: w
n
w
2 w
3 .
. .
.
.
.
.
.
. .
. .
. .
.
w
2n 2
w
n
::: w
2n 2 w
2n 1 3
7
7
7
7
5 2
6
6
6
4 a
n
a
n 1
.
.
.
a
1 3
7
7
7
5
(6)
andidenti cationofthecoeÆcientsforz n
;:::;z;1give
2
6
6
6
4
1
2
.
.
.
n 3
7
7
7
5
= 2
6
6
6
4 w
1
w
2
.
.
.
w
n 3
7
7
7
5 +
2
6
6
6
6
4
1 0 ::: 0
w
1 1
.
.
. .
.
.
.
.
. .
.
. .
.
.
0
w
n 1
::: w
1 1
3
7
7
7
7
5 2
6
6
6
4 a
1
a
2
.
.
.
a
n 3
7
7
7
5
: (7)
IftheHankelmatrixin (6)isnonsingularthecoeÆcientsofthepolyno-
mial a(z)can be determined. In general, it cannot be guaranteed that
a(z) is a Schur polynomial, i.e. a polynomial with all roots inside the
unit circle. Since this is related to stability of the system, this is an
importantissuein manyapplications. Assuminga(z)isaSchurpolyno-
mialobtainedfrom(6),then(7)determinesapolynomial(z)suchthat
w(z)=(z)=a(z)interpolatestheMarkovparameters1;w
1
;:::;w
2n .
Further, for an arbitrary Schur polynomial a(z), (7) determines a
polynomial (z) such that w(z) = (z)=a(z) interpolates the Markov
parameters 1;w
1
;:::;w
n
. Actually, if we consider all rational transfer
functions w(z)=(z)=a(z)ofordern that interpolatestheMarkovpa-
rameters 1;w
1
;:::;w
n
, it follows from (7) that there is such a unique
transferfunction forevery choice of theSchurpolynomiala(z),and all
interpolating transferfunctions of order n canbeobtained in this way.
ThustheSchurpolynomialsa(z)parameterizeallsolutions. Thistypeof
parameterizationswillbeusedfrequentlyinthisthesis,andthereforethe
conceptofparameterizationisformalizedandexempli ed inSection5.
2.2 Stochastic Realization Theory
Instochasticrealization theory, theinputisassumedto beanunknown
sequence of (white) noise f:::;u
1
;u
0
;u
1
;:::g 2 U, and the only data
