Linköping Studies in Science and Technology. Thesis.
No. 1531
Estimation in Multivariate Linear
Models with Linearly Structured
Covariance Matrices
Joseph Nzabanita
Department of Mathematics
Linköping Studies in Science and Technology. Thesis. No. 1531
Estimation in Multivariate Linear Models with Linearly Structured Covariance Matrices Joseph Nzabanita joseph.nzabanita@liu.se www.mai.liu.se Mathematical Statistics Department of Mathematics Linköping University SE–581 83 Linköping Sweden LIU-TEK-LIC-2012:16 ISBN 978-91-7519-886-6 ISSN 0280-7971
Copyright c 2012 Joseph Nzabanita
Abstract
This thesis focuses on the problem of estimating parameters in multivariate linear mod-els where particularly the mean has a bilinear structure and the covariance matrix has a linear structure. Most of techniques in statistical modeling rely on the assumption that data were generated from the normal distribution. Whereas real data may not be exactly normal, the normal distributions serve as a useful approximation to the true distribution. The modeling of normally distributed data relies heavily on the estimation of the mean and the covariance matrix. The interest of considering various structures for the covari-ance matrices in different statistical models is partly driven by the idea that altering the covariance structure of a parametric model alters the variances of the model’s estimated mean parameters.
The extended growth curve model with two terms and a linearly structured covariance matrix is considered. In general there is no problem to estimate the covariance matrix when it is completely unknown. However, problems arise when one has to take into ac-count that there exists a structure generated by a few number of parameters. An estimation procedure that handles linear structured covariance matrices is proposed. The idea is first to estimate the covariance matrix when it should be used to define an inner product in a regression space and thereafter reestimate it when it should be interpreted as a dispersion matrix. This idea is exploited by decomposing the residual space, the orthogonal comple-ment to the design space, into three orthogonal subspaces. Studying residuals obtained from projections of observations on these subspaces yields explicit consistent estimators of the covariance matrix. An explicit consistent estimator of the mean is also proposed and numerical examples are given.
The models based on normally distributed random matrix are also studied in this the-sis. For these models, the dispersion matrix has the so called Kronecker product structure and they can be used for example to model data with spatio-temporal relationships. The aim is to estimate the parameters of the model when, in addition, one covariance matrix is assumed to be linearly structured. On the basis of n independent observations from a matrix normal distribution, estimation equations in a flip-flop relation are presented and numerical examples are given.
Populärvetenskaplig sammanfattning
Många statistiska modeller bygger på antagandet om normalfördelad data. Verklig da-ta kanske inte är exakt normalfördelad men det är i många fall en bra approximation. Normalfördelad data kan modelleras enbart genom dess väntevärde och kovariansmatris och det är därför ett problem av stort intresse att skatta dessa. Ofta kan det också vara intressant eller nödvändigt att anta någon struktur på både väntevärdet och/eller kovari-ansmatrisen.
Den här avhandlingen fokuserar på problemet att skatta parametrarna i multivariata linjära modeller, speciellt den utökade tillväxtkurvemodellen innehållande två termer och med en linjär struktur för kovariansmatrisen. I allmänhet är det inget problem att skatta kovariansmatrisen när den är helt okänd. Problem uppstår emellertid när man måste ta hänsyn till att det finns en struktur som genereras av ett färre antal parametrar. I många exempel kan maximum-likelihoodskattningar inte erhållas explicit och måste därför be-räknas med någon numerisk optimeringsalgoritm. Vi beräknar explicita skattningar som ett bra alternativ till maximum-likelihoodskattningarna. En skattningsprocedur som skat-tar kovariansmatriser med linjära strukturer föreslås. Tanken är att först skatta en kovari-ansmatris som används för att definiera en inre produkt i två steg, för att sedan skatta den slutliga kovariansmatrisen.
Även enkla tillväxtkurvemodeller med matrisnormalfördelning studeras i den här av-handlingen. För dessa modeller är kovariansmatrisen en Kroneckerprodukt och dessa mo-deller kan användas exempelvis för att momo-dellera data med spatio-temporala förhållande. Syftet är att skatta parametrarna i modellen när dessutom en av kovariansmatriserna antas följa en linjär strukturer. Med n oberoende observationer från en matrisnormalfördelning tas skattningsekvationer fram som löses med den så kallade flip-flop-algoritmen.
Acknowledgments
First of all, I would like to express my deep gratitude to my supervisors Professor Dietrich von Rosen and Dr. Martin Singull. Thank you Dietrich for guiding and encouraging me throughout my studies. The enthusiasm you constantly show makes you the right person to work with. Thank you Martin. You are always available to help me when I am in need and without your help this thesis would not be completed.
My deep gratitude goes also to Bengt Ove Turesson, to Björn Textorius, and to all the administrative staff of the Department of Mathematics for their constant help in different things.
I have also to thank my colleagues at the Department of Mathematics, especially Jolanta (my office mate), for making life easier during my studies.
My studies are sponsored through Sida/SAREC-Funded NUR-LiU Cooperation and all involved institutions are acknowledged.
Linköping, May 21, 2012 Joseph Nzabanita
Contents
1 Introduction 1 1.1 Background . . . 1 1.2 Outline . . . 2 1.2.1 Outline of Part I . . . 2 1.2.2 Outline of Part II . . . 2 1.3 Contributions . . . 3I
Multivariate Linear Models
5
2 Multivariate Distributions 7 2.1 Multivariate Normal distribution . . . 72.2 Matrix Normal Distribution . . . 8
2.3 Wishart Distribution . . . 10
3 Growth Curve Model and its Extensions 13 3.1 Growth Curve Model . . . 13
3.2 Extended Growth Curve Model . . . 15
3.3 Maximum Likelihood Estimators . . . 16
4 Concluding Remarks 21 4.1 Conclusion . . . 21
4.2 Further research . . . 22
xii Contents
II
Papers
27
A Paper A 29
1 Introduction . . . 32
2 Maximum likelihood estimators . . . 33
3 Estimators of the linearly structured covariance matrix . . . 36
4 Properties of the proposed estimators . . . 41
5 Numerical examples . . . 44
References . . . 48
B Paper B 51 1 Introduction . . . 54
2 Explicit estimators when Σ is unknown and has a linear structure and Ψ is known . . . 55
3 Estimators when Σ is unknown and has a linear structure and Ψ is unknown 56 4 Numerical examples: simulated study . . . 59
1
Introduction
T
HEgoals of statistical sciences are about planning experiments, setting up models to analyze experiments and to study properties of these models. Statistical applica-tion is about connecting statistical models to data. Statistical models are essentially for making predictions; they form the bridge between observed data and unobserved (future) outcomes (Kattan and Gönen, 2008). The general statistical paradigm constitutes of the following steps: (i) set up a model, (ii) evaluate the model via simulations or comparisons with data, (iii) if necessary refine the model and restart from step (ii), and (iv) accept and interpret the model. From this paradigm it is clear that the concept of statistical model lies in the heart of Statistics. In this thesis our focus is to linear models, a class of statistical models that play a key role in Statistics. If exact inference is not possible then at least a linear approximate approach can often be curried out (Kollo and von Rosen, 2005). In particular, we are concerned with the problem of estimation of parameters in multivariate linear models where the covariance matrices have linear structures.1.1
Background
The linear structures for the covariance matrices emerged naturally in statistical applica-tions and they are in the statistical literature for some years ago. These structures are, for example, the uniform structure (or intraclass structure), the compound symmetry struc-ture, the matrix with zeros, the banded matrix, the Toeplitz or circular Toeplitz, etc. The uniform structure, a linear covariance structure which consists of equal diagonal elements and equal off-diagonal elements, emerged for the first time in Wilks (1946) while deal-ing with measurements on k psychological tests. An extension of the uniform structure due to Votaw (1948) is the compound symmetry structure, which consists of blocks each having uniform structure. In Votaw (1948) one can find examples of psychometric and medical research problems where the compound symmetry covariance structure is appli-cable. The block compound symmetry covariance structure was discussed by Szatrowski
2 1 Introduction
(1982) who applied the model to the analysis of an educational testing problem. Ohlson et al. (2011b) proposed a procedure to obtain explicit estimator of a banded covariance matrix. The Toeplitz or circular Toeplitz discussed in Olkin and Press (1969) is another generalizations of the intraclass structure.
The interest of considering various structures for the covariance matrices in different statistical models is partly driven by the idea that altering the covariance structure of a parametric model alters the variances of the model’s estimated mean parameters (Lange and Laird, 1989). In this thesis we focus on the problem of estimation of parameters in multivariate linear models where particularly the mean has a bilinear structure as in the growth curve model (Pothoff and Roy, 1964) and the covariance matrix has a linear structure. The linear structured covariance matrix in the growth curve model have been studied in the statistical literature. For examples, Khatri (1973) considered the intraclass covariance structure, Ohlson and von Rosen (2010) studied the classical growth curve model, when the covariance matrix has some specific linear structure.
The main themes of this thesis are (i) to derive explicit estimators of parameters in the extended growth curve model with two terms, when the covariance matrix is linearly structured and (ii) to propose estimation equations of the parameters in the multivariate linear models with a mean which has a bilinear structure and a Kronecker covariance structure, where one of the covariance matrix has a linear structure.
1.2
Outline
This thesis consists of two parts and the outline is as follows.
1.2.1
Outline of Part I
In Part I the background and relevant results that are needed for an ease reading of this thesis are presented. Part I starts with Chapter 2 which gives a brief review on the multi-variate distributions. The main focus is to define the multimulti-variate normal distribution, the matrix normal distribution and the Wishart distribution. The maximum likelihood estima-tors in multivariate normal model and matrix normal model, for the unstructured cases, are given. Chapter 3 is devoted to the growth curve model and the extended growth curve model. The maximum likelihood estimators, for the unstructured cases, are presented. Part I ends with Chapter 4, which gives some concluding remarks and suggestions for further work.
1.2.2
Outline of Part II
Part II consists of two papers. Hereafter a short summary for each of the papers is pre-sented.
Paper A: Estimation of parameters in the extended growth curve
model with a linearly structured covariance matrix
In Paper A, the extended growth curve model with two terms and a linearly structured covariance matrix is studied. More specifically, the model considered is defined as
fol-1.3 Contributions 3
lows. Let X: p × n, Ai : p × qi, Bi : qi× ki, Ci : ki× n, r(C1) + p ≤ n, i = 1, 2,
C(C′2) ⊆ C(C′1), where r( · ) and C( · ) represent the rank and column space of a matrix, respectively. The extended growth curve model with two terms is given by
X= A1B1C1+ A2B2C2+ E,
where columns of E are assumed to be independently distributed as a multivariate nor-mal distribution with mean zero and a positive definite dispersion matrix Σ; i.e., E ∼ Np,n(0, Σ, In). The design matrices Aiand Ciare known matrices whereas matrices Bi and Σ are unknown parameter matrices. Moreover, we assume that the covariance matrix
Σ is linearly structured. In this paper an estimation procedure that handles linear
struc-tured covariance matrices is proposed. The idea is first to estimate the covariance matrix when it should be used to define an inner product in a regression space and thereafter reestimate it when it should be interpreted as a dispersion matrix. This idea is exploited by decomposing the residual space, the orthogonal complement to the design space, into three orthogonal subspaces. Studying residuals obtained from projections of observations on these subspaces yields explicit estimators of the covariance matrix. An explicit estima-tor of the mean is also proposed. Properties of these estimaestima-tors are studied and numerical examples are given.
Paper B: Estimation in multivariate linear models with Kronecker
product and linear structures on the covariance matrices
This paper deals with models based on normally distributed random matrices. More specifically the model considered is X ∼ Np,q(M, Σ, Ψ) with mean M, a p × q ma-trix, assumed to follow a bilinear structure, i.e., E[X] = M = ABC, where A and C
are known design matrices, B is unkown parameter matrix, and the dispersion matrix of
X has a Kronecker product structure, i.e., D[X] = Ψ ⊗ Σ, where both Ψ and Σ are
unknown positive definite matrices. The model may be used for example to model data with spatio-temporal relationships. The aim is to estimate the parameters of the model when, in addition, Σ is assumed to be linearly structured. In the paper, on the basis of
n independent observations on the random matrix X, estimation equations in a flip-flop
relation are presented and numerical examples are given.
1.3
Contributions
The main contributions of the thesis are as follows.
• In Paper A, we studied the extended growth curve model with two terms and a
lin-early structured covariance matrix. A simple procedure based on the decomposition of the residual space into three orthogonal subspaces and the study of the residuals obtained from projections of observations on these subspaces yields explicit and consistent estimators of the covariance matrix. An explicit unbiased estimator of the mean is also proposed.
• In Paper B, the multivariate linear model with Kronecker and linear structures on
ob-4 1 Introduction
simulations show that solving these equations with a flip-flop algorithm gives esti-mates which are in a well agreement with the true parameters.
Part I
2
Multivariate Distributions
T
HISchapter focuses on the normal distribution which is very important in statistical analyses. In particular, our interest here is to define the matrix normal distribution which will play a central role in this thesis. The Wishart distribution will also be looked at for easy reading of papers. The well known univariate normal distribution has been used in statistics for about two hundreds years and the multivariate normal distribution, understood as a distribution of a vector, has been also used for a long time (Kollo and von Rosen, 2005). Due to the complexity of data from various field of applied research, inevitable extensions of the multivariate normal distribution to the matrix normal distribu-tion or even more generalizadistribu-tion to multilinear normal distribudistribu-tion have been considered. The multilinear normal distribution will not be considered in this thesis. For more rele-vant results about multilinear normal distribution one can consult Ohlson et al. (2011a) and references cited therein.Before defining the multivariate normal distribution and the matrix normal distribution we remember that there are many ways of defining the normal distributions. In this thesis we will define the normal distributions via their density functions assuming that they exist.
2.1
Multivariate Normal distribution
Definition 2.1 (Multivariate normal distribution). A random vector x: p × 1 is
mul-tivariate normally distributed with mean vector µ: p × 1 and positive definite covariance
matrix Σ: p × p if its density is
f(x) = (2π)−p2|Σ|−12e−12tr{Σ−1(x−µ)(x−µ)′}, (2.1)
where| · | and tr denote the determinant and the trace of a matrix respectively. We usually
use the notation x∼ Np(µ, Σ).
The multivariate normal model x ∼ Np(µ, Σ), where µ and Σ are unknown pa-rameters, is used in the statistical literature for a long time. To find estimators of the
8 2 Multivariate Distributions
parameters, the method of maximum likelihood is often used. Let a random sample of
n observation vectors x1, x2, . . . , xncome from the multivariate normal distribution, i.e.
xi∼ Np(µ, Σ). The xi’s constitute a random sample and the likelihood function is given by the product of the densities evaluated at each observation vector
L(x1, x2, . . . , xn, µ, Σ) = n Y i=1 f(xi, µ, Σ) = n Y i=1 (2π)−p2|Σ|−12e− 1 2tr{Σ−1(xi−µ)(xi−µ)′} = (2π)−pn2 |Σ|−n2e− Pn i=1(xi−µ)′Σ−1(xi−µ)/2.
The maximum likelihood estimators (MLEs) of µ and Σ resulting from the maximization of this likelihood function, for more details see for example Johnson and Wichern (2007), are respectively b µ = 1 n n X i=1 xi = 1 nX1n, b Σ = 1 nS, where S= n X i=1 (xi−µ)(xb i−µ)b ′= X(In− 1 n1n1 ′ n)X′,
X= (x1, x2, . . . , xn), 1nis the n−dimensional vector of 1s, and Inis the n× n identity matrix.
2.2
Matrix Normal Distribution
Definition 2.2 (Matrix normal distribution). A random matrix X : p × q is matrix
normally distributed with mean M : p × q and positive definite covariance matrices Σ: p × p and Ψ : q × q if its density is
f(X) = (2π)−pq2 |Σ|−q2|Ψ|− p 2e−
1
2tr{Σ−1(X−M)Ψ−1(X−M)′}. (2.2)
The model based on the matrix normally distributed is usually denoted as
X∼ Np,q(M, Σ, Ψ), (2.3) and it can be shown that X∼ Np,q(M, Σ, Ψ) means the same as
vecX ∼ Npq(vecM, Ψ ⊗ Σ), (2.4) where⊗ denotes the Kronecker product. Since by definition of the dispersion matrix of X is D[X] = D[vecX], we get D[X] = Ψ ⊗ Σ. For the interpretation we note that Ψ
2.2 Matrix Normal Distribution 9
describes the covariances between the columns of X. These covariances will be the same for each row of X. The other covariance matrix Σ describes the covariances between the rows of X which will be the same for each column of X. The product Ψ⊗ Σ takes
into account the covariances between columns as well as the covariances between rows. Therefore, Ψ⊗ Σ indicates that the overall covariance consists of the products of the
covariances in Ψ and in Σ, respectively, i.e.,Cov[xij, xkl] = σikψjl, where X = (xij),
Σ= (σik) and Ψ = (ψjl).
The following example shows one possibility of how a matrix normal distribution may arise.
Example 2.1
Let x1, . . . , xn be an independent sample of n observation vectors from a multivariate normal distribution Np(µ, Σ) and let the observation vectors xibe the columns in a ma-trix X= (x1, x2, . . . , xn). The distribution of the vectorization of the sample observation matrixvecX is given by
vecX = (x′1, x′2, . . . , x′n) ′
∼ Npn(1n⊗ µ, Ω) ,
where Ω= In⊗ Σ, 1n is the n−dimensional vector of 1s, and Inis the n× n identity matrix. This is written as
X∼ Np,n(M, Σ, In) ,
where M= µ1′ n.
The models (2.3) and (2.4) have been considered in the statistical literature. For exam-ple Dutilleul (1999), Roy and Khattree (2005) and Lu and Zimmerman (2005) considered the model (2.4), and to obtain MLEs these authors solved iteratively the usual likelihood equations, one obtained by assuming that Ψ is given and the other obtained by assuming that Σ is given, by what was called the flip-flop algorithm in Lu and Zimmerman (2005). Let a random sample of n observation matrices X1, X2, . . . , Xnbe drawn from the matrix normal distribution, i.e. Xi ∼ Np(M, Σ, Ψ). The likelihood function is given by the product of the densities evaluated at each observation matrix as it was for the multivariate case. The log-likelihood, ignoring the normalizing factor, is given by
ln L(X, M, Σ, Ψ) = −qn 2 ln |Σ| − pn 2 ln |Ψ| −1 n X tr{Σ−1(Xi− M)Ψ−1(Xi− M)′}.
10 2 Multivariate Distributions
The likelihood equations for likelihood estimators are given by (Dutilleul, 1999)
c M = 1 n n X i=1 Xi= X, (2.5) b Σ = 1 nq n X i=1 (Xi− cM) bΨ−1(Xi− cM)′, (2.6) b Ψ = 1 np n X i=1 (Xi− cM)′Σb−1(Xi− cM), (2.7) There is no explicit solutions to these equations and one must rely on an iterative algo-rithm like the flip-flop algoalgo-rithm (Dutilleul, 1999). Srivastava et al. (2008) pointed out that the estimators found in this way are not uniquely determined. Srivastava et al. (2008) showed that solving these equations with additional estimability conditions, using the flip-flop algorithm, the estimates in the algorithm converge to the unique maximum likelihood estimators of the parameters.
The model (2.3), where the mean has a bilinear structure was considered by Srivastava et al. (2008). In Paper B, we consider the problem of estimating the parameters in the model (2.3) where the mean has a bilinear structure (see the mean structure in the growth curve model Section 3.1) and, in addition, the covariance matrix Σ is assumed to be linearly structured.
2.3
Wishart Distribution
In this section we present the definition and some properties of another important distri-bution which belongs to the class of matrix distridistri-butions, the Wishart distridistri-bution. First derived by Wishart (1928), the Wishart distribution is usually regarded as a multivari-ate analogue of the chi-square distribution. There are many ways to define the Wishart distribution and here we adopt the definition by Kollo and von Rosen (2005).
Definition 2.3 (Wishart distribution). The matrix W : p × p is said to be Wishart
distributed if and only if W = XX′ for some matrix X, where X ∼ N
p,n(M, Σ, I),
Σ≥ 0. If M = 0, we have a central Wishart distribution which will be denoted W ∼ Wp(Σ, n), and if M 6= 0, we have a non-central Wishart distribution which will be denoted Wp(Σ, n, ∆), where ∆ = MM′.
The first parameter Σ is usually supposed to be unknown. The second parameter n, which stands for the degree of freedom is usually considered to be known. The third parameter ∆, which is used in the central Wishart distribution, is called the non-centrality parameter.
2.3 Wishart Distribution 11
The following theorem contains some properties of the Wishart distribution which are to be used in the papers.
Theorem 2.1
(i) Let W1∼ Wp(Σ, n, ∆1) be independent of W2∼ Wp(Σ, m, ∆2). Then
W1+ W2∼ Wp(Σ, n + m, ∆1+ ∆2).
(ii) Let X∼ Np,n(M, Σ, Ψ), where C(M′) ⊆ C(Ψ). Put W = XΨ−X′. Then
W∼ Wp(Σ, r(Ψ), ∆),
where ∆= MΨ−M′.
(iii) Let W∼ Wp(Σ, n, ∆) and A ∈Rq×p. Then
AWA′ ∼ Wp(AΣA′, n, A∆A′).
(iv) Let X ∼ Np,n(M, Σ, I) and Q : n × n be symmetric. Then XQX′ is Wishart
distributed if and only if Q is idempotent.
(v) Let X ∼ Np,n(M, Σ, I) and Q : n × n be symmetric and idempotent, so that
MQ= 0. Then XQX′ ∼ W
p(Σ, r(Q)).
(vi) Let X∼ Np,n(M, Σ, I), Q1: n × n and Q2: n × n be symmetric. Then XQ1X′
and XQ2X′are independent if and only if Q1Q2= Q2Q1= 0.
The proofs of these results can be found, for example, in Kollo and von Rosen (2005).
Example 2.2
In Section 2.1, the MLEs of µ and Σ in the multivariate normal model x ∼ Np(µ, Σ) were given. These are respectively
b µ = 1 nX1n, b Σ = 1 nS, where S= X(In− 1 n1n1 ′ n)X′,
X= (x1, x2, . . . , xn), 1nis the n−dimensional vector of 1s, and Inis the n× n identity matrix.
It is easy to show that the matrix Q= In−n11n1′nis idempotent and r(Q) = n − 1. Thus,
S∼ Wp(Σ, n − 1).
3
Growth Curve Model and its
Extensions
T
HEgrowth curve analysis is a topic with many important applications within medicine, natural sciences, social sciences, etc. Growth curve analysis has a long history and two classical papers are Box (1950) and Rao (1958). In Roy (1957) or Anderson (1958), one considered the MANOVA modelX= BC + E, (3.1)
where X : p × n, B : p × k, C : k × n, E ∼ Np,n(0, Σ, I). The matrix C called the between-individuals design matrix is known, B and the positive definite matrix Σ are unknown parameter matrices.
3.1
Growth Curve Model
In 1964 the well known paper by Pothoff and Roy (1964) extended the MANOVA model (3.1) to the model which was later termed the growth curve model.
Definition 3.1 (Growth curve model). Let X: p × n, A : p × q, q ≤ p, B : q × k, C : k× n, r(C) + p ≤ n, where r( · ) represents the rank of a matrix. The Growth Curve
Model is given by
X= ABC + E, (3.2)
where columns of E are assumed to be independently distributed as a multivariate nor-mal distribution with mean zero and a positive definite dispersion matrix Σ; i.e. E ∼ Np,n(0, Σ, In).
The matrices A and C, often called respectively within-individuals and between-individuals design matrices, are known matrices whereas matrices B and Σ are unknown parameter matrices.
14 3 Growth Curve Model and its Extensions
The paper by Pothoff and Roy (1964) is often considered to be the first where the model was presented. Several prominent authors wrote follow-up papers, e.g. Rao (1965) and Khatri (1966). Notice that the growth curve model is a special case of the matrix normal model where the mean has a bilinear structure. Therefore, we may use the notation
X∼ Np,n(ABC, Σ, I).
Also, it is worth noting that the MANOVA model with restrictions
X = BC+ E, (3.3)
GB = 0
is equivalent to the Growth Curve model. GB= 0 is equivalent to B = (G′)oΘ, where
(G′)ois any matrix spanning the orthogonal complement to the space generated by the columns of G′. Plugging(G′)oΘ in (3.3) gives
X= (G′)oΘC+ E,
which is identical to the growth curve model (3.2).
Example 3.1: Potthoff & Roy (1964) dental data
Dental measurements on eleven girls and sixteen boys at four different ages (t1= 8, t2=
10, t3= 12, and t4 = 14) were taken. Each measurement is the distance, in millimeters, from the center of pituitary to pteryo-maxillary fissure. These data are presented in Table 1 and plotted in Figure 3.1. Suppose linear growth curves describe the mean growth for
Table 3.1: Dental data
id gender t1 t2 t3 t4 id gender t1 t2 t3 t4 1 F 21.0 20.0 21.5 23.0 12 M 26.0 25.0 29.0 31.0 2 F 21.0 21.5 24.0 25.5 13 M 21.5 22.5 23.0 26.0 3 F 20.5 24.0 24.5 26.0 14 M 23.0 22.5 24.0 27.0 4 F 23.5 24.5 25.0 26.5 15 M 25.5 27.5 26.5 27.0 5 F 21.5 23.0 22.5 23.5 16 M 20.0 23.5 22.5 26.0 6 F 20.0 21.0 21.0 22.5 17 M 24.5 25.5 27.0 28.5 7 F 21.5 22.5 23.0 25.0 18 M 22.0 22.0 24.5 26.5 8 F 23.0 23.0 23.5 24.0 19 M 24.0 21.5 24.5 25.5 9 F 20.0 21.0 22.0 21.5 20 M 23.0 20.5 31.0 26.0 10 F 16.5 19.0 19.0 19.5 21 M 27.5 28.0 31.0 31.5 11 F 24.5 25.0 28.0 28.0 22 M 23.0 23.0 23.5 25.0 23 M 21.5 23.5 24.0 28.0 24 M 17.0 24.5 26.0 29.5 25 M 22.5 25.5 25.5 26.0 26 M 23.0 24.5 26.0 30.0 27 M 22.0 21.5 23.5 25.0
both girls and boy. Then we may use the growth curve model
3.2 Extended Growth Curve Model 15 1 1.5 2 2.5 3 3.5 4 16 18 20 22 24 26 28 30 32 Age Growth measurements Girls profile Boys profile
Figure 3.1: Growth profiles plot of Potthoff and Roy (1964) dental data In this model, the observation matrix is X = (x1, x1, . . . , x27), in which eleven first columns correspond to measurements on girls and sixteen last columns correspond to measurements on boys. The design matrices are
A′ = 1 1 1 1 8 10 12 14 , C= 1′11⊗ 1 0 : 1′16⊗ 0 1 ,
and B is the unknown parameter matrix and Σ is the unknown positive definite covariance matrix.
3.2
Extended Growth Curve Model
One of limitations of the growth curve model is that different individuals should follow the same growth profile. If this does not hold there is a way to extend the model. A natural extension of the growth curve model, introduced by von Rosen (1989), is the following
Definition 3.2 (Extended growth curve model). Let X: p×n, Ai: p×qi, Bi: qi×ki,
Ci : ki× n, r(C1) + p ≤ n, i = 1, 2, . . . , m, C(C′i) ⊆ C(C′i−1), i = 2, 3, . . . , m, where r( · ) and C( · ) represent the rank and column space of a matrix respectively. The
Extended Growth Curve Model is given by
X=
m
X
i=1
AiBiCi+ E,
where columns of E are assumed to be independently distributed as a multivariate nor-mal distribution with mean zero and a positive definite dispersion matrix Σ; i.e. E ∼
16 3 Growth Curve Model and its Extensions
The matrices Ai and Ci, often called design matrices, are known matrices whereas matrices Bi and Σ are unknown parameter matrices. As for the growth curve model the notation X∼ Np,n m X i=1 AiBiCi, Σ, I !
may be used for the extended growth curve model. The only difference with the growth curve model in Definition 3.1 is the presence of a more general mean structure. When
m = 1, the model reduces to the growth curve model. The model without subspace
conditions was considered before by Verbyla and Venables (1988) under the name of
sum of profiles model. Also observe that the subspace conditionsC(C′
i) ⊆ C(C′i−1),
i= 2, 3, . . . , m may be replaced by C(Ai) ⊆ C(Ai−1), i = 2, 3, . . . , m. This problem was considered for example by Filipiak and von Rosen (2011) for m= 3.
In Paper A, we consider the problem of estimating parameters in the extended growth curve model with two terms (m= 2), where the covariance matrix Σ is linearly
struc-tured.
Example 3.2
Consider again Potthoff & Roy (1964) classical dental data. But now assume that for both girls and boys we have a linear growth component but additionally for the boys there also exists a second order polynomial structure. Then we may use the extended growth curve model with two terms
X∼ Np,n(A1B1C1+ A2B2C2, Σ, I), where A′1= 1 1 1 1 8 10 12 14 , C1= 1′11⊗ 1 0 : 1′16⊗ 0 1 A′2= 82 102 122 142 , C 2= (0′11: 1′16), are design matrices and B1=
β11 β12
β21 β22
and B2= (β32) are parameter matrices and
Σ is the same as in Example 3.1.
3.3
Maximum Likelihood Estimators
The maximum likelihood method is one of several approaches used to find estimators of parameters in the growth curve model. The maximum likelihood estimators of parameters in the growth curve model have been studied by many authors, see for instance (Srivastava and Khatri, 1979) and (von Rosen, 1989). For the extended growth curve model as in Definition 3.2 an exhaustive description of how to get those estimators can be found in Kollo and von Rosen (2005). Here we present some important results from which the main ideas discussed in Paper A are derived. The following results due to von Rosen (1989) gives the MLEs of parameters in the extended growth curve model.
3.3 Maximum Likelihood Estimators 17
Theorem 3.1
Consider the extended growth curve model as in Definition 3.2. Let
Pr = Tr−1Tr−2× · · · × T0, T0= I, r = 1, 2, . . . , m + 1, Ti = I− PiAi(A′iP′iS−1i PiAi)−A′iP′iS−1i , i= 1, 2, . . . , m, Si = i X j=1 Kj, i= 1, 2, . . . , m, Kj = PjXPC′ j−1(I − PC′j)PC′j−1X ′P′ j, C0= I, PC′ j = C ′ j(CjC′j)−Cj.
Assume that S1is positive definite.
(i) The representations of maximum likelihood estimators of Br, r= 1, 2, . . . , m and
Σ are b Br = (A′rP′rS−1r PrAr)−A′rP′rS−1r (X − m X i=r+1 AiBbiCi)C′r(CrC′r)− +(A′rP′r)oZr1+ A′rP′rZr2Cor ′ , n bΣ = (X − m X i=1 AiBbiCi)(X − m X i=1 AiBbiCi)′ = Sm+ Pm+1XC′m(CmC′m)−CmX′Pm+1,
where Zr1and Zr2are arbitrary matrices andPmi=m+1AiBbiCi= 0.
(ii) For the estimators bBi,
Pr m X i=r AiBbiCi= m X i=r (I − Ti)XC′i(CiC′i)−Ci.
The notation Costands for any matrix of full rank spanningC(C)⊥, and G−denotes an
arbitrary generalized inverse in the sense that GG−G= G.
A useful results is the corollary of this theorem when r= 1, which gives the estimated
mean structure. Corollary 3.1 [ E[X] =Xm i=1AiBbiCi= m X i=1 (I − Ti)XC′i(CiC′i)−Ci.
Another consequence of Theorem 3.1 that is considered in Paper A, corresponds to the case of m= 2. Set m = 2 in the extended growth curve model of Definition 3.2. Then,
the maximum likelihood estimators for the parameter matrices B1and B2are given by
b
B2 = (A′2P′2S−12 P2A2)−A′2P′2S2−1XC′2(C2C′2)−+ (A2′P2)oZ21+ A′2Z22Co ′ 2
18 3 Growth Curve Model and its Extensions
where
S1 = X I− C′1(C1C′1)−C1X′,
P2 = I− A1(A′1S1−1A1)−A′1S−11 ,
S2 = S1+ P2XC′1(C1C′1)−C1 I− C′2(C2C′2)−C2C′1(C1C′1)−C1X′P′2,
Zklare arbitrary matrices.
Assuming that matrices Ai’s, Ci’s are of full rank and thatC(A1) ∩ C(A2) = {0}, the unique maximum likelihood estimators are
b
B2 = (A′2P′2S−12 P2A2)−1A′2P′2S−12 XC2′(C2C′2)−1,
b
B1 = (A′1S1−1A1)−1A′1S1−1(X − A2Bb2C2)C′1(C1C′1)−1.
Obviously, under general settings, the maximum likelihood estimators bB1and bB2are not unique due to the arbitrariness of matrices Zkl. However, it is worth noting that the estimated mean
[
E[X] = A1Bb1C1+ A2Bb2C2 is always unique and therefore bΣ given by
n bΣ= (X − A1Bb1C1− A2Bb2C2)(X − A1Bb1C1− A2Bb2C2)′ is also unique.
Example 3.3: Example 3.2 continued
Consider again Potthoff & Roy (1964) classical dental data and the model of Example 3.2. Then, the maximum likelihood estimates of parameters are
b B1 = 20.2836 21.9599 0.9527 0.5740 , bB2= (0.2006), b Σ = 5.0272 2.5066 3.6410 2.5099 2.5066 3.8810 2.6961 3.0712 3.6410 2.6961 6.0104 3.8253 2.5099 3.0712 3.8253 4.6164 .
The estimated mean growth curves, plotted in Figure 3.2, for girls and boys are respec-tively
b
µg(t) = 20.2836 + 0.9527 t,
b
3.3 Maximum Likelihood Estimators 19 1 1.5 2 2.5 3 3.5 4 16 18 20 22 24 26 28 30 32 Age Growth Girls profile Boys profile
4
Concluding Remarks
T
HISchapter is reserved to the summary of the thesis and suggestions for furtherre-search.
4.1
Conclusion
The problem of estimating parameters in different statistical models is in the center of the statistical sciences. The main theme of this thesis is about the estimation of param-eters in multivariate linear models where the covariance matrices have linear structures. The linear structures for the covariance matrices occur naturally in statistical applications and many authors have been interested in those structures. It is well known that normal distributed data can be modeled only by its mean and covariance matrix. Moreover, the inference on the mean parameters heavily depends on the estimated covariance matrix and the dispersion matrix for the estimator of the mean is a function of it. Hence, it is believed that altering the covariance structure of a parametric model alters the variances of the model’s estimated mean parameters. Therefore, considering various structures for the covariance matrices in different statistical models is a problem of great interest.
In Paper A, we study the extended growth curve model with two terms and a linearly structured covariance matrix. An estimation procedure that handles linear structured co-variance matrices was proposed. The idea is first to estimate the coco-variance matrix when it should be used to define an inner product in the regression space and thereafter rees-timate it when it should be interpreted as a dispersion matrix. This idea is exploited by decomposing the residual space, the orthogonal complement to the design space, into three orthogonal subspaces. Studying residuals obtained from projections of observations on these subspaces yields explicit consistent estimators of the covariance matrix. An ex-plicit consistent estimator of the mean was also proposed. Numerical simulations show that the estimates of the linearly structured covariance matrices are very close to the true covariance matrices. However, for the banded matrix structure, it was noted that the
es-22 4 Concluding Remarks
timates of the covariance matrix may not be positive definite for small n whereas it is always positive definite for the circular Toeplitz structure.
In Paper B, the models based on normally distributed random matrix are studied. For these models, the dispersion matrix has the so called Kronecker product structure and they can be used for example to model data with spatio-temporal relationships. The aim is to estimate the parameters of the model when, in addition, one covariance matrix is assumed to be linearly structured. On the basis of n independent observations from a matrix normal distribution, estimation equations in a flip-flop relation are presented. Numerical simulations show that the estimates of parameters are in a well agreement with the true parameters.
4.2
Further research
At the completion of this thesis some points have to be pointed out as suggestions for further work.
• The proposed estimators in Paper A have good properties like unbiasedness and/or
consistency. However, to be more useful their other properties (e.g. their distri-butions) have to be studied. Also, more studies on the positive definiteness of the estimates for the covariance matrix is of interest.
• In Paper B, numerical simulations showed that the proposed algorithm produces
estimates of parameters that are in a well agreement with the true values. The algorithm was established in a fair heuristic manner and more rigorous studies are needed.
• Application of procedures developed in Paper A and Paper B to concrete real data
Bibliography
Anderson, T. (1958). An Introduction to Multivariate Statistical Analysis. Wiley, New York, USA.
Box, G. E. P. (1950). Problems in the analysis of growth and wear curves. Biometrics, 6:362–389.
Dutilleul, P. (1999). The MLE algorithm for the matrix normal distribution. Journal of statistical Computation Simulation, 64:105–123.
Filipiak, K. and von Rosen, D. (2011). On MLEs in an extended multivariate linear growth curve model.Metrika, doi: 10.1007/s00184-011-0368-2.
Johnson, R. and Wichern, D. (2007). Applied Multivariate Statistical Analysis. Pearson Education International, USA.
Kattan, W. M. and Gönen, M. (2008). The prediction philosophy in statistics. Urologic Oncology: Seminars and Original Investigations, 26:316–319.
Khatri, C. G. (1966). A note on a manova model applied to problems in growth curve.
Annals Institute Statistical Mathematics, 18:75–86.
Khatri, C. G. (1973). Testing some covariance structures under a growth curve model.
Journal of Multivariate Analysis, 3:102–116.
Kollo, T. and von Rosen, D. (2005). Advanced Multivariate Statistics with Matrices. Springer, Dordrecht, The Netherlands.
Lange, N. and Laird, N. M. (1989). The effect of covariance structure on variance es-timation in balanced growth-curve models with random parameters. Journal of the American Statistical Association, pages 241–247.
24 Bibliography
Lu, N. and Zimmerman, L. D. (2005). The likelihood ratio test for a separable covariance matrix. Statistics Probability Letters, 73:449–457.
Ohlson, M., Ahmad, M. R., and von Rosen, D. (2011a). The multilinear normal dis-tribution: Introduction and some basic properties. Journal of Multivariate Analysis, doi:10.1016/j.jmva.2011.05.015.
Ohlson, M., Andrushchenko, Z., and von Rosen, D. (2011b). Explicit estimators under m-dependence for a multivariate normal distribution. Annals of the Institute of Statistical Mathematics, 63:29–42.
Ohlson, M. and von Rosen, D. (2010). Explicit estimators of parameters in the growth curve model with linearly structured covariance matrices.Journal of Multivariate Anal-ysis, 101:1284–1295.
Olkin, I. and Press, S. (1969). Testing and estimation for a circular stationary model.The Annals of Mathematical Statistics, 40(4):1358–1373.
Pothoff, R. and Roy, S. (1964). A generalized multivariate analysis of variance model useful especially for growth curve problems. Biometrika, 51:313–326.
Rao, C. (1958). Some statistical metods for comparisons of growth curves. Biometrics, 14:1–17.
Rao, C. (1965). The theory of least squares when the parameters are stochastic and its application to the analysis of growth curves. Biometrika, 52:447–458.
Roy, A. and Khattree, R. (2005). On implementation of a test for Kronecker product covariance structure for multivariate repeated measures data. Statistical Methodology, 2:297–306.
Roy, S. (1957).Some Aspects of Multivariate Analysis. Wiley, New York, USA. Srivastava, M. S. and Khatri, C. (1979).An Introduction to Multivariate Statistics.
North-Holland, New York, USA.
Srivastava, M. S., von Rosen, T., and von Rosen, D. (2008). Models with Kronecker prod-uct covariance strprod-ucture: estimation and testing. Mathematical Methods of Statistics, 17:357–370.
Szatrowski, T. H. (1982). Testing and estimation in the block compound symmetry prob-lem. Journal of Educational Statistics, 7(1):3–18.
Verbyla, A. and Venables, W. (1988). An extension of the growth curve model.
Biometrika, 75:129–138.
von Rosen, D. (1989). Maximum likelihood estimators in multivariate linear normal models. Journal of Multivariate Analysis, 31:187–200.
Votaw, D. F. (1948). Testing compound symmetry in a normal multivariate distribution.
Bibliography 25
Wilks, S. S. (1946). Sample criteria for testing equality of means, equality of variances, and equality of covariances in a normal multivariate distribution.The Annals of Math-ematical Statistics, 17(3):257–281.
Wishart, J. (1928). The generalized product moment distribution in samples from a nor-mal multivariate population.Biometrika, 20 A:32–52.
