More on the Kronecker Structured Covariance
Matrix
Martin Ohlson, M. Rauf Ahmad and Dietrich von Rosen
Linköping University Post Print
N.B.: When citing this work, cite the original article.
This is an electronic version of an article published in:
Martin Ohlson, M. Rauf Ahmad and Dietrich von Rosen, More on the Kronecker Structured Covariance Matrix, 2012, Communications in Statistics - Theory and Methods, (41), 13-14, 2512-2523.
Communications in Statistics - Theory and Methods is available online at informaworldTM:
http://dx.doi.org/10.1080/03610926.2011.615971
Copyright: Taylor & Francis: STM, Behavioural Science and Public Health Titles / Taylor & Francis
http://www.tandf.co.uk/journals/default.asp
Postprint available at: Linköping University Electronic Press
MORE ON THE KRONECKER STRUCTURED COVARIANCE MATRIX
Martin Ohlson∗, M. Rauf Ahmad∗ and Dietrich von Rosen† ∗
∗Department of Mathematics,
Link¨oping University,
SE–581 83 Link¨oping, Sweden. E-mail: martin.ohlson@liu.se.
†Energy and Technology,
Swedish University of Agricultural Sciences, SE–750 07 Uppsala, Sweden.
Key Words: Kronecker product structure, Separable covariance, Double separable covari-ance, Maximum likelihood estimators.
ABSTRACT
In this paper, the multivariate normal distribution with a Kronecker product structured covariance matrix is studied. Particularly focused is the estimation of a Kronecker struc-tured covariance matrix of order three, the so called double separable covariance matrix. The suggested estimation generalizes the procedure proposed by Srivastava et al. (2008) for a sep-arable covariance matrix. The restrictions imposed by separability and double separability are also discussed.
1. INTRODUCTION
In this paper we consider estimation of a Kronecker structured covariance matrix of order three. The main goal is to extend the estimation procedure, suggested by Srivastava et al. (2008), for the matrix normal distribution vecX ∼ Npq(vecM , Ψ ⊗ Σ) to the case where
vecX ∼ Npqr(vecM, Θ ⊗ Ψ ⊗ Σ), with some vectorization, vecX , of the third order tensor
X = (xijk) : p × q × r (see Section 2), and where ⊗ denotes the Kronecker product. We will
say that the covariance matrix D(vecX ) = Θ ⊗ Ψ ⊗ Σ is double separable (or three-factor separable) compared to the separable covariance matrix D(vecX) = Ψ ⊗ Σ. The Kronecker
product restrictions make the family of densities to be curved, i.e., it belongs to the curved exponential family
In a recent paper Hoff (2011) discusses separability of higher order than double separa-bility, but with focus on Bayesian estimation, also Roy and Leiva (2011) have studied doubly exchangeable linear models, which are suitable for three-level multivariate data, and closely related to double separability. Doubly exchangeable covariance structure assumes a block circulant covariance structure consisting of three unstructured covariance matrices for three multivariate levels.
Several authors have considered estimation and testing under the separability assump-tion, see for example Naik and Rao (2001); Roy and Khattree (2003); Lu and Zimmerman (2005); Mitchell et al. (2005, 2006); Srivastava et al. (2008). Srivastava et al. (2008) discussed estimability of the paramters under the separability assumption. From the likelihood func-tion, constructed of independent observation matrices, Srivastava et al. (2008) proved that the maximum likelihood estimates under the restriction ψqq = 1, where Ψ = (ψij) : q × q,
are found by an iterative flip-flop algorithm. They also showed that the likelihood equations provide unique estimators. A similar algorithm has been suggested by Mardia and Goodall (1993); Dutilleul (1999); Brown et al. (2001); Lu and Zimmerman (2005); Roy (2007), but without the restriction ψqq = 1.
In many applications, different structures of the covariance matrices have been discussed. Roy and Khattree (2005a) and Srivastava et al. (2008), for example, consider the intraclass covariance structure, whereas an autoregressive structure is discussed in Roy and Khattree (2005b).
There have also been attempts to study a structure on the mean. Srivastava et al. (2009), for example, assume the growth curve model for the mean M = ABC, where A : p × s and C : t × q are known design matrices and B : s × t is the parameter matrix. Under the restriction ψqq = 1 and some full rank assumption, the unique estimators for B, Σ, Ψ are
derived.
matrix of the multivariate normal distribution. For the estimation of this doubly separable covariance matrix, we present a generalization of the estimation of separable covariance matrix given in Srivastava et al. (2008). The rest of the article is organized as follows. We begin, in the next section, with the normal distribution for the third order tensor X = (xijk) : p × q × r, mainly focusing on the vectorization of the third order tensor, and on the
permutation of this vectorization to present the data in a proper way. Section 3 is dedicated to the estimation procedure of the covariance matrix. Finally, in Section 4, the restrictions imposed on the matrices Θ, Ψ and Σ, similar to ψqq = 1, by the Kronecker product structure,
are discussed. 2. MODEL
Let X be a tensor of order three, with the dimension p, q and r in the x, y and z direction, respectively, see Figure 1. If r = 1 we have a special case with the tensor equal to a p × q matrix. For such a matrix X = (x1, . . . , xq) : p × q, the standard way to vectorize is as
vecX = (x01, . . . , x0q)0. k = 1, . . . , r x11k . . . x1qk .. . . .. ... xp1k . . . xpqk X =
Figure 1: The box visualizes a three dimensional data set as a third order tensor. The extension of this vectorization to a three dimensional tensor X can be formulated in several ways. We use the following definition, as given in Kolda and Bader (2009). Definition 1 Let X = (xijk) : p × q × r be a three dimensional tensor. Define the
vector-ization of X as vecX = p X i=1 q X j=1 r X k=1 xijke3k⊗ e 2 j ⊗ e 1 i,
where e3
k, e2j and e1i are the unit basis vectors of size r, q and p, respectively.
We shall assume that the vectorization of X follows a multivariate normal distribution with a double separable covariance matrix
D(vecX ) = Θ ⊗ Ψ ⊗ Σ,
where Σ : p×p, Ψ : q ×q and Θ : r ×r are assumed to be positive definite. This structure is a generalization of the separable covariance matrix discussed in, for example, Galecki (1994); Dutilleul (1999); Roy and Khattree (2003); Lu and Zimmerman (2005); Srivastava et al. (2008).
Following Kollo and von Rosen (2005, Definition 2.2.3), we can write the double separable model for X (or vecX ) as
vecX =X ijk µijke3k⊗ e 2 j ⊗ e 1 i + X ijk X i0j0k0 ςii0τjj0ϑkk0ui0j0k0e3k⊗ e2j ⊗ e1i,
where M = (µijk) : p × q × r, Σ = ςς0, Ψ = τ τ0 and Θ = ϑϑ0, and where ui0j0k0 ∼ N (0, 1),
iid (independent and identically distributed). The density of X can now be written as
(2π)−pqr/2|Θ|−pq/2|Ψ|−pr/2|Σ|−qr/2exp −1 2vec 0 (X − M)(Θ ⊗ Ψ ⊗ Σ)−1vec(X − M) , and is denoted as X ∼ Np,q,r(M, Σ, Ψ, Θ) . (1)
For more details about the multilinear normal distribution (1), see Kollo and von Rosen (2005, p 215). Furthermore, the tensor in Figure 1 can be viewed from different directions by unfolding it into different matrices. Using matricization in three different modes, as given in Kolda and Bader (2009), we have the following matrices (which are just the transposes of the matrices in Kolda and Bader (2009))
X =X ijk xijk(e3k⊗ e 2 j)(e 1 i) 0 : (qr) × p, (2) Y = X ijk xijk(e1i ⊗ e 3 k)(e 2 j) 0 : (pr) × q, (3) Z =X ijk xijk(e2j ⊗ e 1 i)(e 3 k) 0 : (pq) × r. (4)
Using these matrices, and the fact that vec(ab0) = b ⊗ a, ∀ a ∈ Rp, b ∈ Rq, we get
vecX = vecZ = Kqr,pvecX = Kqr,pKpr,qvecY ,
where Kp,q : pq × pq is the commutation matrix. Further, using properties of the
commuta-tion matrix, we have
vec0X (Θ ⊗ Ψ ⊗ Σ)−1vecX = vec0Z(Θ ⊗ Ψ ⊗ Σ)−1vecZ = trΘ−1Z0(Ψ ⊗ Σ)−1Z = vec0X(Σ ⊗ Θ ⊗ Ψ)−1vecX = trΣ−1X0(Θ ⊗ Ψ)−1X (5) = vec0Y (Ψ ⊗ Σ ⊗ Θ)−1vecY = trΨ−1Y0(Σ ⊗ Θ)−1Y . (6)
3. ESTIMATION
All the parameters of Σ, Ψ and Θ in the covariance matrix D(vecX ) = Θ ⊗ Ψ ⊗ Σ are not uniquely defined. Several authors have discussed this for a separable covariance matrix D(vecX) = Ψ ⊗ Σ; see, for example, Galecki (1994), and Naik and Rao (2001). The parametrization problem is related to the fact that
Ψ ⊗ Σ = (cΨ) ⊗ 1 cΣ
, (7)
and this leads to estimability problems. Recently, Srivastava et al. (2008) also considered the problem and suggested setting ψqq = 1, without any loss of generality. For a double
separable covariance matrix we have a similar problem since Θ ⊗ Ψ ⊗ Σ = 1
abΘ
⊗ (aΨ) ⊗ (bΣ) .
In this case, to get a unique parametrization, without any loss of generality and similar to Srivastava et al. (2008), we suppose Σ: p × p to be unstructured, Ψ = (ψij): q × q with
ψqq = 1 and Θ = (θij): r × r with θrr = 1. Now, assume that we have n independent
observations Xd : p × q × r, d = 1, . . . , n, from (1), where n > max{p, q, r}. One can
easily see that M : p × q × r will be estimated by averaging. Hence, in the subsequent computations, we may take M = 0, without loss of generality.
Furthermore, with M = 0, the likelihood function for Σ, Ψ and Θ is proportional to |Θ|−pqn/2|Ψ|−prn/2|Σ|−qrn/2exp ( −1 2 n X d=1 vec0Xd(Θ ⊗ Ψ ⊗ Σ) −1 vecXd ) , which can be written as
|Θ|−pqn/2|Ψ|−prn/2|Σ|−qrn/2exp ( −1 2 n X d=1 vec0Xd(Σ ⊗ Θ ⊗ Ψ) −1 vecXd ) = |Θ|−pqn/2|Ψ|−prn/2|Σ|−qrn/2exp ( −1 2 n X d=1 trΣ−1 X0d(Θ ⊗ Ψ)−1Xd ) , (8) using (5). Now, the trace in (8) can be rewritten as
trΣ−1X0d(Θ ⊗ Ψ)−1Xd = tr n Σ−1X0dIr⊗ Ψ−1/2 Θ−1⊗ Iq Ir⊗ Ψ−1/2 Xd o = tr ( Σ−1X0dIr⊗ Ψ−1/2 Θ−1⊗ q X l=1 e2l(e2l)0 !! Ir⊗ Ψ−1/2 Xd ) = q X l=1 trnΣ−1X0dIr⊗ Ψ−1/2 Θ−1⊗e2l e2l0 Ir⊗ Ψ−1/2 Xd o = q X l=1 trnΣ−1X0dIr⊗ Ψ−1/2e2lΘ−1Ir⊗ e2l0Ψ−1/2Xd o = q X l=1 trΣ−1X0dlΘ−1Xdl , where Xdl = Ir⊗
(e2l))0Ψ−1/2Xd, which implies that the likelihood function is
pro-portional to |Θ|−pqn/2|Ψ|−prn/2|Σ|−qrn/2exp ( −1 2 n X d=1 q X l=1 trΣ−1X0dlΘ−1Xdl ) , (9)
which further implies that we have nq independent observations, Xdl d = 1, . . . , n and
l = 1, . . . , q. From Srivastava et al. (2008), under the restriction θrr = 1, we obtain the
likelihood equations b Σ = 1 qrn n X d=1 q X l=1 X0dlΘb −1 Xdl, b Θ = 1 pqn n X d=1 q X l=1 XdlΣb −1 X0dl,
which can, using similar calculations as above but reverse order, be reformulated as b Σ = 1 qrn n X d=1 q X l=1 X0dlΘb −1 Xdl = 1 qrn n X d=1 q X l=1 X0dIr⊗ bΨ −1/2 e2lΘb −1 Ir⊗ (e2l) 0 b Ψ−1/2Xd = 1 qrn n X d=1 X0dΘ ⊗ bb Ψ −1 Xd, (10) b Θ = 1 pqn n X d=1 q X l=1 XdlΣb −1 X0dl = 1 pqn n X d=1 q X l=1 Ir⊗ e2l0Ψb −1/2 XdΣb −1 X0dIr⊗ b Ψ−1/2e2l. (11)
Using (6), the likelihood function (8) can also be expressed as
|Θ|−pqn/2|Ψ|−prn/2|Σ|−qrn/2exp ( −1 2 n X d=1 trΨ−1Y0d(Σ ⊗ Θ)−1Yd ) = |Θ|−pqn/2|Ψ|−prn/2|Σ|−qrn/2exp ( −1 2 n X d=1 r X l=1 trΨ−1Y0dlΣ−1Ydl ) = |Θ|−pqn/2|Ψ|−prn/2|Σ|−qrn/2exp ( −1 2 n X d=1 r X l=1 trΣ−1YdlΨ−1Y0dl ) , (12) where Ydl = Ip⊗ (e3l)0Θ−1/2
Yd. Now, with ψqq = 1, and again following Srivastava
et al. (2008), we arrive at the following likelihood equations
b Ψ = 1 prn n X d=1 r X l=1 Y0dlΣb −1 Ydl = 1 prn n X d=1 r X l=1 Y0dIp⊗ b Θ−1/2e3lΣb −1 Ip⊗ e3l0Θb −1/2 Yd = 1 prn n X d=1 Y0dΣ ⊗ bb Θ −1 Yd, (13) b Σ = 1 qrn n X d=1 r X l=1 YdlΨb −1 Y0dl = 1 qrn n X d=1 r X l=1 Ip⊗ e3l0Θb −1/2 YdΨb −1 Y0d Ip⊗ b Θ−1/2e3l . (14)
The following theorem can now be stated.
Theorem 1 The likelihood equations which maximize the likelihood function (8), under the conditions ψqq = 1 and θrr = 1, are given as following.
b Σ = 1 qrn n X d=1 X0dΘ ⊗ bb Ψ −1 Xd, (15) b Ψ = 1 prn n X d=1 Y0d b Σ ⊗ bΘ −1 Yd, (16) b Θ = 1 pqn n X d=1 Z0dΨ ⊗ bb Σ −1 Zd. (17)
Furthermore, equation (10) equals equation (14).
Proof The likelihood functions (9) and (12) are the same but with different factorizations. Hence, equations (10) and (14) should also be the same. To see this, let Xd = xdijk,
d = 1, . . . , n be the observations. Then, using (2) and (3), bΣ in (14) can be re-written as
b Σ = 1 qrn n X d=1 r X l=1 (Ip⊗ ((e3l) 0 b Θ−1/2))YdΨb −1 Y0d(Ip⊗ ( bΘ −1/2 e3l)) = 1 qrn n X d=1 r X l=1 X ijk X i0j0k0 xdijkxdi0j0k0 (Ip⊗ ((e3l) 0 b Θ−1/2))(e1i ⊗ e3 k) h (e2j)0Ψb −1 e2j0 i ((e1i0)0⊗ (e3k0)0)(Ip⊗ ( bΘ −1/2 e3l)) = 1 qrn n X d=1 X ijk X i0j0k0 ( (xdijkxdi0j0k0 h (e2j)0Ψb −1 e2j0 i r X l=1 Ip⊗ ((e3l) 0 b Θ−1/2) e1i(e1i0)0 ⊗ e3k(e3k0)0 Ip⊗ ( bΘ −1/2 e3l) ) = 1 qrn n X d=1 X ijk X i0j0k0 ( xdijkxdi0j0k0 h (e2j)0Ψb −1 e2j0 i r X l=1 e1i(e1i0)0⊗ h (e3l)0Θb −1/2 e3ki h(e3k0)0Θb −1/2 e3li ) = 1 qrn n X d=1 X ijk X i0j0k0 ( xdijkxdi0j0k0 h (e2j)0Ψb −1 e2j0 i (e3k0)0Θb −1/2 Xr l=1 e3l(e3l)0 ! b Θ−1/2e3ke1i(e1i0)0 ) ,
which, using Pr l=1e 3 l(e3l) 0 = I r, further simplifies to b Σ = 1 qrn n X d=1 X ijk X i0j0k0 xdijkxdi0j0k0 h (e2j)0Ψb −1 e2j0 i h (e3k0)0Θb −1 e3kie1i(e1i0)0 = 1 qrn n X d=1 X ijk X i0j0k0
xdijkxdi0j0k0((e3k⊗ e2j)(e1i)0)0( bΘ ⊗ bΨ)−1(e3k0⊗ e2j0)(e1i0)0
= 1 qrn n X d=1 X0d( bΘ ⊗ bΨ)−1Xd.
Proceeding on the same lines, equation (11) can be simplified to
b Θ = 1 pqn n X d=1 Z0jΨ ⊗ bb Σ −1 Zj, (18)
which completes the proof.
Note that the likelihood equations (15)-(17) are nested and there exist no explicit solution. However, we can solve (15)-(17) using the so called flip-flop algorithm. We may also note that by using the results of Srivastava et al. (2008), the algorithm converges to a unique solution provided there is enough data, and ψqq = θrr = 1 is chosen to be part of the starting
values.
4. RESTRICTIONS IMPOSED BY THE KRONECKER PRODUCT
Many statistical hypotheses can be formulated in terms of polynomial equalities and inequalities in the unknown parameters. Hence, under the null hypothesis the parameter space corresponds to semi-algebraic subsets of the parameter space. In statistical testing, it is important to consider the parameter space under the null hypothesis carefully, see, for example, Rao (1973, pp 415-420) for some general classes of large sample tests. See also Self and Liang (1987), and Drton (2009) for more details.
The double separable covariance matrix, D(X) = Ω = Θ ⊗ Ψ ⊗ Σ, imposes a number of restrictions on the parameter space of the variances and covariances. Hence, the hypothesis
can be written as
H0 : Rs(Ω) = 0 for s = 1, . . . , S vs. H1 : not H0,
where Rs(Ω), s = 1, . . . , S, are some functions of the variances and covariances.
The restrictions imposed by separability, i.e., Ω = Ψ ⊗ Σ, are shortly discussed by Lu and Zimmerman (2005), and can be stated as
ω11 ωip+1,ip+1 = ω22 ωip+2,ip+2 = · · · = ωpp ωip+p,ip+p , i = 1, . . . , q − 1, ρ[ii][kl] = ρ[11][kl], i = 2, . . . , q; k = 1, . . . , p; l = k + 1, . . . , p, ρ[ij][kk] = ρ[ij][11], i = 1, . . . , q; j = i + 1, . . . , q; k = 2, . . . , p, (19) ρ[ij][kl] = ρ[ij][lk], i = 1, . . . , q; j = i + 1, . . . , q; l = k + 1, . . . , p, ρ[ij][kl] = ρ[ij][11]ρ[11][kl], i = 1, . . . , q; j = i + 1, . . . , q; k = 1, . . . , p; l = k + 1, . . . , p,
where Ω = (ωij) and ρ[ij][kl]is the (k, l)th element of the (i, j)th p × p block of the correlation
matrix RΩ. Since the nature (singularities, number of free parameters etcetera) of the
functions Rs(Ω), s = 1, . . . , S, is important, we state the following proposition.
Proposition 1 The functions Rs(Ω), s = 1, . . . , S, imposed by the Kronecker product
struc-ture, Ω = Θ ⊗ Ψ ⊗ Σ, are smooth functions.
Proof We consider the simple cases p = q = 2 and p = q = r = 2 for a separable and double separable covariance matrix, respectively, and use these examples to understand the restrictions imposed by the double separability. We start with separability and p = q = 2. Consider the Kronecker product
Ω = Ψ ⊗ Σ = ψ11σ11 ψ11σ12 ψ12σ11 ψ12σ12 · ψ11σ22 ψ12σ21 ψ12σ22 · · ψ22σ11 ψ22σ12 · · · ψ22σ22 = (ωij) ,
where Σ = (σij) and Ψ = (ψij). We can directly identify one restriction since ω11 ω22 = ω33 ω44 . (20)
More restrictions can be found from the correlation matrix of Ω, denoted RΩ. The
cor-relation matrix RΩ is nothing else than the Kronecker product of the correlation matrices
corresponding to Ψ and Σ, i.e.,
RΩ= RΨ⊗ RΣ = 1 ρΣ ρΨ ρΨρΣ · 1 ρΨρΣ ρΨ · · 1 ρΣ · · · 1 = (ρij) , (21) where RΣ= 1 ρΣ · 1 and RΨ = 1 ρΨ · 1 ,
are the correlation matrices corresponding to Σ and Ψ, respectively. From (21) we see that we have the following restrictions
ρ12= ρ34, ρ13= ρ24, ρ14 = ρ23 and ρ14 = ρ12ρ13. (22)
The restrictions (20) and (22) are, of course, nothing else than the restrictions (19) given by Lu and Zimmerman (2005). Written in the original covariances ωij, the restrictions are
ω11ω14 = ω12ω13, ω23 = ω14, ω11ω24 = ω13ω22,
ω11ω34 = ω12ω33, ω11ω44= ω22ω33,
which implies that the functions Rs(Ω), s = 1, . . . , 5, can be formulated as
R1(Ω) = ω11ω14− ω12ω13, R2(Ω) = ω23− ω14, R3(Ω) = ω11ω24− ω13ω22,
R4(Ω) = ω11ω34− ω12ω33, R5(Ω) = ω11ω44− ω22ω33.
Similar argument can be used when considering the double separable covariance matrix. For the case with three Kronecker products, and with p = q = r = 2, we have the following covariance matrix
Ω = Θ ⊗ Ψ ⊗ Σ = (ωij), (23)
which leads to the following restrictions ω11 ω33 = ω22 ω44 = ω55 ω77 = ω66 ω88 and ω11 ω55 = ω22 ω66 . (24)
Furthermore, the correlation matrix
RΩ= RΘ⊗ RΨ⊗ RΣ= (ρij) , where RΘ = 1 ρΘ · 1 ,
gives all the other restrictions
ρ12= ρ34 = ρ56 = ρ78, ρ13 = ρ24 = ρ57= ρ68, ρ14= ρ23 = ρ58 = ρ67, ρ15 = ρ26 = ρ37= ρ48, ρ16= ρ25 = ρ38 = ρ47, ρ17 = ρ28 = ρ35= ρ46, ρ18= ρ27 = ρ36 = ρ45 (25) and ρ14= ρ12ρ13, ρ16= ρ12ρ15, ρ17 = ρ13ρ15, ρ18= ρ12ρ13ρ15. (26)
These 29 restrictions, (24)-(26), are similar and a direct generalization of the restrictions (20) and (22) imposed by separability. Since they have the same form, the functions Rs(Ω),
s = 1, . . . , 29, given by the double separability are also smooth functions. Moving further by induction, the smoothness for general dimensions, p, q and r, can also be shown.
From Proposition 1, we observe that the functions Rs(Ω), s = 1, . . . , S, have continuous
partial derivatives of the first order and this will facilitate the asymptotics, see Rao (1973, p. 415-420) for more details.
Under separability, the covariance matrix Ω = Ψ ⊗ Σ has 12(p × (p + 1) + q × (q + 1)) parameters. Under H1, the covariance matrix is the unstructured matrix Ω : pq × pq which
has 12pq × (pq + 1) parameters. Hence, under separability and p = q = 2, the covariance matrix Ω = Ψ ⊗ Σ has six parameters, and under the alternative, the covariance matrix Ω > 0 : 4 × 4 has ten parameters. Expressing the parameters ω14, ω23, ω24, ω34 and ω44 as
functions of ω11, ω12, ω13, ω22 and ω33, we have
ω14= ω12ω13 ω11 , ω23 = ω14, ω24 = ω13ω22 ω11 , ω34= ω12ω33 ω11 , ω44 = ω22ω33 ω11 .
Clearly, if we express the five parameters ω11, ω12, ω13, ω22and ω33in terms of six parameters
σ11, σ12, σ22, ψ11, ψ12 and ψ22, we have
ω11= ψ11σ11, ω12 = ψ11σ12, ω13 = ψ12σ11,
ω22= ψ11σ22, ω33 = ψ22σ11,
which gives a system of five equations in six parameters, such that all all parameters of Ψ and Σ are not uniquely defined (see equation (7)). This problem can be overcome by imposing restrictions on certain parameters, for example setting ψ22= 1.
Furthermore, for the double separability case with p = q = r = 2, we have 29 equations and 36 parameters in the unstructured covariance matrix Ω > 0, i.e., we have seven free parameters instead of nine as in the Kronecker product Ω = Θ ⊗ Ψ ⊗ Σ. Hence, we can again set the restrictions ψ22= 1 and θ22 = 1. The same can be extended for higher values
of p, q, and r. For example, for the present case of double separability, the total number of free parameters for Ω can be computed as
p(p + 1) 2 + q(q + 1) 2 + r(r + 1) 2 − 2.
This helps computing unique estimates of the unknown matrices. Further, the restrictions obviously lead to setting the null hypotheses that can be tested, based on properly formulated test procedures, for example, likelihood ratio tests. For a study of multiple covariance matrices, based on their spectral decomposition, see Boik (2002).
BIBLIOGRAPHY
Boik, R. J. (2002). Spectral models for covariance matrices. Biometrika, 89(1):159–182. Brown, P. J., Kenward, M. G., and Basset, E. E. (2001). Bayesian discrimination with
longitudinal data. Biostatistics, 2(4):417–432.
Drton, M. (2009). Likelihood ratio tests and singularities. The Annals of Statistics, 37(2):979–1012.
Dutilleul, P. (1999). The MLE algorithm for the matrix normal distribution. Journal of Statistical Computation and Simulation, 64(2):105–123.
Galecki, A. T. (1994). General class of covariance structures for two or more repeated factors in longitudinal data analysis. Communications in Statistics Theory and Methods, 23:3105–3119.
Hoff, P. (2011). Separable covariance arrays via the tucker product, with applications to multivariate relational data. Bayesian Analysis, 6(1):1–18.
Kolda, T. G. and Bader, B. W. (2009). Tensor decompositions and applications. SIAM Review, 51(3):455–500.
Kollo, T. and von Rosen, D. (2005). Advanced Multivariate Statistics with Matrices. Springer, Dordrecht, The Netherlands.
Lu, N. and Zimmerman, D. (2005). The likelihood ratio test for a separable covariance matrix. Statistics and Probability Letters, 73:449–457.
Mardia, K. V. and Goodall, C. R. (1993). Spatial-temporal analysis of multivariate environ-mental monitoring data. Multivariate Environenviron-mental Statistics, 6:347–385.
Mitchell, M. W., Genton, M. G., and Gumpertz, M. L. (2005). Testing for separability of space-time covariances. Environmetrics, 16(8):819–831.
Mitchell, M. W., Genton, M. G., and Gumpertz, M. L. (2006). A likelihood ratio test for separability of covariances. Journal of Multivariate Analysis, 97(5):1025–1043.
Naik, D. and Rao, S. (2001). Analysis of multivariate repeated measures data with a kro-necker product structured covariance matrix. Journal of Applied Statistics, 28:91–105. Rao, C. (1973). Linear Statistical Inference and Its Applications. Wiley, New York, USA. Roy, A. (2007). A note on testing of Kronecker product covariance structures for doubly
multivariate data. In Proceedings of the American Statistical Association, Statistical Com-puting Section, pages 2157–2162.
Roy, A. and Khattree, R. (2003). Tests for mean and covariance structures relevant in repeated measures based discriminant analysis. Journal of Applied Statistical Science, 12(2):91–104.
Roy, A. and Khattree, R. (2005a). On implementation of a test for kronecker product covariance structure for multivariate repeated measures data. Statistical Methodology, 2:297–306.
Roy, A. and Khattree, R. (2005b). Testing the hypothesis of a Kroneckar product covariance matrix in multivariate repeated measures data. In Proceedings of the Thirty Annual SAS R
Users Group International (SUGI) Conference, Philadelphia, Pennsylvania, pages 1–11. Roy, A. and Leiva, R. (2011). Estimating and testing a structured covariance matrix for
three-level multivariate data. To appear in ”Communications in Statistics Theory and Methods”.
Self, S. G. and Liang, K.-Y. (1987). Asymptotic properties of maximum likelihood estima-tors and likelihood ratio tests under nonstandard conditions. Journal of the American Statistical Association, 82(398):605–610.
Srivastava, M., von Rosen, T., and von Rosen, D. (2008). Models with a Kronecker product covariance structure: estimation and testing. Mathematical Methods in Statistics, 17:357– 370.
Srivastava, M., von Rosen, T., and von Rosen, D. (2009). Estimation in general multivariate linear models with kronecker product covariance structure. Sankhy¯a, 71, Ser. A(2):137– 163.
