Notes on Differential Entropy of Mixtures

Full text

(1)Technical report from Automatic Control at Linköpings universitet. Notes on Differential Entropy of Mixtures Umut Orguner Division of Automatic Control E-mail: umut@isy.liu.se. 15th August 2008 Report no.: LiTH-ISY-R-2856. Address: Department of Electrical Engineering Linköpings universitet SE-581 83 Linköping, Sweden WWW: http://www.control.isy.liu.se. Technical reports from the Automatic Control group in Linköping are available from http://www.control.isy.liu.se/publications..

(2) Abstract. This report proves that the dierential entropy of particle mixtures is equal to −∞ unlike the wrong claim in the literature that it is equal to the discrete entropy of particle weights. It then gives an upper bound for the dierential entropy of the Gaussian mixtures which can be used in practical applications.. Keywords: Entropy, particle mixture, Gaussian mixture.

(3) This document consists of some notes on dierential entropies of some mixtures mainly represented by Gaussian components. Using Gaussian mixtures we rst prove that dierential entropy of particle (impulse function) mixtures (like the ones used in particle ltering) is −∞ as opposed to the common mistake that it depends completely on the particle weights. Then for Gaussian mixtures, we calculate an upper bound for dierential entropy which can be used in tracking applications.. 1 Particle Mixtures Suppose that we have an N -component particle mixture given as. p(x) =. N X. wi δ(x − x ¯i ). (1). i=1. PN where i=1 wi = 1, and x, x ¯i ∈ Rnx . It is obvious that we can see this mixture as a limit as follows. p(x) = lim p² , lim ²→0. ²→0. N X. wi N (x, x ¯i , ²Inx ). (2). i=1. where the notation N (x, x ¯, P ) denotes a Gaussian density in dummy variable x with mean x ¯ and covariance P and Inx is an identity matrix os size nx ×nx . The dierential entropy corresponding to any probability density function is dened as Z ∞ Hp (x) , − p(x) log(p(x))dx (3) −∞. Assumption • The mean values {¯ xi }N i=1 are all dierent from each other. This is not a strict condition because if two of them are the same, the two corresponding impulses can be summed with their corresponding weights to form a similar mixture with now N − 1 components with dierent means. Then,. Z Hp² (x) , −. =−. ∞. N X. −∞ i=1 N X. Z.   N X wi N (x, x ¯i , ²Inx ) log  wj N (x, x ¯j , ²Inx ) dx ∞. wi −∞. i=1. (4). j=1.   N X N (x, x ¯i , ²Inx ) log  wj N (x, x ¯j , ²Inx ) dx. (5). j=1. Since ² is very small, the integration above is eectively around a very small neighborhood of the ith mean x ¯i . Again for the same reason in the same neighborhood N X j=1. wj N (x, x ¯j , ²Inx ) ≈ wi N (x, x ¯i , ²Inx ). (6).

(4) Then. Hp² (x) = −. N X. Z. −∞. i=1. =−. N X. ∞. wi. N (x, x ¯i , ²Inx ) log (wi N (x, x ¯i , ²Inx )) dx. wi log(wi ) +. i=1. N X. wi H {N (x, x ¯i , ²Inx )}. (7) (8). i=1. Now taking the limit as ² → 0 we see that the terms H {N (x, x ¯i , ²Inx )} ∝ log(²) go to −∞ for i = 1, . . . , N and we get. Hp (x) = −. N X. wi log(wi ) − ∞ = −∞. (9). i=1. Therefore, the dierential entropy of densities represented with particles is −∞. A common mistake made here is to take lim²→0 H {N (x, x ¯i , ²Inx )} = 0 which gives an entropy result that equals the entropy of the discrete density formed by the particle weights, which is wrong.. 2 Upper Bound for dierential Entropy of Gaussian Mixtures The upper bound uses the concavity of the logarithm function. Suppose we have the Gaussian mixture. p(x) =. N X. wi N (x; x ¯ i , Pi ). (10). i=1. then by the concavity of the logarithm function Ã ! N N X X log wi wi N (x; x ¯ i , Pi ) ≥ wi log (N (x; x ¯i , Pi )) i=1. Applying this to the dierential entropy Z ∞X N N X wi N (x, x ¯i , ²Inx ) wj log (N (x; x ¯j , Pj )) Hp (x) ≤ − −∞ i=1. N. N. 1 XX = wi wj 2 i=1 j=1. (11). i=1. (12). j=1. Z. ∞. −∞. ¡ ¢ N (x, x ¯i , ²Inx ) log (|2πPj |) + (x − x ¯j )T Pj−1 (x − x ¯j ) (13). N. =. N. £ © ª¤ 1 XX wi wj log (|2πPj |) + tr Pj−1 Pi + Pj−1 (¯ xi − x ¯j )(¯ xi − x ¯ j )T 2 i=1 j=1 (14). =. 1 2. N X. wj log (|2πPj |). j=1 N. +. N. © ª 1 XX wi wj tr Pj−1 Pi + Pj−1 (¯ xi − x ¯j )(¯ xi − x ¯ j )T 2 i=1 j=1 2. (15).

(5) To gain more insight into what this bound calculates, we suppose Pi = αInk for i = 1, . . . , N . Then N. Hp (x) ≤. N. N. © ª 1 XX 1X wj nx log (2πα) + wi wj tr Inx + (¯ xi − x ¯j )T (¯ xi − x ¯j ) 2 j=1 2 i=1 j=1 (16) N. N. nx 1 XX 1 + = nx log (2πα) + wi wj k¯ xi − x ¯j k2 2 2 2α i=1 j=1 N. (17). N. £ ¤ nx log(2παe) 1 XX + wi wj k¯ xi k2 + k¯ xj k2 − 2¯ xTi x ¯j 2 2α i=1 j=1  °N °2  N °X ° nx log(2παe) 1 X ° ° = + wi k¯ xi k2 − ° wi x ¯i °  ° ° 2 α =. i=1. (18). (19). i=1. It is also interesting that when means and the covariances are equal i.e., x ¯i = x ¯j and Pi = Pj for 1 ≤ i, j ≤ N (this is the case when p(x) = N (x, x ¯, P )), the bound (15) is equal to. 1 Hp (x) ≤ log (|2πeP |) 2. (20). which is the dierential entropy of a single Gaussian with any mean and covariance P . Therefore the bound is tight when the means are close to each other. When we let ² → 0, the upper bound goes to −∞ and therefore do not tell anything about the entropy of resulting particle mixture. On the other hand, if the particle mixture is approximated with a Gaussian mixture, the above formulas can be used instead of entropy as an approximate cost function which would have the implication that we are minimizing an upper bound for the entropy.. 3.

(6)

No results found