1
Independent Component Analysis Using the Hough Transformation
Andrew Pound, Todd Moon and Jacob Gunther
Abstract—This paper compares three methods of Independent Component Analysis (ICA) for the purpose of blindly separating sources. It demonstrates the Hough Transform ICA and shows the gains obtained by using the classic image processing tool in the ICA algorithm.
I. INTRODUCTION
Independent Component Analysis (ICA) is an important analysis tool in signal processing. It allows for sources to be isolated from mixed signals, based on some measure of separation, such as their statistical independence [1] [2]. We assume here that the signals received are mixed instanta- neously. We restrict our attention here to mixtures of two signals. If s= [s1s2]T is our signal, then define x, the mixed signal, as
x= As, (1)
where A is an unknown mixing matrix. The problem is one of finding or producing the signal s again, up to a permutation and/or scaling factor. Whitening is a common first step, which decorrelates the signals [1]. Let z denote a whitened vector, then
z= V x = V A s (2)
where V is the whitening transformation.
The motivation to examine the properties of the Hough Transformation (HT) as an algorithm for ICA, was the obser- vation of linear structure in the histogram of a whitened, mixed signal, such as a uniform random or audio signal. This can be seen in Figure 1. Each signal produces lines evident in the histogram whereby the amount of rotation can be determined.
The HT identifies these features and we can use them to separate the signals [3]. Tests showed very promising results, and gave rise to the interest in comparing the HT to other ICA algorithms.
II. HOUGHICA A. Background
The Hough Transform is a standard image processing tool used to find lines in images. It was developed in 1959 by Hough to detect bubble trails in bubble chamber pictures [4].
In 1972, Duda and Hart demonstrated the capability of using the Hough Transform to detect both lines and curves in images
Authors are with the Department of Electrical and Computer Engineer- ing, Utah State University, Logan, Utah 84212. Correspondence email:
Andrew.Pound@aggiemail.usu.edu
This research was supported by the NSA Coding and Communications Re- search Experience for Undergraduates Program at the Electrical and Computer Engineering Department, Utah State University, Logan, Utah.
−6 −4 −2 0 2 4 6
−4
−2 0 2 4 6
Whitened Signals
(a)
0 1 2 3 4 5
−1
−0.5 0 0.5 1 1.5 2 2.5 3
Whitened Uniform Random Signal
(b)
Fig. 1: Histograms of mixed and whitened signals. (a) Audio signals. (b) Uniform random signals.
[5]. Since that time many improvements and variants have been proposed, such as the Generalized Hough Transform [6], kernel-based Hough Transform [7], and an Adaptive Hough Transform [8].
B. The Hough Transformation
The general idea behind the Hough Transform is the trans- lation from the conventional x and y cartesian coordinates into a polar-like coordinate space. In the present situation, this corresponds to a mapping of a line in (x, y) coordinates to a(ρ, θ), where ρ is the distance from some set origin of the
2
θ ρ
Fig. 2: Any line can be parameterized by ρ and θ as shown, relative to the origin.
1 2 3 4 5 6 7
1 2 3 4 5 6 7
(a)
−80 −60−40−20 0 20 40 60 80
−8
−6
−4
−2 0 2 4 6 8
θ
ρ
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
(b)
(c) (d)
Fig. 3: (a) A single point (b) Its Hough Transform. (c) An image containing two lines. (d) The Hough Transform of (c);
the bright points of convergence are indicative of the two lines in the image.
image to the line in question, and θ is the angle corresponding to the angle at which that line is rotated from a set axis. The transformation can be expressed mathematically as
ρ= x cos(θ) + y sin(θ). (3) Each line in the data space is mapped to a point (ρ, θ) in Hough space. Figure 2 shows the mapping of a line to Hough space coordinates. For any given point(x, y), because there are an infinite number of lines that can pass through it (every θ is possible), then the point (x, y) gets mapped to a sinusoidal curve in the Hough space. A simple example is shown in Figure 3.
C. Application to ICA
Ultimately, the goal of the ICA algorithm is to perform the rotation from the whitened signal as quickly and accurately as possible. An ideal solution would be to determine the angle at which the signals are rotated, and perform a matrix rotation to restore them to the desired orientation. The Hough Transform finds the most dominant intrinsic lines in the histogram image and their rotation, which allows for a single rotation algorithm.
(a)
(b)
Fig. 4: The Hough Transform produces some erroneous arti- facts which can be greater in magnitude than the true peak. (a) This illustrates some of these artifacts. The arrow corresponds to some of the worst aberrations. (b) The filtered HT
D. Smoothing Techniques
Some issues arose with respect to artifacts appearing in the Hough Transform because of the binning of the data points to form an image for the Hough Transform to work with. A close examination of Figure 4a will reveal spurious lines at approximately 45o. Two pixels that are diagonally adjacent will be recognized as part of a line. This happens as a result of the spatial separation of the data into appropriate bins. An analogous line appears at −45o. These lines can interfere with finding the peaks of the Hough Transform, and thus must be removed in some fashion. Different smoothing techniques were tried, including standard image processing filters (Gaussian and median filters), but the best performing filter was a standard 11-tap low-pass 2D filter. The simple filter outperformed the others, both in computational time and in results. The resulting filtered Hough Transform can be seen in Figure 4b.
3
E. Algorithm The algorithm is:
1) Whiten the data.
2) Form the histogram image, using some bin spacing.
3) Compute the Hough Transform of the histogram image.
4) Spatially filter the Hough Transform to remove spurious peaks.
5) Find the largest peak.
6) Use θ value to produce a rotation matrix.
7) Rotate the data to the separated signals.
III. OTHERALGORITHMS
The HT-based ICA was compared with several existing ICA algorithms based on nongaussianity maximization using kurtosis as a metric. The algorithms are summarized here, but a more in depth basis for and analysis of them is given in [1]
and [2].
A. Maximization of Kurtosis
The kurtosis for a random variable X is defined as kurt(X) = EX4 − 3 E X22
. (4)
The kurtosis is a measure of the Gaussianity of the variable.
By the Central Limit Theorem, mixtures of random variables tend toward a Gaussian distribution as more and more random variables are mixed together. Thus, in order to unmix them, we need to minimize the Gaussianity of the mixture, or equivalently, maximize the kurtosis of the mixed variables.
L=
kurt(wTx) +λ
2 wTw− 1
(5) Updates are based upon the gradient of this cost function,
∂L
∂w = λw +
4 sign(kurt(wTx))E{(wTx)} − 3w .
(6)
B. “Slow” ICA
The slow algorithm implemented was based on the kurtosis maximization problem defined in Section III-A. The algorithm takes the whitened noise and an initial vector w and then updates the vector by gradient descent,
wn+1= wn+ µ∂L
∂w w
n
. (7)
The convergence may take many iterations to actualize. Aver- age number of iterations using a random mixing matrix were on the order of 300 iterations.
C. Fast ICA
The fast ICA algorithm was also based on the afore men- tioned kurtosis maximization problem. Like the slow iterative implementation, this algorithm depends on the whitened noise and an initial vector w, but unlike the slow implementation, does not use and additive update rule. The update rule used is:
w= µ∂L
∂w wn
wn+1= w
||w||
(8)
On average, this implementation converges within 10 itera- tions.
IV. RESULTS ANDCONCLUSIONS
A. Simulations
The simulations were run for various parameters of the Hough ICA and compared with the slow and fast ICA im- plementations discussed in Section III. Each test was run with 1000 randomly generated mixing matrices.The error was calculated using the fact that for this problem,
W V A= P D = C (9)
where P is a permutation matrix, and D is a diagonal matrix.
Thus, C is an approximation of a scaled, permuted identity matrix.The off-diagonal elements of C may not be exactly0, and thus we can compare the algorithms by looking at some measure of how close the C is to a permutation matrix. The formula used is
e=X
i
X
j
|ci,j| maxk |ci,k|
+X
j
X
i
|ci,j| maxk |ck,j|
− 2n, (10) where n is the dimension of the matrix C. This is used to allow for a comparison between any of the algorithms based on their ability to separate the signals [2].
B. Results
As can be seen in Figure 5, the fast ICA algorithm dramat- ically outperforms the slow ICA in terms of computational speed, as expected. In these tests, the HT-based ICA is com- parable to or better than the fast ICA. However, the error of the HT ICA algorithm shows much more variation. Thus, the majority of the time, HT performed faster, but the error may be higher than that of the other algorithms. More extensive testing has been done to confirm this, and it seems that an HT- based algorithm that produces approximately the same error as the fast ICA will run in about the same time, choosing the correct parameters. The parameters used in the HT ICA algorithm featured in Figure 5 were a low resolution image (approximately100 × 100 pixels), a 21-tap low-pass filter and a θres= 1o angle resolution, without taking into account the density of points.
A feature of the Hough Transform ICA algorithm is that various resolutions can be used for the input image. Figure 6 shows the effects on time and error of different resolutions.
4
0 50 100 150 200 250 300 350 400 450 500
10−2 10−1 100 101
Tests
Time (sec)
Comparison of ICA methods Time
slow ICA fast ICA HT ICA
(a)
0 50 100 150 200 250 300 350 400 450 500
10−3 10−2 10−1 100
Tests
Error
Comparison of ICA methods Error
slow ICA fast ICA HT ICA
(b)
Fig. 5: (a) This is a comparison of the time to run to comple- tion for each of the ICA algorithms. The Hough Transform ICA algorithm was run with an extremely low resolution image of the signal, and with a θres = 1o resolution. (b) A comparison of the error for each of the tests in (a).
0 20 40 60 80 100 120
10−1 100 101 102
Tests
Time(sec)
Time Comparison of HT Resolution Effects
Low Res Mid Res Hi Res
(a)
0 20 40 60 80 100 120
10−3 10−2 10−1 100 101
Tests
Error
Error Comparison of HT Resolution Effects
Low Res Mid Res Hi Res
(b)
Fig. 6: A comparison of the times and errors of different resolutions of the input images. The low, middle, and high resolutions are approximatley (100 × 100), (800 × 1000), and (12000 × 15000) pixels, respectively.
C. Conclusions
The fact that source separation can be performed in varied manners is self evident in the numerous ICA algorithms pro- posed already. The possibility of using image processing tech- niques to develop a faster and more accurate ICA algorithm has been explored, and shows promise from a computational point of view.
REFERENCES
[1] A. Hyvarinen and E. Oja, “Independent component analysis: algorithms and applications,” Neural Networks, vol. 13, no. 4-5, pp. 411 – 430, 2000.
[2] A. Hyvarinen, J. Karhunen, and E. Oja, Independent Component Analysis.
London: J. Wiley, 2001.
[3] J. Lin, D. Grier, and J. Cowan, “Feature extraction approach to blind source separation,” in Neural Networks for Signal Processing [1997] VII.
Proceedings of the 1997 IEEE Workshop, pp. 398–405, Sep 1997.
[4] P. V. C. Hough, “Machine analysis of bubble chamber pictures,” Inter- national Conference on High Energy Accelerators and Instrumentation, 1959.
[5] R. O. Duda and P. E. Hart, “Use of the hough transformation to detect lines and curves in pictures,” Commun. ACM, vol. 15, no. 1, pp. 11–15, 1972.
[6] D. H. Ballard, “Generalizing the hough transform to detect arbitrary shapes,” Pattern Recognition, vol. 13, no. 2, pp. 111–122, 1981.
[7] L. A. F. Fernandes and M. M. Oliveira, “Real-time line detection through an improved hough transform voting scheme,” Pattern Recognition, vol. 41, no. 1, pp. 299–314, 2008.
[8] J. Illingworth and J. Kittler, “The adaptive hough transform,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. PAMI-9, no. 5, pp. 690–698, 1987.
