A Real-Time Classification approach of a Human Brain-Computer Interface based on Movement Related Electroencephalogram

(1)

(2)

(3)

A Real-Time Classification Approach of a

Human Brain-Computer Interface Based on

Movement Related Electroencephalogram

by

Martin D. Mileros

INSTITUTE OF TECHNOLOGY AT LINK ¨OPING UNIVERSITY

LITH-IKP-EX–04/2196–SE RT-BCI SYSTEM DEVELOPEMENT

Link¨oping, December 2004

Examiner: Kjell Ohlsson, LiU Supervisors: Hans Jander, FOI

Magnus Borga, LiU Pontus Forslund, LiU

(4)

(5)

Thesisnumber: LITH-IKP-EX–04/2196–SE Author: Martin D. Mileros

Examiner: Kjell Ohlsson

Supervisors: Hans Jander, Magnus Borga, Pontus Forslund Figures: Martin D. Mileros with assistance of Jimmy Berggren Link¨oping University LiU

Swedish Defense Research Agency FOI Thesis Language - English

Printed in Link¨oping, Sweden 2004 Text Editor: LA_TEX

Program Editor: Matlab6.5

Graphic Editors: Adobe Photoshop CS, 3D Studio Max6.0

This thesis is submitted in partial fulfillment of the requirements for the degree of Master of Science in Applied Physics and Electrical Engineering at Link¨oping University, Sweden. The project was made in cooperation with the Swedish Defense Research Agency at the department of Man System Interaction.

(6)

(7)

Abstract

A Real-Time Brain-Computer Interface is a technical system classifying increased or decreased brain activity in Real-Time between different body movements, actions performed by a person. Focus in this thesis will be on testing algorithms and settings, finding the initial time interval and how increased activity in the brain can be distinguished and satisfyingly classified. The objective is letting the system give an output somewhere within 250ms of a thought of an action, which will be faster than a persons reaction time.

Algorithms in the preprocessing were Blind Signal Separation and the Fast Fourier Transform. With different frequency and time interval settings, the algo-rithms were tested on an offline Electroencephalographic data file based on the Ten-Twenty Electrode Application System and classified using an Artificial Neural Network.

A satisfying time interval could be found between 125-250ms, but more re-search is needed to investigate that specific interval. A reduction in frequency resulted in a lack of samples in the sample window preventing the algorithms from working properly. A high frequency is therefore proposed to help keeping the sample window small in the time domain. Blind Signal Separation together with the Fast Fourier Transform had problems finding appropriate correlation using the Ten-Twenty Electrode Application System. The electrodes should be placed more selectively at the parietal lobe, in case of requiring motor responses.

Keywords:

Real-Time Brain-Computer Interface (RT-BCI), Blind Signal Separation (BSS), Canonical Correlation Analysis (CCA), Electroencephalogram (EEG), Artificial Neural Networks (ANN), Real-Time Classification (RTC), Ten-Twenty Electrode Application System (10-20 System)

(8)

(9)

Acknowledgments

First of all I would like to thank my advisor Professor Kjell Ohlsson at the Depart-ment of Mechanical Engineering (IKP) at Link¨oping University (LiU) and Profes-sor Erland Svensson at the Department of Man System Interaction (MSI) at the Swedish Defense Research Agency (FOI), for their support and trust in my ideas and for giving me the opportunity to proceed with my work. Thank you, I really appreciate it!

I would like to aim a special thank to my supervisors: Hans Jander (MSI), for his engagement and support and for guiding me through this thesis. Magnus Borga at the Department of Biomedical Engineering (IMT), for sharing his time and knowledge leading me in to the appropriate methods and technique needed in this masterwork. Your knowledge and help has been the key to all my work, thanks! Pontus Forslund (ACE Simulation AB), for letting me use his material and data, and for showing interest and consideration of my work, thank you Pontus. Your knowledge and kindness has helped me tremendously. Thank you all!

Thanks also to: Martin Castor, Staffan N¨ahlinder and Jimmy Berggren at MSI, Martin Folkesson and all other friends who have taken their time helping me with my work at FOI. I would also like to show greatness to Anders Brun, Hans Knutsson and Joakim Rydell at IMT, for taking their time during discussions concerning algorithms and especially Anders who was the first to introduce me to work with EEG and BCI. Jonas Ericsson and Peter Johansson at the Department of Electrical Engineering (ISY), Tommy Elfving and Kurt Hansson at the Department of Mathematics (MAI), for helping me with my theories and confirming my ideas. Ros-Mari Andreasson, Mia Bruhagen and Wilhelm Isoz, for reading and helping me with my thesis. Last, but not least, Tomas and Ewa Mileros for their support in all matters, thank you! I would also like to thank all others who in one way or another contributed.

Thanks everybody!

(10)

(11)

Terminology

ABBREVIATIONS

System

BCI Brain-Computer Interface

RTS Real-Time System

RTC Real-Time Classification

RT-BCI Real-Time Brain-Computer Interface

Human

ALS Amyotrophic Lateral Sclerosis

Medical

CT Computer Tomography

EEG Electroencephalogram

fMRI Functional Magnetic Resonance Imaging

Mathematical

TFM Time and Frequency Modulation

FFT Fast Fourier Transform

BSS Blind Signal Separation

CCA Canonical Correlation Analysis

PCA Principal Component Analysis

ANN Artificial Neural Networks

MLP Multi Layer Perceptron

(14)

a, b, c, d Constants w, t Variables j Imaginary part x, y, s Vector representation ρ Correlation coefficient w Eigenvector

OPERATORS AND FUNCTIONS

x(t) A function x

E Statistical mean value

D A finite set

∂ Partial derivative

F (w) Fourier transform

OPERATIONS

s(t − a) Sample vector displaced in time

xT Transpose of a vector

(15)

Chapter 1 Introduction

One of the most complex things a scientist can deal with is the human brain. In computer science terms the brain could be seen as a highly complex, non-linear, parallel computer with capability to adapt by organizing itself to its environment. [5]. Science is still far away from knowing and understanding all functionality of the brain. Humanity has created and accomplished so much in the world with something no human really fully understand.

If our past relatives could be told that you can speak on the phone with people almost everywhere, they probably would say that you are crazy. If I told you that a pilot could steer his airplane only by using his mind in 2020, you would perhaps believe me. The modern society is probably more used to unbelievable developments and fiction than our past generations.

Nowadays a pilot cannot yet steer an airplane with his mind, but by using the power of a Real-Time Brain-Computer Interface (RT-BCI) System, an increased activity in different areas in the pilots brain can be detected and classified. The detection and action made by the RT-BCI System could be instantiated faster than the pilots reaction time, letting the airplane system be fully prepared to perform the intended action given by the pilot. For example, if a pilot would like to make a fast right turn, the airplane system could be fully prepared almost moving the pilots hand to the right. A RT-BCI could perhaps also be used with Computer Games, improving players’ reaction time.

(16)

1.1 Background

Since the second half of the twentieth century, almost all people in the modern society has come in contact with computers. Already in the mid 1950’s people quickly realized the power of computers and it was only a question of time until the computer got related to the human mind. The fascinating speed, memory and correctness of a computer could work as a good complement to the human brain.

A lot of research and effort is put into development in the area of Artificial Intelligence (AI), where scientists concentrate on creating robots with humanitarian qualities [18]. Other scientists go in the opposite direction implementing Microchips or other non-biological tissues into the human body. Those humans are called Cyborgs [3]. Development in both Artificial Intelligence and Cyborgs has grown a

Figure 1.1. An illustration of the BCI laying between the area of AI and Cyborgs.

lot since the mid 1980’s [3].

The area of BCI can be located somewhere between the area of AI and Cyborg. An illustration of the areas can be shown in Figure1.1. By looking at the gradient axis it is shown how the three areas more or less can be combined or merged together. Some BCI developers use surgery in their work and approach the Cyborg area [3]. In this thesis surgery for the human-computer integration will not be applied.

The ideas of creating a BCI System goes back to the 1970’s where the United States Department of Defense tried to develop systems helping pilots to interact with their aircraft. Unfortunately that dream was ahead of its time [11]. Computers those days were too slow, and the appropriate methods for handling the big streams of data were not developed until the 1980’s. Still a lot of research is done and since 1980 several of the techniques have been modified and improved [11]. As will follow, other non-conventional methods for maneuvering an airplane have also been developed [6]:

(17)

1.2 MOTIVE 3 • Maneuvering based on Eye and Head Movements

• Maneuvering based on Electromyography

• Maneuvering based on Voice and Sound

1.2 Motive

Since the power of computers increases each day, a whole new interesting world with new applicable areas appear. One area could be letting the RT-BCI System interact with simulators or Virtual Reality computer games for fast decisions and for avoiding obstacles. One other major area could be rehabilitation, medical help for people with different motor impairments or Amyotrophic Lateral Sclerosis (ALS), letting them communicate with other people by using a RT-BCI System.

1.3 Purpose

The main objective of the present thesis is to investigate the possibilities of letting a computer react faster on an motor action performed by a human, than the human itself. One further purpose is to make it possible for the BCI System to work in a Real-Time Environment. Reduction of data while keeping the accuracy in the classification is also of great concern, and the Blind Signal Separation (BSS) algorithm together with the Fast Fourier Transform (FFT) using the Ten-Twenty Electrode Application System (10-20 system) will also be studied, (see section2.3.2

(18)

1.4 Hypotheses

h1i How fast can a RT-BCI System be capable of detecting and classi-fying an action given by a motor hand movement?

h2i Can the Sample Frequency be kept low, or will problems appear according to reduced sample frequency? - Section 2.4.1

h3i What size should the Sample Window have to fit the requirements of a Real-Time Environment?

h4i Can Blind Signal Separation be able to correlate data using the 10-20 system? - Section 2.3.2 and 2.4.2

(19)

1.5 LIMITATIONS 5

1.5 Limitations

The contents of this thesis will not be completely based on a fundamental level. It is assumed that the reader has essential knowledge within neuroscience and more advanced knowledge in mathematics.

The system used in this thesis is developed only to distinguish between an action by a motor movement or rest, and the Artificial Neural Network (ANN) will be using the Multi-Layer Perceptron (MLP) for the classification.

All experimental data is taken from an offline Electroencephalographic (EEG) data file created by Forslund [5]. The file is containing samples from four partici-pants and is used to simulate an approach of a RT-BCI System. All EEG data files are based on the 10-20 System and no real sampling will be done in this thesis.

1.6 Outline

Chapter 2 - Theory

The first sections of this chapter present fundamental Theory of Anatomical Struc-ture and its Functionality.

The latter sections present an overview of a BCI System and Theory of the Algorithms.

Chapter 3 - Method

In this chapter the first section handles the System Setup and Background of the settings used in the Experiment.

Finally, the Second Section shows how the Experiment was conducted, and how the amount of different settings were reduced.

Chapter 4 - Results

In this chapter Results are presented and compared with the Hypotheses defined in the Introduction.

Chapter 5 - Discussion

A Discussion based on the Experimental Results is made in this chapter. Chapter 6 - Conclusion

The Conclusions are presented in this chapter. Chapter 7 - Further Research

(20)

Appendix A - Thesis Encyclopedia

For interested readers with lack of knowledge and without easy access to previous work in this topic, a special thesis lexicon has been developed. The purpose of the lexicon is to supply the reader with significant words used in this Thesis and to give the reader a short fundamental background.

Appendix B - RT-BCI System Development

The first section is an overview of the interactions between different functions in the Developed Software.

The second section contains all Software Source Code which were developed in Matlab6.5.

Appendix C - Result Images

Experimental Results are presented as Images. Help with interpreting the Images can be found in chapter4.

(21)

Chapter 2 Theory

The chapter consists of four major sections, videlicet Neuroanatomy, Neurophysi-ology, Hardware System, and Software System.

The first two sections describes Neuroanatomy and Neurophysiology, which are closely related where the anatomy is the foundation of the physiology. On one hand neuroanatomy describes the structure of tissue in the brain, while neurophysiology on the other hand describes functionality and relations between different tissues and areas in the brain and in the nervous system.

The third section starts with an overview of the system setup and design, and will mainly focus on Hardware used in the RT-BCI System.

The fourth and last section will handle the Software explaining the use of the algorithms.

(22)

2.1 Neuroanatomy

Neuroanatomy is a word created from ”neuro”-(neuron), ”ana”-(up) and ”tomy”-(cut ). The latter two together, ”anatomy”, means the study of the structure of the body. When interpreting all words together it means - The study of the structure of neurons in the human body.

2.1.1 Structure of the Neuron

Figure 2.1. The structure of a pyramidal neuron with propagation from dendrites to axon through the cell body.

Things that our hands touch or things touch-ing our skin make us feel heat, coldness, pain or pressure. All these different sensors have their own transmission paths through the body, where some of the paths are slower than others. If a human cut herself on a sharp knife, she may notice that the pain usually comes after a few seconds, but if she burn her hand on a hot plate, she immediately pull away. Some signals goes only to the (Spinal Cord ), e.g. when the hand is burned, while other signals go to the brain e.g. when a person’s hand gets cut. To trans-mit those signals through the body, the nervous system must be used. When the human body feels pain, the brain sends signals to the muscles to perform a movement and avoid the harm.

The nerve cells in the brain and Spinal Cord are called neurons. There are several types of neurons, but the most important one is the pyramidal neuron which has long axons reach-ing up to the scull. Let us first take a look at a pyramidal neuron before its structure will be described (see Figure 2.1).

The neuron consists of three major parts, the cell body or Soma, the dendrites and an

axon. The dendrites are connected to other neurons through Synapses and the connections between the neurons lies approximately somewhere around 104 _{or 10}5_.

The output from the cell is propagated through the axon. Some of the axons are shielded with a white fat which is called myelin. The myelin insulates the axon and makes the transmission faster than in non myelinated axons. [21]

(23)

2.1 NEUROANATOMY 9

2.1.2 Structure of the Brain

The two Hemispheres

The left side of the brain which controls the right side of the body is called the left hemisphere and the right side that controls the left side is called the right hemisphere. In Figure 2.2 different functionality of the two hemispheres can be seen. [12]

Figure 2.2. Functional differences between the two Cerebral Hemispheres. No consid-eration is taken concerning the placement of the functionality on each hemisphere. [12]

(24)

The Motor and Somatosensory Area

The brain can be classified in more specific areas, see Figure2.3, where it is divided into Lobes. Eye blinks can easily be detected by increased activity in the Occipital Lobe where the Visual Area is located. In the center of the left and right hemisphere the Auditory Area can be found. [17]

Figure 2.3. A view of the human brain, showing different areas of functionality with a close up on the Primary Motor Area and the Somatosensory Area. [5, 12]

Two important areas are studied separately in Figure 2.3, the Primary Motor Area and the Somatosensory Area. Sensory inputs are things that can be heard, seen or felt and motor outputs are mostly body movements using muscle activity. If a motor movement of a person is to be localized in the right thumb, activity in the brain will mostly be generated in the left upper part of the motor and

(25)

2.2 NEUROPHYSIOLOGY 11

2.2 Neurophysiology

In this section concentration will be on the functionality of nerves e.g. speed and concentration.

2.2.1 Functionality of the Neuron

Transmission in Nerves

Signals generated by neurons are electrical, but the communication between them is chemical. Incoming signals to the nervous system are called Afferent, and outgoing signals are called Efferent signals. The main duty of the nervous system is to collect, process and act by transmitting signals through the nervous system and there are two types of neuron tissue, gray and white, where the latter is myelinated. Three different types of nerves exist: A-fiber, B-fiber and C-fiber, where A and B have myelinated axons and C unmyelinated. The speed of the myelinated A-fibers reach from 12-130m/s, and it is the nerve with best transmission. Its thickness is 5-20µm. The nodes between myelinated shields are called Nodes of Ranvier, Figure 2.1. There is a major difference in transmission in neurons than in other cells. Messages between neurons are sent like the alphabet of Morse, which makes them robust for interference in the transmission [12]. [21]

Depolarizing and Repolarizing

The following four figures (2.4 - 2.7), show the depolarizing and repolarizing phase of the action potential of the signal propagation in axons. [21]

(26)

1 - Resting state The voltage-gated K+ and Na+ chan-nels are closed. The cell potential is -70mV [21].

Figure 2.4. Step 1 - the resting state of the neuron.

2 - Depolarizing phase Depolarization begins and opens the Na+ _{channel activation gate. The Na}+

inflow further depolarizes the mem-brane, opening more Na+ _channel

acti-vation gates. The action potential now rises up to +30mV.

(27)

2.2 NEUROPHYSIOLOGY 13 3 - Repolarizing phase The Na+ channel inactivation gates

close and K+ _{channels open.} _Now

the outflow of K+ _causes

repolariza-tion. The action potential decreases again from +30mV down to -70mV.

Figure 2.6. Step 3 - the repolarizing phase in the neuron.

4 - Repolarization con-tinues

The K+ outflow restores the resting membrane potential, and Na+

chan-nel inactivation gates open. Resting state appears when K+gates close, and the action potential has reached -70mV once again and the period can start all over again.

Figure 2.7. Step 4 - continuing repolarization in the neuron until a resting state is achieved.

(28)

2.2.2 Functionality of the Brain

The Brain related to the EEG

Millions of pyramidal cells reaching up to the scull generate sufficient potential differences, which can be measured with EEG (see Figure 2.8).

Figure 2.8. An illustration of how the pyramidal cells are generating potentials differ-ences, which can be detected with EEG.

Frequencies of the brain

Every second millions of neurons are generating potential differences on the scull. These electro neuronal signals are called brain waves. The waves consist of dif-ferent types of frequencies, which are called Delta, Theta, Alpha and Beta waves (see Figure 2.9). It is generally believed that frequencies above 40Hz convey little information. Therefore, the more interesting frequency intervals have their own name according to their anatomical functionality [1]. [22]

(29)

2.2 NEUROPHYSIOLOGY 15

Figure 2.9. Different EEG waves generated by the brain listed in order of frequency. [22]

Delta waves 1-5Hz - These waves have the lowest frequencies and occur during deep sleep.

Theta waves 4-7Hz - Occurs in children and adults experiencing emo-tional stress.

Alpha waves 8-13Hz - Normally occur in EEG’s when the person is awake and resting. It can be supplemented that these waves disappear during sleep.

Beta waves 14-30Hz - Beta waves are along with the Alpha waves the most interesting waves when using EEG on resting adults. The Beta waves generally arise in active ner-vous systems. They are generated by sensory inputs and mental activity.

(30)

2.3 Hardware System

This section starts with an overview of the whole system seen from the outside. In Figure 2.10 it can be noticed that the anatomical part is completed and this section will go further on handling the rest of the steps.

Background of the offline EEG data files

The signals or electrical impulses generated from the brain are transmitted through the electrodes and into the EEG-Recorder. In the Recorder signals are modified by a band-pass filtration and amplified.

The Continuous signal then continues into an IO Card on the Computer. Be-cause a computer only can handle Discrete values the continuous signal is converted into discrete values. The procedure of this conversion is called Sampling. The of-fline EEG data files were produced (by Forslund [5]) using the setup shown in Figure 2.10. [5]

Figure 2.10. An illustration of the offline EEG data files, starting with the EEG elec-trodes going into the EEG amplifier and further into the Computer producing the files.

(31)

2.3 HARDWARE SYSTEM 17

2.3.1 Electroencephalogram

EEG is a way of detecting brain waves. Encephalon means brain and gives us that ”Electroencephalogram” is a word for scanning electrical activity of the brain. EEG is maybe not used as frequently as ECG (Electrocardiogram), which is a way of listening to the human heart [14]. EEG signals are hard to distinguish and different sources of disturbance can make the interpretation even harder. These disturbances are called artifacts and may appear as a direct consequence of: [10]

• Heart beats • Breathing • Eye blinks • Eye movements • Swallowing • Body movements

• Bad electrode contact with the skin

• Movements of the electrode cables

When filtering the signal, frequencies between 48-62Hz should be removed, because technical equipment tend to send out waves in that specific frequency interval. To separately remove those frequencies a notch filter can be used. The notch filter does not filter frequencies below and above the specified frequency threshold. [5]

The EEG electrodes are made of metal, and are covered with a thin layer of silver chloride. They are attached on the head and a paste is often used letting the electrodes attach to the scalp. The contact with the scull is very important and the impedance should not exceed 10kΩ. [5]

Nowadays a lot of new techniques are developed for examination of the brain, like the power of functional Magnetic Resonance Imaging (fMRI) or Computer Tomography (CT). fMRI does not hold up to the temporal resolution necessary for measuring signals in Real-Time and cannot be used in a Real-Time environment [13]. CT is cut out because radiation would give more harm than the benefits gained from the BCI System. EEG which is a very old method, is still most suited for measuring signals of the brain, and in small time intervals around one second the EEG can be considered stationary. [16]

(32)

2.3.2 The 10-20 System

The 10-20 System is a standard way of placing electrodes on the scull, see Fig-ure 2.11. This standard was developed so results from different scientists easily could be compared.

The electrodes according to the 10-20 System are distributed quite homogenous over the whole scull where each electrode is given a specific name. [5]

(33)

2.3 HARDWARE SYSTEM 19

2.3.3 Optimal Data Representation

Sampling a Signal

Figure 2.12. The sampling procedure of transforming a Continuous Signal into a Discrete.

The most common way of transforming a continuous signal into a discrete signal is made by equidistant sampling (see Fig-ure2.12). Values in the continuous signal are read at periodic sample intervals nT, where n ∈ Z. The distance T between two sam-ples is called sampling interval, and the re-ciprocal quantity is named sample frequency. Equidistant sampling can mathematically be expressed by the Nyquist theorem fs ≥ 2f0

where fs is the sample frequency. The

sam-ple frequency has to be twice as high as the highest frequency component f0 in the given

signal, for not loosing required frequencies in the continuous signal. Sampling in a fi-nite time interval gives us a Sample Window, which will be used frequently further in this thesis. [20]

Representing Data

When talking about speed and computers it is important to work with data representa-tions (numbers) given by the power of two. Instead of sampling with 35Hz, 32Hz should be used and 256Hz instead of 260Hz.

To motivate the given statement the

foundation of the computer has to be considered. All computers works with binary numbers, which means working with ones and zeros. There are several benefits working with numbers perfectly suited for the computer.

The FFT (section 2.4.3) is based on working with numbers by the power of two, and working with other representations would only make the FFT perform worse. Another example is when dividing or multiplying numbers with 2, 4, 8 etc. it is optimal for a computer just to make an arithmetic shift to the right, one, two or more times. A division by 2 equals shifting the binary number one step to the right and a multiplication by 4 equals shifting the number two steps to the left. When shifting the binary number one step to the right or left a zero always has to fill up the empty space caused by the shift. Consider the number 11002 = 1210, by

moving the number one or two shifts to the right 01102 = 610 and 00112 = 310 will

appear. It can be noticed that 01102 got one zero filled on the left side, and 00112

(34)

2.3.4 Brain-Computer Interface

Figure 2.13. Illustration of how signals can be hidden in a set of samples.

A BCI System is a system that can learn to recognize patterns of increased activity in local areas of the brain. The system cannot read minds, but it can be trained to distinguish between different actions made by a person. The brain is filled with signals that can be interpretable. This is a quite complex problem which is illustrated in Figure 2.13. In the two figures consisting of the iden-tical data set, it is shown that by shifting the view an interpretation of the information in the given set can be made. Even more information can be hidden in the set, in Figure2.13there is not only a W hiding, but also a H and an A (the latter two not shown). Those symbols will only appear if the view is set into the right dimensions. In a similar way as illustrated above our RT-BCI Sys-tem will use algorithms and ANN to find interpretable information in the EEG signal.

2.3.5 Real-Time System

A Real-Time System (RTS) is defined as: A system that reacts on external events and execute functions based on those events giving an output within a predefined period of time. Correct behavior depends not only on correct results, but also on at which point in time the result is produced. [15]. If a BCI System can fulfill these RTS requirements it can be handling Real-Time Classification (RTC).

Hard and Soft Real-Time Systems

RTS’s can be divided into Hard and Soft. This separation has do with how impor-tant it is for the RTS producing correct results at the right time. It can be defined as follows:

1. A Hard RTS is a RTS where the cost to fulfill the functional and temporal requirements is very high.

2. A Soft RTS is a RTS that sometimes can tolerate to not fulfill the functional and temporal requirements respectively. It means that a requirement can be missed or a service can be performed a bit to late. Often there is a higher limit of what the system can tolerate.

(35)

2.3 HARDWARE SYSTEM 21 • A Hard RTS: When driving a car, the processor in the car could probably

be handling your: ABS System, servo steering, breaks, and cruise map at the same time. When breaking suddenly, the processor cannot under any circumstances reject the ABS System, while working with the cruise map letting a person crash. The RTS has to have resources handling all important actions at the right time.

• A Soft RTS: When using a cellular phone, the phone is operating through local base stations, provided by the owner of the current cellular network. Sometimes many people may be calling at the same time during a carnival or in another crowded place making the local base stations overloaded. The problem can be solved without overloading the base stations letting people wait with their calls. Here it is okay for the RTS in the base stations to reject new calls.

Given by the scenarios it can be noticed that a RT-BCI System can be either a Hard or a Soft RTS, depending on requirements of the area it will be implemented in. If the RT-BCI System should be implemented to steer an airplane completely, it should be implemented as a Hard RTS, but if its function would be to prepare the airplane system for fast turns, it could be implemented as a Soft RTS. Nothing would happen if the system should miss to prepare for the turn, the pilot will make the turn without the prepared system. Each turn has to be confirmed by the pilot before turning.

(36)

2.4 Software System

When talking about ”the system” further on in this thesis, it will always refer to the RT-BCI System.

2.4.1 Time and Frequency Modulation

When using offline EEG data it is important to be able to classify data at

different time intervals in that file. Functions are needed for not only cutting out time intervals, but also modifying the sample frequency in the file. If an EEG file is sampled at 256Hz, the file can easily be modified into 64Hz, just by keeping every fourth sample and throwing the rest of the samples away. When down sampling the signal from 256Hz into 64Hz important information can be lost and disturbance in the file can occur. To prevent disturbances from high frequencies that remain in the down sampled file a Butterworth Filter can be used, to filter those high frequencies away before the down sampling is made.

In using an offline EEG file, the ability of removing separate channels in the file must exist. If a specific electrode in the file is used to detect eye blinks e.g. the O1 or O2 electrode, the channel of that electrode should be removed before using the EEG file for any experimental data. All modifications in an EEG file as given above will be referred to as Time and Frequency Modulation (TFM).

2.4.2 Blind Signal Separation

Blind Signal Separation is usually related to when talking about the ”Cocktail problem”. The Cocktail problem is an example of the brain discerning a specific person’s voice at a crowded Cocktail party. The problem can also be described mathematically finding source signals given linear combinations of mixed signals. If a person concentrate on a specific conversation between two people talking to each other at the cocktail party, the person automatically can filter all other conver-sations and noise away just hearing the two persons talking. By placing out several microphones at the party BSS can accomplish the same filtration mathematically as the brain do in practice.

Listening to brain waves with EEG using BBS, the waves could be described as the conversations at the party, EEG electrodes as the microphones and BSS as the filtration in the brain.

Canonical Correlation Analysis

BSS can be based on Canonical Correlation Analysis which was developed by Hotelling[8] 1933. The algorithm has since then been used in medical studies, economics and meteorology, but surprisingly it has been unknown in the fields of Signal Processing and Learning.

(37)

2.4 SOFTWARE SYSTEM 23

CCA is a way of measuring linear relationships between two multidimensional variables. CCA finds two basis vectors, one for each signal that are optimal with respect to correlation. The correlation matrix between the two signals is maximized on the diagonal letting the dimensionality of the new bases be equal or less than the smallest dimensionality of the two signals. The most powerful thing about CCA is that it is Invariant against affine transformations (i.e. Scaling, Rotation and Translation).

Let x and y be two multidimensional signals with not necessary the same dimensionality. The mean value is subtracted from each signal and gives us: E[x] = E[y] = 0. If the two linear combinations of the two signals are consid-ered respectively it gives that x = xT_w

x and y = yTwy. The function to be

maximized can be written as:

ρ = E[xy] pE[x2_]E[y2_] = E[wT_xxyTwy] q E[wT xxxTwx]E[wTyyyTwy] = w T xCxywy q wT xCxxwxwTyCyyWy (2.1) Maximizing equation 2.1 with respect to wx and wy, is setting

∂ρ ∂wx

= ∂ρ

∂wy

= 0

The canonical correlations between x and y can be found in these following eigen-value equations, where only one equation needs to be solved:

C−1_xxCxyC−1yyCyxwx = ρ2wx (2.2)

C−1_yyCyxC−1xxCxywy = ρ2wy

ρ2 _{is the squared canonical correlation representing the eigenvalue of each}

respec-tive equation, and wx and wy are the eigenvectors representing the basis vectors.

The number of non zero solutions to equation 2.2 is limited to the smallest di-mensionality of x and y. E.g. if a didi-mensionality for x is 3 and y is 4 then the maximum of canonical correlations will be 3.

A CCA based BSS method

In finding signals of the brain BSS is used as in Figure 2.14. s(t) is a multidimen-sional signal which can be written as:

s(t) = s1(t) s2(t)

(38)

Figure 2.14. An illustration of a CCA based BSS method.

Let us say that two mixes of signals are given, s1(t) and s2(t), CCA will then find

two linear combinations, x(t) = as1(t) + bs2(t) and y(t) = cs1(t) + ds2(t), were a

and b are constants. x(t) and y(t) correlates maximal with x(t − 1) and y(t − 1). The autocorrelation will be maximized in one time step. It is therefore fair that BSS should reconstruct signals to their original values. The mixes of signals contain less autocorrelation than original signals. It can be noticed that CCA is mathematically quite similar to Principal Component Analysis (PCA) which also is referred to as the Karhunen-Loeve transform. The main difference is that PCA finds the maximum variance in one multidimensional signal and CCA finds the maximized correlation in a two dimensional signal. [2]

2.4.3 Fast Fourier Transform - FFT

Fast Fourier Transforms are algorithms using symmetry and smart implementations of the Fourier Transform seen below:

F (w) = Z ∞

−∞

x(t)e−jωtdt

The FFT is not a real transform, it is an algorithm and there are several different algorithms for calculating the FFT. The FFT is based on a sample window being infinite, but that problem can be solved using a smooth filter in the sampling window. Letting the endpoints in the sample window approach zero a sample window could be treated as infinite (see Figure 2.15). [20]

Figure 2.15. Left: A sample window with a filter applied. Right: The sample window after filtration. The samples approach zero at the endpoints.

(39)

2.4.4 Artificial Neural Networks

Before a system can be trained with the ANN, data has to be modified fitting into the network representation. The conversion of an input data matrix can be done as follows: Let us now assume that the file above is representing the interval between

Figure 2.16. A representation of the data matrix that is going to be converted.

0-250ms sampled at 64Hz, giving us 16 samples from that 250ms big window (see Figure 2.16). Usually the sampling is divided into smaller windows in that same time interval. In the figure below, three windows are represented using a window size of eight and a window step by four samples. A window step defines after how many samples a new window is started, and it can be noticed that if the window step is smaller than the window size a sample window is overlapped. See Figure 2.17 to see how the setting of window size and window step affects a data matrix.

(40)

Figure 2.17. An illustration of how the channels are divided into windows.

To suit the ANN, windows are constructed into training vectors as seen in

Figure 2.18. Each vector is representing an ANN training example constituted by the first window in each channel. In the figure the windows have been numbered to simplify the understanding of the conversion.

(41)

The input vector to the ANN consists of the samples x1...xi illustrated in

Figure 2.19. The network consists of two layers, the first layer is a hidden layer (Whidden) and the second an output layer (Wout). The output from the network is

given by Z1 and Z2, representing two different classes.

Figure 2.19. A MLP with the input dimension Xi, a hidden layer Whidden and an output

layer Wout distinguishing between two classes, Z1 and Z2.

An Artificial Neural Network is a network that is supposed to work like the neurons in our head. Each neuron is connected to other neurons. In a human brain each neuron can be connected to 104 _{other neurons. ANN separate between}

classes using weights. Each node in the network which represent a neuron

junction, have its own weight. The weights are then updated for each example in the training session. If an example belongs to class A then the weights are

adjusted in A:s direction and if the example belongs to class B, the weights are updated in B:s direction. The step in the updating procedure of the weights is controlled by setting the step variable η. The ANN also has the ability to train with an example more than one time. There is a parameter called epoches which works as a counter letting the weights be updated several times for a training example. [7]

(42)

Testing the System

After the training session the weights are tested on new examples. By multiplying the weights with an input example, the example can be distinguished into the right class. An output of the ANN is given by ”1” or ”2” where ”1” is representing class Z1 and ”2” class Z2. Figure 2.20 is an illustration of how Z1 and Z2 can be

separated. Each ”•” and ”x” represent a test example classified through the ANN. As seen in the picture some of the examples are distinguished incorrect.

Figure 2.20. An illustration of how two classes can be separated from each other using an ANN.

(43)

Chapter 3 Method

The theory presented in the previous chapter has constituted a fundamental back-ground needed for understanding the following experimental part in this thesis. This chapter will concentrate on explaining the methods used in the experiment. First the background of the experimental data will be explained followed by the experiment setup.

(44)

3.1 System Setup

Before going through the setup used in the experiment, a background has to be given of how the offline EEG data file was created and which electrodes in the 10-20 System were used.

3.1.1 The EEG Data File

In Table3.1it can be seen how the electrodes were placed when creating the offline EEG file. The relation between channels and electrodes can be viewed in Figure3.1

Table 3.1. A table of settings and connections between the channels and electrodes.

Figure 3.1. An illustration of the displacement of the EEG electrodes used in the offline EEG file.

(45)

3.1 SYSTEM SETUP 31 The experiment is based on the data files produced by Forslund[5]. The data files contain real EEG data from four participants. The sampling rate used was 256Hz and 7 EEG channels were available in each file. Cz and C4 were used as reference nodes, X and Y as EOG (Electrooculogram) electrodes for detection of eye blinks. The amplifier used was a Grass model 12B EEG amplifier which had a built in band-pass filter. The filter was set between 0.3 and 30 Hz, but the filter was not close to an Ideal filter. All test files used in our experiment are based on those EEG files containing real body movements. The movements (actions) were made by moving a joystick in four directions, up, down, left and right or not moving it at all (rest). Each file consists of one of the described actions sampled from -1 second to 7 seconds. The sampling started one second before a direction was presented for the participant. Each participant has a label given by the

participants position, sex, age and if the participant was left or righthanded (see Table 3.2). Twenty files (examples) of each class have been used. The first 13 examples of respective class have been used to train the network, and the latter 7 to test the network. [5]

Table 3.2. Data of the participants creating the offline EEG file.

3.1.2 The Developed Software

The main file in the software is the ”Experiment file” where all different settings are specified. One by one each setting is tested by calling the ”Test file”. The test file is handling the training and testing session. It also handles the images that are presenting the results. The test file calls the ”Train BCI” and ”Test BCI” file which are handling the algorithms in the software. The algorithms are divided into separate functions as seen below. Both the training and testing session of the software use the same functions.

(46)

1-TFM Input: Matrix

Output: Matrix

Modification: Channels can be removed, a specified time interval can be cut out and down sampling can be performed using a but-terworth filter

2-BSS Input: Matrix

Output: Matrix

Modification: Finds the linear combinations in the set of mixed signals and returns data in or-der of best correlation

3-MakeWindows Input: Matrix

Output: AnnMatrix

Modification: Takes the first window from each channel and creates an input vector suited for the ANN. The next set of windows given by the step size and proceeds until all win-dows have been converted

4-FFT Input: ANNmatrix (In time)

Output: ANNmatrix (In frequency)

Modification: Applies the FFT on each window sepa-rately

5-ANN Input: ANNmatrix

Output: Matrix containing results

Modification: Uses the backpropagation algorithm to train the network updating the weights in the training session. In the test session the weights are tested on new examples of data

3.2 Experiment setup

A set of settings is defined for each variable parameter and the parameters are the following:

D = Setstart, Setlength, Setfrequency, Setfilter, SetBSS, SetFFT, eta=η, epoches, Wsize, Wstep, Channels

Each of the configuration parameters can be set to different values. The settings of the parameters in set D will now be described:

(47)

3.2 EXPERIMENT SETUP 33

S = Setstart [ 0, 0.125, 0.1875] (Seconds)

L = Setlength [0.25, 0.125, 0.0625] (Seconds) F = Setfrequency [32, 32, 64, 64, 256] (Hz)

fil = Setfilter [0 , 1, 0, 1, 0] (Boolean)

BSS = SetBSS [0, 0, 4, 4] (0 to Channels)

FFT = SetFFT [0, 1, 0, 1] (Boolean)

eta [0.001] (Will be set to 0.001 as a constant)

epoches [3000] (Will be set to 3000 as a constant)

Wsize [1/1, 1/2, 1/2, 1/2]

Wstep [1/1, 1/2 ,1/4, 1/8]

Channels [1 1 1 1 1 1 1 0] (Boolean)

Explanations

-S - -Start point in the file, in seconds L - Length of input, in seconds F - Sample frequency, in Hertz

fil - Off or On, a Butterworth filter of 5:th order Wsize - Split Length into windows

Wstep - Windows overlapping, in fraction of a whole window BSS - Off if 0, or set to number of returned dimensions FFT - Off or On, The Fast Fourier Transform

eta - A constant. Step size for updating the ANN weights

epoches - A constant, number of times an example updates an ANN weight Channels - 1 1 1 1 1 1 1 0, The ”visual” channel is removed

3.2.1 Optimization of Experimental Settings

The settings described above can be combined in many different ways giving us a complex Optimization problem:

Start = 3 Length = 3 Frequency = 3 BSS = 2 FFT = 2 epoches = 1 eta = 1 Window size = 4 Window step = 4 Filter = 2 Channels = 1

(48)

Simplification of the optimization problem

Because all different combinations of the input settings would give us 3456 com-binations, it is of our concern to reduce and to find the most interesting settings suited for our hypotheses.

h1i - How fast can a RT-BCI System be capable of detecting and classifying an action given by a motor hand movement? This hypothesis is depending on the start and the length parameter. The reduction of the two parameters can be done according to these combinations:

Start = 0ms and Length = 250ms Start = 125ms and Length = 125ms Start = 187.5ms and Length = 62.5ms

From the six combinations the problem is simplified into three interesting combi-nations, with the limitation to not exceed 250ms or 0.25 (1/22) seconds as used in the software.

h2i - Can the Sample Frequency be kept low, or will problems appear according to reduced sample frequency? The frequency can be either filtered or not filtered through a Butterworth filter. The file with 256Hz needs no filtration and the fol-lowing combinations can be found:

Frequency = 32Hz and Filter = 0; Frequency = 32Hz and Filter = 1; Frequency = 64Hz and Filter = 0; Frequency = 64Hz and Filter = 1; Frequency = 256Hz and Filter = 0;

The problem is simplified from six into five combinations.

h3i - What size should the Sample Window have to fit the requirements of a Real-Time Environment? The window size and window step will be set as follows: Window size = 1/1 and Window step = 1/1

Window size = 1/2 and Window step = 1/2 Window size = 1/2 and Window step = 1/4 Window size = 1/2 and Window step = 1/8

The window will be divided giving us 1 window, 2 windows, 3 windows and 5 win-dows.

h4i - Can Blind Signal Separation be able to correlate data using the

10-20 System? All four combinations between the BSS and the FFT will be used: BSS = 0, FFT = 0

BSS = 0, FFT = 1 BSS = 1, FFT = 0 BSS = 1, FFT = 1

(49)

Chapter 4 Results

(50)

Interpretation of the Results

(a) Illustration of the interpretation of the result image with this figure represent-ing one window.

(b) Illustration of the interpretation of the result image with this figure represent-ing three windows. The mid window is representing an overlapped window.

Figure 4.1. Illustration of how the image output should be interpreted.

Instead of viewing lots of numbers, the results will be displayed as images, having the struc-ture as seen in Figure 4.1.

First Figure 4.1(a) will be explained and it can be seen that the image contains two parts, one black and one white. For all pic-tures, black is assigned to represent class1 and white class2. Looking at the y-axis numbers can be seen reaching from 1-8. These num-bers represent the EEG test files (examples). In the real experiment 13 examples of each class were used in the training session. In all im-ages ”black” will always represent Rest, and will be compared with four separate actions for each person. The actions used are Up, Down, Left or Right. If a look is taken at the top of Figure 4.1(a) the time interval is seen. From the method described in this thesis it is known that the interval can be either 0-250ms, 125ms-250ms or 187.5ms-125ms-250ms. For example, if the time interval is set to 250ms and the frequency to 64Hz or 62.5ms with 256Hz, 16 samples will be given. Time (in seconds) and frequency can be multiplied giving the amount of samples. The samples are shown at the bottom of Fig-ure 4.1(a).

Figure 4.1(b) do not look so much different from Figure 4.1(a). This second figure is an illustration of using an overlapped window. The window size in Fig-ure 4.1(b) is set to 8, and the window step to 4. The samples in the first sub window are reaching from 1-8, by the sample step of 4 the last four samples from the first sub window reaching from 5-8 and the first four samples from the last sub 9-12 window are combined creating an overlapped window. The last sub window is reaching from 9-16 samples. If a window step is set to be identical or greater than the window size, only one window is produced.

A white window on the black half represents an incorrect output given from the ANN, and vice versa. When using sub windows the incorrect output do not always mean that the result is bad. If the column to the right is correct, it can be concluded that the ANN cannot classify the signal correct until the second half of the interval. Remember that the column in the middle is the overlapped (virtual) sub window.

(51)

37

Experimental Results

In this section some simulated results will be compared with the hypotheses. The most interesting result images can be found in Appendix C.

h1i How fast can a RT-BCI System be capable of detecting and classifying an action given by a motor hand movement?

As seen in Figure 4.2 for participant A-W29R, the result can be compared with the result given in our interpretation example in Figure 4.1(b). The columns are getting slightly better, and the last column is quite good, showing that the system should be able to detect a reliable output somewhere after 125ms with those cur-rent settings.

Figure 4.2. A result of how the output is improved in the second window.

h2i Can the Sample Frequency be kept low, or will problems appear according to reduced sample frequency?

As seen in Figure 4.3 a down sampling to 32Hz could be used with a satisfying accuracy in the ANN. The first three participants show quite good results with the current settings shown in the figure. In fact both our figures of the result images in this chapter have been using the frequency of 32Hz.

(52)

Figure 4.3. A result of high correctness in the classification.

When implementing the system it can be noticed that there is a software limitation. The limitation forces us to use a window size of four samples as a minimum. The FFT algorithm will not be able to handle data with fewer samples than described. A special constraint in the developed software controls this limitation of the FFT.

h4i Can Blind Signal Separation be able to correlate data using the 10-20 System?

The output images are more or less inconsistent when using CCA together with the FFT. There can be several explanations for that which will discussed in the next chapter.

(53)

Chapter 5 Discussion

(54)

Result Discussion

A brief discussion of the hypotheses will be made concerning the results:

h1i It seems like an improved classification can be made on the second half of the 250ms time interval. This is because the activity in the brain needs some time to generate the increased activity so it can be detected. More research has to be done at the interval of 125-250ms and 187.5-250ms. Hopefully the interval can be set somewhere between 125-187.5ms, but this interval has not been tested in this thesis. h2i A low frequency should reduce the amount of data, but in the implemen-tation, problems occurred. A low frequency gave too few samples in the sample window, so the FFT could not handle the data. The only solution to this problem would be to keep the sample frequency at 128Hz or 256Hz.

h3i The resolution of the reaction time of the system is strongly determined by the sample window size and the window step. A problem is that a small window cannot fit a whole period of low frequencies in the window. The sample frequency and window width are dependent on each other. There is a problem when sam-ple windows are getting too small. The first problem was already related to low frequency, but there is one more problem.

To get a whole period of a certain frequency in the interval of 62.5ms the minimum frequency is 16Hz. A frequency of 4Hz have four periods in a second and therefore requires at least 250ms to fit a whole period of that frequency. By breaking down the time interval into small sample windows as done in the software, the lowest frequencies will not fit into the sample window. Using alfa and beta frequencies, the time interval can be set to 62.5ms detecting whole periods of 16Hz, see Figure 5.1 for an illustration of the problem described. The settings of 250ms, 125ms, 62.5ms and 31.25ms can be used according to optimal data representation.

Figure 5.1. Illustration of the problem where the sample window is too small. Important low frequencies cannot fit a whole period in the sample window.

(55)

41 The frequency should be quite high and set to 128Hz or 256Hz. The benefit of doing that is that more samples will be generated for the FFT operation. The same size of a window measured in time will contain more samples, and letting us make the sample windows smaller.

h4i There may be different explanations of why the BSS together with the FFT gave so inconsistent data. One of the answers to our fourth hypothesis could be that the BSS cannot handle the 10-20 System. By placing electrodes too close to each other could result in making the signals in the electrodes too equal, giving the BSS too little information. Spreading the electrodes around the motor and somatosensory area could perhaps improve the results.

Method discussion

The methods and algorithms used in this thesis are not optimal. The ANN can for instance, be improved by letting each sample window have its own weights. All windows of the same action had the same weights and that could have caused the ANN to perform worse than it should. The problem arise because the frequencies are not the same in all windows in the time domain, but they all update the same weights.

The number of training examples were also too small. An ANN should have 10 times more examples than its input dimensionality. In the experiment the examples should have been many more. The lack of examples could be one of the reasons of the inconsistent results given by the ANN. The methods and algorithms used in this thesis could also be combined with other known methods for a better performance. Further research in this topic may give us the answer.

The network could also have been modified using the Matrix representation in the input instead of using the ANNmatrix representation. The ”MakeWindows” conversion for transforming the Matrix into an ANNmatrix could then be simplified.

(56)

(57)

Chapter 6 Conclusion

(58)

Conclusion

Conclusions based on the hypotheses:

h1i How fast can a RT-BCI System be capable of detecting and classifying an action given by a motor hand movement?

It seems as the system can find signals after 125ms, but this interval should be in-vestigated more. A specific area not inin-vestigated in this thesis could be the interval of 125ms to 187.5ms.

h2i Can the Sample Frequency be kept low, or will problems appear according to reduced sample frequency?

Good results with 32Hz has been achieved, but a higher sample frequency is sug-gested to help keeping the sample window small in the time domain.

Limitations by the system and the lack of whole periods of low frequencies, re-stricted us to keep a window minimum of four samples. In the time domain a good window size seems to be around 31.25-62.5ms. Overlapped windows should not be used, a single window placed in the right time interval seems to be the best.

h4i Can Blind Signal Separation be able to correlate data using the 10-20 System?

No consistency in the results using BSS with the FFT could be seen. The offline EEG file based on the 10-20 System could make it hard for BSS to find correlation because of the distances between the electrodes. Given from the theory electrodes could perhaps be placed on or nearby the motor area on the opposite side on the scalp relative to the performed body movement.

(59)

45

Derivation of the best settings

Given the results and theory in this thesis, the best settings can be empirically derived:

Optimal Data Representation

By the definition of optimal data representation numbers used in a computer should be given by the power of two. This first limitation gives us the following possible intervals within 250ms:

1. - 250ms 2. - 125ms 3. - 62.5ms 4. - 31.25ms

Time interval and Window Size

In the set above only one interval suits the requirements of a RT-BCI System. Because no interesting signals are found within the interval of 0-125ms, the sam-pling in the training session has to start after 125ms. The first and the second intervals are too long for sampling and processing data giving an output within the predefined period of time (250ms). The last window is too short according to the requirements of the algorithms used. The only interval suited for handling RTC in the system is the interval of 62.5ms. The interval is small enough to be applied somewhere in the sample interval between 125ms and 250ms. Overlapped windows will no longer be needed because of the small sample interval.

Frequencies

By using the 62.5ms big sample window frequencies below 16Hz will have a lack of whole periods in the window. As given from the theory, Beta waves with the most important waves reaching from 14-30Hz will be sufficiently covered. Our suggested sample window can almost fulfill the requirements of that constraint too, and frequencies from at least 16Hz will be detected using the 62.5ms sample window.

BSS and FFT

When using the algorithms there were requirements of having enough data given by the sample window. The sample frequency was related to the sample window and sampling with 256Hz would give 16 samples in a window of 62.5ms. This amount of data is enough to make the algorithms work properly. Sampling with 128Hz could also be an option, giving eight samples.

(60)

Recommended RT-BCI settings

The proposal settings derived from above can be concluded as:

Time interval: 125-250ms

Sample frequency: 256Hz

Window size: 62.5ms

Algorithms: BSS and FFT

By using the settings above a properly working RT-BCI System using RTC could be in reach.

(61)

Chapter 7 Further Research

(62)

Further Research

As mentioned BSS had problems finding correlated signals in the EEG-files. Fur-ther research can be done by placing the electrodes more selectively and see if the algorithm will be able to detect the EEG waves better. Electrodes can, for instance, be placed near the motor and somatosensory area.

All sample windows representing the same action could be implemented having their own weights in the ANN instead of letting the sample windows share weights. Other research could be concentrating on the time interval 125ms to 250ms where an interesting signal is appearing.

In a matter of time perhaps ”new knowledge” easily could be downloaded into the human mind by a reversed BCI System, and twenty years of going to school would maybe no longer be necessary. As mentioned before, new technologies has to be developed with consideration of human integrities.

(63)

Bibliography

[1] Anderson, C., Devulapalli, S., and Stolz, E. EEG signal classification with different signal representations. IEEE Service Center (1995), 475–483. [2] Borga, M., Knutsson, H., and Landelius, T. A unified approach to

PCA, PLS, MLR and CCA. Tech. Rep. LiTH-ISY-R ; 1992, Department of Electrical Engineering, Link¨oping University, Link¨oping, Sweden, 1997. [3] Clark, A. Natural-Born Cyborgs - Minds, Technologies, and the Future

Human Intelligence. Oxford University Press, 2003.

[4] Collins, H. The Golem - What you should know about science, 2nd ed. No. ISBN: 0521645506. Cambridge University Press, 1998.

[5] Forslund, P. A neural network based brain-computer interface for classifica-tion of movement related eeg. Master’s thesis LiTH-IKP-EX-2107, Department of Electrical Engineering, Link¨oping University, Link¨oping, Sweden, december 2003.

[6] Grant, R., Timothy, R., and McMillan. Nonconventional Controls -Handbook of Human Factors and Ergonomics. No. ISBN: 0471116904. John Wiley and Sons, 1997.

[7] Haykin, S. Neural Networks - A comprehensive foundation, 2nd ed. Tom Robbins, 1999. pages 156-248.

[8] Hotelling, H. Analysis of a complex of statistical variables into principal components. Journal of Educational Psychology (1933), 498–520. 24:417-441. [9] Jasper, H. The ten-twenty electrode system of the international federation.

Electroencephalography and Clinical Neurophysiology, 1958. 10:371-375. [10] Johansson, O. A user-friendly system for presentation of eeg signals in the

time and frequency domain. Master’s thesis ULi-IMT-EX-201, Department of Biomedical Engineering, Link¨oping University, Link¨oping, Sweden, december 2001. In Swedish.

(64)

[11] Karlsson, M. Brain computer interface: A study of eeg-based control in fighter aeroplanes. Master’s thesis LiU-IMT-EX-273, Department of Biomedi-cal Engineering, Link¨oping University, Link¨oping, Sweden, november 1998. In Swedish.

[12] Larsson, L. Neurofysiologi - En bok om hur hj¨arnan fungerar. Studentlit-teratur, Lund, Sweden, 2000. In Swedish.

[13] Lee, T. Independent Component Analysis - Theory and Applications. No. ISBN: 0-7923-8261-7. 1998. page 146.

[14] Nordin, M. Classification of ecg signals using canonical correlation analy-sis. Master’s thesis LiU-IMT-EX-316, Department of Biomedical Engineering, Link¨oping University, Link¨oping, Sweden, december 2001.

[15] Norström, C., Sandström, K., Mäki-Turja, J., Hansson, H., Thane, H., and Gustafsson, J. Robusta Realtidssystem. Mälardalen Real-Time Research Centre, Väster˚as, Sweden, november 2000. In Swedish, pages 14-17.

[16] Pardey, J., Roberts, S., and Tarassenko, L. A review of parametric modelling techniques for eeg analysis. Medical Engineering and Physics (1996). 18(1):2-11.

[17] Penfield, W., and Rasmussen, T. The Cerebral Cortex of Man. Macmil-lan, New York, 1950.

[18] Russel, S., and Norvig, P. Artificial Intelligence - A modern approach. Wiley, 1995.

[19] S¨oderkvist, S. Tidskontinuerliga Signaler och System, 2nd ed. Tryckeriet Erik Larsson AB : Link¨oping, 1993. In Swedish, pages 17-21.

[20] S¨oderkvist, S. Tidsdiskreta Signaler och System, 2nd ed. Tryckeriet Erik Larsson AB : Link¨oping, 1994. In Swedish.

[21] Torota, G., and Grabowski, S. Principles of anatomy and physiology, 10th ed. Wiley, 2003. Pages 385-418.

[22] Torota, G., and Grabowski, S. Principles of anatomy and physiology, 10th ed. Wiley, 2003. Pages 452-497.

(65)

Appendix A

Thesis Encyclopedia

ABS Stands for Anti Blocking System, and is an

elec-tronic break system used in cars preventing breaks from locking.

Algorithm Broadly defined, an algorithm is an interpretable

finite set of instructions dealing with contingencies and accomplishing some task which can be any-thing that has a recognizable end-state, end-point, or result for all inputs. Algorithms often have steps that repeat (iterate) or require decisions (logic and comparison) until the task is completed. [23]

ALS Amyotrophic Lateral Sclerosis, also known as Lou

Gehrig’s Disease. It is an incurable, neurological disease that affects the ability to control muscles and gradually leads to paralysis. Does not affect the intellect of the brain, but it ceases the nerve cells. [5]

Anterior The front of the human head, organ or body. [11]

Arithmetic Shift Right Moves the register in Arithmetic Logic Unit(ALU) in the processor one step to the right:

01102

00112

From the right to the left a binary number in-creases with the power of two when converting to a decimal number: 01102 = 0*23+1*22+1*21+0*20 = 0+4+2+0 = 610. 00112 = 0*23+0*22+1*21+1*20 = 0+0+2+1 = 310. 51

(66)

Artificial Intelligence Some of the definitions: ”The study of the compu-tations that make it possible to perceive, reason, and act” ”The art of creating machines that form functions that require intelligence when per-formed by people” or in short terms, systems that think and act like humans. [18]

Butterworth Filter A filtration of a continuous or discrete signal. The filter has an order of steepness, see Ideal Filter. [20]

Continuous Signal A continuous signal can be compared with a smooth wave on the ocean. Every atom in the wave is connected to another atom in a smooth way. [19]

CT Computer Tomography - A survey based on X-rays

by creating slices of the human body, and then put into a three dimensional picture of the tissues inside the human body. High doses of X-rays is dangerous.

Cyborg Manfred Clynes first suggested the acronym

”Cy-borg” which stands for Cybernetic Organism or Cybernetically Controlled Organism. A cyborg is a human-machine hybrid with physical merging of flesh and electronic circuitry. [3]

Discrete Signal A discrete signal is constituted of single points, with jumps and spaces among adjacent values.

EEG Means Electroencephalogram and was introduced

by the German psychiatrist Hans Berger in 1924 to try to locate the source of epilepsy. Encephalon means brain in ”Latin”, and the method was de-veloped to measure activity of the brain, without opening the scull. [11]

Electromyogram Measuring electric activity in muscular tissues. [11]

fMRI Functional Magnetic Resonance Tomography - A

method of imaging tissues in the human body with electro magnetic waves based on the spin of the atom nucleus.

Ideal Filter A filter that is cutting the frequencies exactly at a requested place. In practice a continuous filter can not cut exactly at a threshold. [20]

(67)

53

IO Card An Input-Output card used to make the computer

communicate with continuous signals, in our case signals from the EEG-recorder. When collecting data through the input connection it transforms analog signals to discrete by sampling the signal.

Myelin A white fat twisted around the nerve, which isolate

the nerve and makes it transmit signals faster than non myelinated nerves. [21]

Neuron Cell used in the brain and the central nervous

sys-tem. [21]

Perceptron A Perceptron is a possessing element with a

num-ber of input and outputs. [5]

Posterior Means the back of the human head, organ or body.

[11]

Post hoc Means ”after this” and can be explained in words

like: One event causes another, simply because the proposed cause occurred before the proposed ef-fect. [4]

Real-Time System It is a system that reacts on external events and execute functions based on those events giving an output within a predefined period of time. [15]

Synapse Connection between nerve cells, where the

(68)

(69)

Appendix B

RT-BCI System Development

B.1 Program Overview

An overview of how functions in the program interact with each other. The prop-agation can be followed by the numbers assigned to the functions:

Figure B.1. Propagation through the implemented matlab program.

A Real-Time Classification approach of a Human Brain-Computer Interface based on Movement Related Electroencephalogram