Temporal representation of Motor Imagery : towards improved Brain-Computer Interface-based strokerehabilitation

(1)

Brain-Computer Interface-based stroke

rehabilitation

TE M P O R A L R EP R ES EN TA TIO N O F M O TO R I M A G ER Y - T O W A R D S I M P R O V ED B R A IN -C O M PU TE R I N TE R FA C E-B A SE D S TR O K E R EH A B IL ITA TIO N 202 1 ISBN 978-91-7485-495-4 ISSN 1651-9256

Address: P.O. Box 883, SE-721 23 Västerås. Sweden Address: P.O. Box 325, SE-631 05 Eskilstuna. Sweden E-mail: info@mdh.se Web: www.mdh.se

(2)

Mälardalen University Press Licentiate Theses No. 301

TEMPORAL REPRESENTATION OF MOTOR IMAGERY

TOWARDS IMPROVED BRAIN-COMPUTER

INTERFACE-BASED STROKEREHABILITATION

Jonatan Tidare 2021

School of Innovation, Design and Engineering

Mälardalen University Press Licentiate Theses No. 301

TEMPORAL REPRESENTATION OF MOTOR IMAGERY

TOWARDS IMPROVED BRAIN-COMPUTER

INTERFACE-BASED STROKEREHABILITATION

Jonatan Tidare 2021

(3)

ISSN 1651-9256

Printed by E-Print AB, Stockholm, Sweden

ISSN 1651-9256

(4)

Abstract

Practicing Motor Imagery (MI) with a Brain-Computer Interface (BCI) has shown promise in promoting motor recovery in stroke patients. A BCI records a person’s brain activity and provides feedback to the person in real time, which allows the person to practice his or her brain activity. By imagining a movement (performing MI) such as gripping with their hand, cortical areas in the brain are activated that largely overlaps with those activated during the actual hand movement. A BCI can provide positive feedback when the hand-related cortical areas are activated during MI, which helps a person to learn how to perform MI. Despite evidence that stroke patients may recover some motor function from practicing MI with BCI feedback thanks to the feedback provided from a BCI, the effectiveness and reliability of BCI-based rehabili-tation are still poor.

A BCI can detect MI by analyzing patterns of features from the brain activ-ity. The most common features are extracted from the oscillatory activity in the brain. In BCI research, MI is often treated as a static pattern of features, which is detected by using machine learning algorithms to assign activity into a binary state. However, this model of MI may be inaccurate. Analyzing brain activity as dynamically varying over time and with a continuous measure of strength could better represent the cortical activity related to MI.

In this Licentiate thesis, I explore a method for analyzing the temporal dy-namic of MI-activity with a continuous measure of strength. Brain activity was recorded with electroencephalography (EEG) and subject-specific fea-ture patterns were extracted from a group of healthy subjects while they per-formed MI of two opposing hand movements: opening and closing the hand. Although MI of the two same-hand movements could not be discriminated, the continuous output from a machine learning algorithm was shown to corre-late well with MI-recorre-lated feature patterns. The temporal analysis also revealed that MI is dynamically encoded early, but later stabilizes into a more static pattern of brain activity. Last, to accommodate for higher temporal resolution of MI, I designed and evaluated a BCI framework by its feedback delay and uncertainty as a function of the stress on the system and found a non-linear correlation. These results could be essential for developing a BCI with time-critical feedback.

To summarize, in this Licentiate thesis I propose a promising method for analyzing and extracting a temporal representation of MI, enabling relevant and continuous neurofeedback which may contribute to clinical advances in BCI-based stroke rehabilitation.

Abstract

Practicing Motor Imagery (MI) with a Brain-Computer Interface (BCI) has shown promise in promoting motor recovery in stroke patients. A BCI records a person’s brain activity and provides feedback to the person in real time, which allows the person to practice his or her brain activity. By imagining a movement (performing MI) such as gripping with their hand, cortical areas in the brain are activated that largely overlaps with those activated during the actual hand movement. A BCI can provide positive feedback when the hand-related cortical areas are activated during MI, which helps a person to learn how to perform MI. Despite evidence that stroke patients may recover some motor function from practicing MI with BCI feedback thanks to the feedback provided from a BCI, the effectiveness and reliability of BCI-based rehabili-tation are still poor.

A BCI can detect MI by analyzing patterns of features from the brain activ-ity. The most common features are extracted from the oscillatory activity in the brain. In BCI research, MI is often treated as a static pattern of features, which is detected by using machine learning algorithms to assign activity into a binary state. However, this model of MI may be inaccurate. Analyzing brain activity as dynamically varying over time and with a continuous measure of strength could better represent the cortical activity related to MI.

In this Licentiate thesis, I explore a method for analyzing the temporal dy-namic of MI-activity with a continuous measure of strength. Brain activity was recorded with electroencephalography (EEG) and subject-specific fea-ture patterns were extracted from a group of healthy subjects while they per-formed MI of two opposing hand movements: opening and closing the hand. Although MI of the two same-hand movements could not be discriminated, the continuous output from a machine learning algorithm was shown to corre-late well with MI-recorre-lated feature patterns. The temporal analysis also revealed that MI is dynamically encoded early, but later stabilizes into a more static pattern of brain activity. Last, to accommodate for higher temporal resolution of MI, I designed and evaluated a BCI framework by its feedback delay and uncertainty as a function of the stress on the system and found a non-linear correlation. These results could be essential for developing a BCI with time-critical feedback.

To summarize, in this Licentiate thesis I propose a promising method for analyzing and extracting a temporal representation of MI, enabling relevant and continuous neurofeedback which may contribute to clinical advances in BCI-based stroke rehabilitation.

(5)

(6)

Sammanfattning

Att träna motorisk föreställning (Motor Imagery, MI) med ett hjärna-datorgränssnitt (Brain-Computer Interface, BCI) har visat sig att vara en lovande metod för motorisk återhämtning hos strokepatienter. En BCI läser av en persons hjärnaktivitet och ger återkoppling i realtid, vilket hjälper personen att träna upp sin hjärnaktivitet. Genom att föreställa sig en rörelse (utföra MI), t.ex. att föreställa sig att greppa sin hand, så aktiveras områden i hjärnan som relaterar till handens faktiska rörelse. En BCI kan ge positiv återkoppling när handrelaterad hjärnaktivitet upptäcks vilket hjälper en person att lära sig att utföra MI. Trots att strokepatienter har visat sig kunna återfå viss motorisk förmåga efter MI-träning tack vare återkopplingen från en BCI så är tekniken ännu ineffektiv och har låg tillförlitlighet.

En BCI kan upptäcka MI genom att analysera mönster av attribut från hjär-naktiviteten. De vanligaste attributen extraheras från den oscillatoriska ak-tiviteten i hjärnan. I BCI-forskning så behandlas MI ofta som om den vore ett statiskt mönster av attribut, vilket kan upptäckas med maskininlärningsalgo-ritmer som tolkar hjärnaktiviteten till binära tillstånd. Den här modellen av MI kan dock vara felaktig. Att analysera hjärnaktivitet som dynamiskt varierande över tid och med en kontinuerlig styrka skulle bättre kunna representera MI i hjärnan.

I den här Licentiatuppsatsen utforskar jag en metod för att analysera tem-porell dynamik i MI och extrahera MI på en kontinuerlig skala. Hjärnaktivitet spelades in med elektroencefalografi (EEG) och individuella mönster av at-tribut extraherades från en grupp friska testpersoner medan de utförde MI av två motsatta handrörelser: öppna och stänga handen. Trots att MI av dessa två rörelser ej kunde särskiljas så visade det sig att det kontinuerliga mätvärdet från en maskininlärningsalgoritm korrelerade väl med MI-relaterade attribut-mönster. Den temporella analysen visade också att MI kodas dynamiskt tidigt i rörelsen, men övergår sedan till ett mer statiskt mönster av hjärnaktivitet. Slut-ligen, för att ta hänsyn till den högre temporella upplösningen av MI så desig-nade och utvärderade jag ett BCI-ramverk med avseende på tidsfördröjningar och osäkerhet som funktion av belastning på ramverket, och fann ett icke-linjärt samband. Dessa resultat tros ha stor betydelse för utveckling av en BCI med tidskritisk återkoppling.

Sammanfattningsvis, i den här Licentiatuppsatsen presenterar jag en lo-vande metod för att kunna analysera och extrahera en temporell representation av MI. Detta möjliggör relevant och kontinuerlig hjärnåterkoppling vilket kan bidra till kliniska framsteg i BCI-baserad stroke-rehabilitering.

Sammanfattning

Att träna motorisk föreställning (Motor Imagery, MI) med ett hjärna-datorgränssnitt (Brain-Computer Interface, BCI) har visat sig att vara en lovande metod för motorisk återhämtning hos strokepatienter. En BCI läser av en persons hjärnaktivitet och ger återkoppling i realtid, vilket hjälper personen att träna upp sin hjärnaktivitet. Genom att föreställa sig en rörelse (utföra MI), t.ex. att föreställa sig att greppa sin hand, så aktiveras områden i hjärnan som relaterar till handens faktiska rörelse. En BCI kan ge positiv återkoppling när handrelaterad hjärnaktivitet upptäcks vilket hjälper en person att lära sig att utföra MI. Trots att strokepatienter har visat sig kunna återfå viss motorisk förmåga efter MI-träning tack vare återkopplingen från en BCI så är tekniken ännu ineffektiv och har låg tillförlitlighet.

En BCI kan upptäcka MI genom att analysera mönster av attribut från hjär-naktiviteten. De vanligaste attributen extraheras från den oscillatoriska ak-tiviteten i hjärnan. I BCI-forskning så behandlas MI ofta som om den vore ett statiskt mönster av attribut, vilket kan upptäckas med maskininlärningsalgo-ritmer som tolkar hjärnaktiviteten till binära tillstånd. Den här modellen av MI kan dock vara felaktig. Att analysera hjärnaktivitet som dynamiskt varierande över tid och med en kontinuerlig styrka skulle bättre kunna representera MI i hjärnan.

I den här Licentiatuppsatsen utforskar jag en metod för att analysera tem-porell dynamik i MI och extrahera MI på en kontinuerlig skala. Hjärnaktivitet spelades in med elektroencefalografi (EEG) och individuella mönster av at-tribut extraherades från en grupp friska testpersoner medan de utförde MI av två motsatta handrörelser: öppna och stänga handen. Trots att MI av dessa två rörelser ej kunde särskiljas så visade det sig att det kontinuerliga mätvärdet från en maskininlärningsalgoritm korrelerade väl med MI-relaterade attribut-mönster. Den temporella analysen visade också att MI kodas dynamiskt tidigt i rörelsen, men övergår sedan till ett mer statiskt mönster av hjärnaktivitet. Slut-ligen, för att ta hänsyn till den högre temporella upplösningen av MI så desig-nade och utvärderade jag ett BCI-ramverk med avseende på tidsfördröjningar och osäkerhet som funktion av belastning på ramverket, och fann ett icke-linjärt samband. Dessa resultat tros ha stor betydelse för utveckling av en BCI med tidskritisk återkoppling.

Sammanfattningsvis, i den här Licentiatuppsatsen presenterar jag en lo-vande metod för att kunna analysera och extrahera en temporell representation av MI. Detta möjliggör relevant och kontinuerlig hjärnåterkoppling vilket kan bidra till kliniska framsteg i BCI-baserad stroke-rehabilitering.

(7)

(8)

I thank my family, friends and colleagues for encouraging me

during my studies and my supervisors for their indispensable

support.

I thank my family, friends and colleagues for encouraging me

during my studies and my supervisors for their indispensable

support.

(9)

(10)

List of Papers

This thesis is based on the following papers, which are referred to in the text by their Roman numerals.

I Tidare, J., Leon, M., Xiong, N., Astrand, E. Discriminating EEG spec-tral power related to mental imagery of closing and opening of hand. Published in 2019 9th International IEEE/EMBS Conference on Neural Engineering (NER).

II Tidare, J., Leon, M., Astrand,E. Time-resolved estimation of strength of Motor Imagery representation by multivariate EEG decoding, Journal of Neural Engineering (JNE, accepted).

III Tidare, J., Astrand, E., Ekström, M E. Evaluation of Closed-Loop Feed-back System Delay. A time-critical perspective for neurofeedFeed-back train-ing. Published in BIODEVICES 2018 - 11th International Conference on Biomedical Electronics and Devices, Proceedings; Part of 11th In-ternational Joint Conference on Biomedical Engineering Systems and Technologies, BIOSTEC 2018, SciTePress, 2018, p. 187-193.

List of Papers

This thesis is based on the following papers, which are referred to in the text by their Roman numerals.

I Tidare, J., Leon, M., Xiong, N., Astrand, E. Discriminating EEG spec-tral power related to mental imagery of closing and opening of hand. Published in 2019 9th International IEEE/EMBS Conference on Neural Engineering (NER).

II Tidare, J., Leon, M., Astrand,E. Time-resolved estimation of strength of Motor Imagery representation by multivariate EEG decoding, Journal of Neural Engineering (JNE, accepted).

III Tidare, J., Astrand, E., Ekström, M E. Evaluation of Closed-Loop Feed-back System Delay. A time-critical perspective for neurofeedFeed-back train-ing. Published in BIODEVICES 2018 - 11th International Conference on Biomedical Electronics and Devices, Proceedings; Part of 11th In-ternational Joint Conference on Biomedical Engineering Systems and Technologies, BIOSTEC 2018, SciTePress, 2018, p. 187-193.

(11)

(12)

Papers not included

These papers were not included in this Licentiate Thesis.

I Leon, M., Ballesteros, J., Tidare, J., Xiong, N., Astrand, E. Feature Se-lection of EEG Oscillatory Activity Related to Motor Imagery Using a Hierarchical Genetic Algorithm. Published in 2019 IEEE Congress on Evolutionary Computation.

II Leon, M., Parkkila, C., Tidare, J., Xiong, N., Astrand, E. Impact of NSGA-II Objectives on EEG Feature Selection Related to Motor Im-agery. Published in GECCO ’20: Proceedings of the 2020 Genetic and Evolutionary Computation Conference.

Papers not included

These papers were not included in this Licentiate Thesis.

I Leon, M., Ballesteros, J., Tidare, J., Xiong, N., Astrand, E. Feature Se-lection of EEG Oscillatory Activity Related to Motor Imagery Using a Hierarchical Genetic Algorithm. Published in 2019 IEEE Congress on Evolutionary Computation.

II Leon, M., Parkkila, C., Tidare, J., Xiong, N., Astrand, E. Impact of NSGA-II Objectives on EEG Feature Selection Related to Motor Im-agery. Published in GECCO ’20: Proceedings of the 2020 Genetic and Evolutionary Computation Conference.

(13)

(14)

Part I:

Thesis

Part I:

Thesis

(17)

(18)

1. Introduction

Stroke-patients have been shown to recover motor function using motor im-agery (MI): the mental imagination of physical movements [1]. This is espe-cially useful for severely motor-impaired patients who are unable to perform conventional physiotherapeutic training. During MI, motor related areas in the brain are activated [2], which is thought to promote plasticity and cortical reorganization in the brain. The theory for MI-based rehabilitation is simi-lar to learning: with guided practice, the cortical reorganization can lead to functional improvement [3, 4], and motor learning principles should drive the design of MI rehabilitation [5]. MI-based rehabilitation can be performed to-gether with a Brain-Computer Interface (BCI), which allows a person to re-ceive feedback based on their brain activity [6]. The BCI can analyze brain activity in real time and provide continual feedback when MI is detected. BCI feedback based on motor activity in the brain has been shown to promote re-covery for stroke patients [7].

After 20 years of intense BCI research, BCI rehabilitation remains a re-search laboratory experiment. Possible causes are that BCIs are technically challenging to use, and that the performance varies between subjects. It is generally acknowledged that successful BCI rehabilitation requires accurate feedback, but for stroke rehabilitation, it may be equally important to provide feedback on the neuroprocesses relevant to recovery [8]. BCI performance has improved substantially over the last two decades, showing that MI activity can be detected with high accuracy [9, 10, 11]. However, little is known about how the brain encodes behavior in real time, which leads to varying experimental BCI setups and inconsistent clinical outcomes [12, 13]. More clinical evidence is required to prove the effectiveness of MI-BCI rehabilitation [14].

Neural activity can be recorded as an electric field with surface electrodes placed on the scalp, known as electroencephalogram (EEG). By recording EEG from a subject while he/she performs an MI task, correlations can be found between the MI task and the EEG data. However, the EEG data time series (the raw data) is very noisy, and multiple repetitions, or trials, of the task is generally recorded to produce an average trial. To further reduce noise, pre-processing and feature extraction is performed. Pre-processing aims to reduce noise in the EEG data, and various features can be extracted from EEG channel that better explain the underlying brain activity [15].

One of the greatest tools for brain imaging and interpretation is the multi-variate analysis of features [16]. Multimulti-variate analysis can be performed with

1. Introduction

Stroke-patients have been shown to recover motor function using motor im-agery (MI): the mental imagination of physical movements [1]. This is espe-cially useful for severely motor-impaired patients who are unable to perform conventional physiotherapeutic training. During MI, motor related areas in the brain are activated [2], which is thought to promote plasticity and cortical reorganization in the brain. The theory for MI-based rehabilitation is simi-lar to learning: with guided practice, the cortical reorganization can lead to functional improvement [3, 4], and motor learning principles should drive the design of MI rehabilitation [5]. MI-based rehabilitation can be performed to-gether with a Brain-Computer Interface (BCI), which allows a person to re-ceive feedback based on their brain activity [6]. The BCI can analyze brain activity in real time and provide continual feedback when MI is detected. BCI feedback based on motor activity in the brain has been shown to promote re-covery for stroke patients [7].

After 20 years of intense BCI research, BCI rehabilitation remains a re-search laboratory experiment. Possible causes are that BCIs are technically challenging to use, and that the performance varies between subjects. It is generally acknowledged that successful BCI rehabilitation requires accurate feedback, but for stroke rehabilitation, it may be equally important to provide feedback on the neuroprocesses relevant to recovery [8]. BCI performance has improved substantially over the last two decades, showing that MI activity can be detected with high accuracy [9, 10, 11]. However, little is known about how the brain encodes behavior in real time, which leads to varying experimental BCI setups and inconsistent clinical outcomes [12, 13]. More clinical evidence is required to prove the effectiveness of MI-BCI rehabilitation [14].

Neural activity can be recorded as an electric field with surface electrodes placed on the scalp, known as electroencephalogram (EEG). By recording EEG from a subject while he/she performs an MI task, correlations can be found between the MI task and the EEG data. However, the EEG data time series (the raw data) is very noisy, and multiple repetitions, or trials, of the task is generally recorded to produce an average trial. To further reduce noise, pre-processing and feature extraction is performed. Pre-processing aims to reduce noise in the EEG data, and various features can be extracted from EEG channel that better explain the underlying brain activity [15].

One of the greatest tools for brain imaging and interpretation is the multi-variate analysis of features [16]. Multimulti-variate analysis can be performed with

(19)

many different decoding algorithms, and they can consider a large number of EEG features, and interactions between them, revealing how each feature contributes to an explanation of the mental activity [17]. Multivariate decod-ing can be performed across time, i.e. Temporal decoddecod-ing [18]. Temporal de-coding has allowed several brain imaging methods to produce fine-grained information of neural activity, and the temporal decoding procedure was per-formed to analyze MI-related patterns of features in a healthy population.

The performance of a decoding algorithm is generally measured by its av-erage predictive power across all trials of EEG data, or its classification accu-racy. However, the continual feedback provided from a BCI relates to a short time window of recently recorded EEG activity [19], thus motivating single-trial analysis of EEG data. In fact, neural activity in single single-trials may not be accurately represented by the trial-wise average [20], indicating that single-trial analysis could substantially benefit accuracy of neurofeedback. Thus, the continuous decoding output, corresponding to a confidence level of the de-coder for a single trial, was used to estimate the strength of feature patterns over the duration of each trial. This method was extended temporally, such that the decoder value was calculated on consecutive time windows of EEG data, extracted from different time points during the MI task. The result is a time-resolved, continuous estimate of subject-specific MI strength. This value was shown to correlate with a full pattern of MI-related activity, as opposed to being driven by a few individual features in the EEG data. This finding is critical, since motor recovery has been correlated with multiple areas of brain activity [21], and neurofeedback should reinforce all relevant areas for recov-ery.

In this Licentiate thesis, I aim to analyze the temporal variations of EEG patterns in single trials, and investigate how the strength of an MI-related EEG pattern can be quantified and visualized. The data in this thesis was recorded from a healthy population performing MI of two different hand movements: open and close right hand. The discriminability of the two tasks were evalu-ated using two decoding algorithms, and an estimate of MI strength was cal-culated based on the difference from baseline activity. The estimation of MI strength was calculated from subject-specific feature patterns, allowing for comparison of MI strength between subjects. This time-resolved measure can be used to improve clinical effectiveness of BCI-based stroke rehabilitation, as the estimate of MI strength represents neural activity more accurately than a discrete prediction class often used in neurofeedback interventions. Last, I de-velop and evaluate a BCI framework to assess the feasibility of implementing a more time-critical type of neurofeedback.

4

many different decoding algorithms, and they can consider a large number of EEG features, and interactions between them, revealing how each feature contributes to an explanation of the mental activity [17]. Multivariate decod-ing can be performed across time, i.e. Temporal decoddecod-ing [18]. Temporal de-coding has allowed several brain imaging methods to produce fine-grained information of neural activity, and the temporal decoding procedure was per-formed to analyze MI-related patterns of features in a healthy population.

The performance of a decoding algorithm is generally measured by its av-erage predictive power across all trials of EEG data, or its classification accu-racy. However, the continual feedback provided from a BCI relates to a short time window of recently recorded EEG activity [19], thus motivating single-trial analysis of EEG data. In fact, neural activity in single single-trials may not be accurately represented by the trial-wise average [20], indicating that single-trial analysis could substantially benefit accuracy of neurofeedback. Thus, the continuous decoding output, corresponding to a confidence level of the de-coder for a single trial, was used to estimate the strength of feature patterns over the duration of each trial. This method was extended temporally, such that the decoder value was calculated on consecutive time windows of EEG data, extracted from different time points during the MI task. The result is a time-resolved, continuous estimate of subject-specific MI strength. This value was shown to correlate with a full pattern of MI-related activity, as opposed to being driven by a few individual features in the EEG data. This finding is critical, since motor recovery has been correlated with multiple areas of brain activity [21], and neurofeedback should reinforce all relevant areas for recov-ery.

In this Licentiate thesis, I aim to analyze the temporal variations of EEG patterns in single trials, and investigate how the strength of an MI-related EEG pattern can be quantified and visualized. The data in this thesis was recorded from a healthy population performing MI of two different hand movements: open and close right hand. The discriminability of the two tasks were evalu-ated using two decoding algorithms, and an estimate of MI strength was cal-culated based on the difference from baseline activity. The estimation of MI strength was calculated from subject-specific feature patterns, allowing for comparison of MI strength between subjects. This time-resolved measure can be used to improve clinical effectiveness of BCI-based stroke rehabilitation, as the estimate of MI strength represents neural activity more accurately than a discrete prediction class often used in neurofeedback interventions. Last, I de-velop and evaluate a BCI framework to assess the feasibility of implementing a more time-critical type of neurofeedback.

(20)

2. Background

A Brain-computer interface (BCI) is a system that can record a signal directly from the brain and provide feedback based on this signal. The general exper-imental setup for BCI recordings can be seen in Figure 2.1. A typical setup has a protocol shown on a screen, instructing the subject which mental task to perform. While the subject performs the task, EEG signals are recorded and processed, and feedback is displayed on the screen in real-time. This method allows a person to improve their mental performance.

Motor Imagery (MI) has been studied thoroughly in sports as a method of improving performance [22], but when MI was shown to generate brain ac-tivity similar to actual motion, or motor execution (ME) [23] it appeared as a proposed input method for Brain-computer interfaces (BCI) in several seminal papers [24, 25, 26]. The MI-based BCI was initially suggested as an assistive tool for persons with severe muscle disabilities, but MI practice was simulta-neously hypothesized to be a feasible stroke rehabilitation method [27, 28]. This has lead to several successful MI-based BCI-rehabilitation experiments where neurofeedback is given in response to MI activity in the brain, suggest-ing that MI is at least comparable to conventional physiotherapy for stroke rehabilitation [6, 1]. Patients with severe motor disabilities may not be able to perform physiotherapy, and MI could be an alternative for this patient group. For patients who are able to participate in physiotherapy, MI training could still be a useful complement [29].

BCI research is an interdisciplinary field, involving neurobiology, psychol-ogy, engineering, mathematics and computer science [24]. This chapter will first introduce the evidence and methods for recovery with BCI. Then, various types of neurofeedback will be discussed followed by an introduction to

MI-Figure 2.1:A simplified example of a BCI. A person sits in front of a screen, with EEG electrodes fixed on the scalp. Instructions on the screen explain which mental task to perform. The EEG signals are recorded, analyzed and feedback is sent back to the screen. When the person receives the feedback, the loop is closed.

2. Background

A Brain-computer interface (BCI) is a system that can record a signal directly from the brain and provide feedback based on this signal. The general exper-imental setup for BCI recordings can be seen in Figure 2.1. A typical setup has a protocol shown on a screen, instructing the subject which mental task to perform. While the subject performs the task, EEG signals are recorded and processed, and feedback is displayed on the screen in real-time. This method allows a person to improve their mental performance.

Motor Imagery (MI) has been studied thoroughly in sports as a method of improving performance [22], but when MI was shown to generate brain ac-tivity similar to actual motion, or motor execution (ME) [23] it appeared as a proposed input method for Brain-computer interfaces (BCI) in several seminal papers [24, 25, 26]. The MI-based BCI was initially suggested as an assistive tool for persons with severe muscle disabilities, but MI practice was simulta-neously hypothesized to be a feasible stroke rehabilitation method [27, 28]. This has lead to several successful MI-based BCI-rehabilitation experiments where neurofeedback is given in response to MI activity in the brain, suggest-ing that MI is at least comparable to conventional physiotherapy for stroke rehabilitation [6, 1]. Patients with severe motor disabilities may not be able to perform physiotherapy, and MI could be an alternative for this patient group. For patients who are able to participate in physiotherapy, MI training could still be a useful complement [29].

BCI research is an interdisciplinary field, involving neurobiology, psychol-ogy, engineering, mathematics and computer science [24]. This chapter will first introduce the evidence and methods for recovery with BCI. Then, various types of neurofeedback will be discussed followed by an introduction to

MI-Figure 2.1:A simplified example of a BCI. A person sits in front of a screen, with EEG electrodes fixed on the scalp. Instructions on the screen explain which mental task to perform. The EEG signals are recorded, analyzed and feedback is sent back to the screen. When the person receives the feedback, the loop is closed.

(21)

related features in the EEG data. Last, an introduction is given on various MI techniques and temporal dynamics of MI activity.

BCI-based Rehabilitation

While this thesis focuses on the technical side of BCIs, it is important to emphasize the primary reason for developing BCI technology and improving upon the current methods.

It has been shown that the brain is able to reorganize itself after stroke to compensate for motor deficits [21], and conventional (physical) rehabilita-tion training is centered around repetitive task-specific motor activities with the disabled limb [30]. Physiotherapeutic evidence suggests that post-stroke rehabilitation of the hand should be focused towards partial or whole skills required in activities of daily living (ADL), such as buttoning, pouring, fold-ing and liftfold-ing [31]. Since MI activates the correspondfold-ing body-part-specific motor representations in the brain [2], the brain could possibly recover from stroke using any method for stimulating the lesioned areas, MI or physical therapy. To further emphasize the possibilities of imagery, motor skills can be learnt entirely using MI [32] and two reviews/meta-analyses suggests that BCI interventions may be a promising rehabilitation approach for stroke patients [33, 1].

A majority of stroke patients can operate EEG-based MI-BCI [34] and the severity of their disability does not correlate with ability to operate a BCI [35]. More importantly, stroke patients with severe hand weakness can achieve a meaningful clinical improvement from no to some activity in paretic muscles [36]. Therefore, MI-based BCI rehabilitation can be used by stroke patients at any stage of motor disability.

BCI-based MI-practice is significantly more effective than MI practice without BCI support [37], although the effect of having a BCI could influence the results. Adding BCI control to exoskeleton-assisted physical therapy can improve post-stroke rehabilitation outcomes, and improved motor function correlated with increased classification accuracy [38]. One study found that Robot-assisted BCI therapy was comparable to intense physiotherapy [6].

Despite many impressive results using BCI technology for rehabilitation, BCI technology is scarcely used outside laboratory experiments [13]. One ex-planation could be that BCI performance can be unreliable. An estimated 20% of subjects cannot attain satisfactory BCI control. This condition is referred to as BCI illiteracy and could be explained by many different factors, for ex-ample the structure of the brain [39] or a less developed brain network for motor skills [14]. Another explanation could be the limited clinical evidence. Even if BCI-training show clinically relevant outcomes, these outcomes must compete with conventional therapies. If BCI-based training and conventional therapy produce similar outcomes for stroke patients, it is difficult to moti-vate hospitals to invest in BCI technology. More convincing evidence is re-quired for BCIs to reach outside the research labs [14, 1]. In order to increase 6

related features in the EEG data. Last, an introduction is given on various MI techniques and temporal dynamics of MI activity.

BCI-based Rehabilitation

While this thesis focuses on the technical side of BCIs, it is important to emphasize the primary reason for developing BCI technology and improving upon the current methods.

It has been shown that the brain is able to reorganize itself after stroke to compensate for motor deficits [21], and conventional (physical) rehabilita-tion training is centered around repetitive task-specific motor activities with the disabled limb [30]. Physiotherapeutic evidence suggests that post-stroke rehabilitation of the hand should be focused towards partial or whole skills required in activities of daily living (ADL), such as buttoning, pouring, fold-ing and liftfold-ing [31]. Since MI activates the correspondfold-ing body-part-specific motor representations in the brain [2], the brain could possibly recover from stroke using any method for stimulating the lesioned areas, MI or physical therapy. To further emphasize the possibilities of imagery, motor skills can be learnt entirely using MI [32] and two reviews/meta-analyses suggests that BCI interventions may be a promising rehabilitation approach for stroke patients [33, 1].

A majority of stroke patients can operate EEG-based MI-BCI [34] and the severity of their disability does not correlate with ability to operate a BCI [35]. More importantly, stroke patients with severe hand weakness can achieve a meaningful clinical improvement from no to some activity in paretic muscles [36]. Therefore, MI-based BCI rehabilitation can be used by stroke patients at any stage of motor disability.

BCI-based MI-practice is significantly more effective than MI practice without BCI support [37], although the effect of having a BCI could influence the results. Adding BCI control to exoskeleton-assisted physical therapy can improve post-stroke rehabilitation outcomes, and improved motor function correlated with increased classification accuracy [38]. One study found that Robot-assisted BCI therapy was comparable to intense physiotherapy [6].

Despite many impressive results using BCI technology for rehabilitation, BCI technology is scarcely used outside laboratory experiments [13]. One ex-planation could be that BCI performance can be unreliable. An estimated 20% of subjects cannot attain satisfactory BCI control. This condition is referred to as BCI illiteracy and could be explained by many different factors, for ex-ample the structure of the brain [39] or a less developed brain network for motor skills [14]. Another explanation could be the limited clinical evidence. Even if BCI-training show clinically relevant outcomes, these outcomes must compete with conventional therapies. If BCI-based training and conventional therapy produce similar outcomes for stroke patients, it is difficult to moti-vate hospitals to invest in BCI technology. More convincing evidence is re-quired for BCIs to reach outside the research labs [14, 1]. In order to increase 6

(22)

effectiveness in BCI-based rehabilitation, a clearer understanding of the in-teractions between intervention design and neurophysiological recovery is re-quired, to better understand the factors that may affect rehabilitation outcomes [40]. Some of those factors are: MI technique, effect of feedback delay, type of feedback and classification performance. Optimizing these parameters will benefit future studies and increase the clinical outcomes in future studies.

EEG suffers from the volume conduction effect, which blurs and distorts neuroelectric signals as they travel from the cortical surface to the scalp [41]. This makes same-hand classification very difficult since the motor areas re-sponsible for different one hand-tasks are spatially close to one another [11]. Instead, many BCI research experiments rely on right and left hand MI as control states for their high discriminability [9]. In rehabilitation however, functional neurofeedback to the disabled hand is the goal.

Neurofeedback

Visual feedback can be given continually during task performance, like an ex-tending bar [42, 43] or a moving cursor [44]; or intermittenly at the end of each task repetition [35]. Feedback can be provided through other modalities than visual, such as electrical stimulation of muscles or robot-assisted movement [35, 45]. Providing neurofeedback in response to motor activity in the brain is crucial for successful neurofeedback training [7], but there are many options for providing feedback to a subject. The following section will first explain the neurological background for neurofeedback in relation to feedback delay, then discuss feedback types in relation to human learning principles.

BCI-training is based on operant conditioning: a behavior is coupled with a reward-feedback, and this is thought to promote neuroplasticity [46]. On a neuronal level, it is related to Hebbian learning [3] which can be summed up as ”neurons that fire together, wire together”. It forms the basis for most models of learning, memory and neural circuits development [47]. Ideally, the neurons generating MI-related EEG patterns should be coupled with si-multaneous feedback from the BCI, but in practice, the BCI feedback will be delayed. A pre-defined time window of EEG activity must be recorded, pro-cessed and transformed into feedback for the BCI user. The feedback is based partially on EEG activity from the first (oldest) sample in the time window. Thus, the maximum feedback delay is the sum of the time window size, the computation time and the presentation time (i.e. the time it takes for the BCI to present the feedback via chosen modality).

During online BCI experiments, feedback is updated continually. Feedback Update Interval (FUI) is the update frequency of feedback. FUI is often shorter than the size of the time window used for feedback [48, 43, 42], meaning that each time window overlaps with adjacent time windows. There is a weak in-dication that short FUI could benefit motor recovery in stroke patients [49]. However, FUI should be discussed together with feedback delay, and the max-imum feedback delay should be reported. The effect of feedback delay and

effectiveness in BCI-based rehabilitation, a clearer understanding of the in-teractions between intervention design and neurophysiological recovery is re-quired, to better understand the factors that may affect rehabilitation outcomes [40]. Some of those factors are: MI technique, effect of feedback delay, type of feedback and classification performance. Optimizing these parameters will benefit future studies and increase the clinical outcomes in future studies.

EEG suffers from the volume conduction effect, which blurs and distorts neuroelectric signals as they travel from the cortical surface to the scalp [41]. This makes same-hand classification very difficult since the motor areas re-sponsible for different one hand-tasks are spatially close to one another [11]. Instead, many BCI research experiments rely on right and left hand MI as control states for their high discriminability [9]. In rehabilitation however, functional neurofeedback to the disabled hand is the goal.

Neurofeedback

Visual feedback can be given continually during task performance, like an ex-tending bar [42, 43] or a moving cursor [44]; or intermittenly at the end of each task repetition [35]. Feedback can be provided through other modalities than visual, such as electrical stimulation of muscles or robot-assisted movement [35, 45]. Providing neurofeedback in response to motor activity in the brain is crucial for successful neurofeedback training [7], but there are many options for providing feedback to a subject. The following section will first explain the neurological background for neurofeedback in relation to feedback delay, then discuss feedback types in relation to human learning principles.

BCI-training is based on operant conditioning: a behavior is coupled with a reward-feedback, and this is thought to promote neuroplasticity [46]. On a neuronal level, it is related to Hebbian learning [3] which can be summed up as ”neurons that fire together, wire together”. It forms the basis for most models of learning, memory and neural circuits development [47]. Ideally, the neurons generating MI-related EEG patterns should be coupled with si-multaneous feedback from the BCI, but in practice, the BCI feedback will be delayed. A pre-defined time window of EEG activity must be recorded, pro-cessed and transformed into feedback for the BCI user. The feedback is based partially on EEG activity from the first (oldest) sample in the time window. Thus, the maximum feedback delay is the sum of the time window size, the computation time and the presentation time (i.e. the time it takes for the BCI to present the feedback via chosen modality).

During online BCI experiments, feedback is updated continually. Feedback Update Interval (FUI) is the update frequency of feedback. FUI is often shorter than the size of the time window used for feedback [48, 43, 42], meaning that each time window overlaps with adjacent time windows. There is a weak in-dication that short FUI could benefit motor recovery in stroke patients [49]. However, FUI should be discussed together with feedback delay, and the max-imum feedback delay should be reported. The effect of feedback delay and

(23)

FUI has been thoroughly studied in motor learning tasks [50], but is largely unexplored in MI learning tasks with BCIs.

Many of the current BCI experiments are inappropriate for practicing MI. Feedback is often only corrective, explaining only if the task was performed correctly or not. Contrarily, human learning principles suggest explanatory feedback by indicating what was right or wrong in the performance. Further-more, BCI tasks should be motivating and engaging, but the typical BCI is set in a non-stimulating (boring) environment to minimize non-task related brain activity and designed to record multiple repetitions, or trials, of an MI task [13, 51].

Many BCI-based rehabilitation studies uses feedback with binary control, such that a predefined feedback is given when MI is detected [6]. However, healthy subjects rate continuous feedback as more motivating and helpful dur-ing MI [43]. It would be appropriate to build a modulated, continuous feed-back that follows the variability of the brain [46]. Such feedfeed-back is more ap-propriate for practicing whole skills required in ADL [31]. The brain does not follow the binary ”all or nothing response” and neither should the feedback. MI-related feature patterns

To analyze EEG activity, the power spectral density (PSD) can be extracted using any time-frequency transform on the EEG data time series. The PSD is a signal’s oscillatory power distribution over frequency and it is one of the most common features extracted in BCI research [19]. EEG activity can be observed as event-related changes in PSD features. An event-related increase in a PSD amplitude is referred to as an event-related synchronization (ERS), and the corresponding decrease is referred to as an event-related desynchro-nization (ERD). ERD and ERS observed in multiple EEG electrodes result in an ERD/ERS pattern.

The most well-studied EEG-PSD features relating to MI activity are the ERD/ERS patterns in the alpha band (8-12 Hz) and the beta band (14-25 Hz) [52, 53]. During MI, a bilateral ERD can be observed in both alpha and beta centrally over the head, often stronger on the contralateral side, and this in-dicates increased activity in motor-related areas in the brain, [25]. MI-related PSD activity has also been found in frequencies outside the range of alpha and beta, such as frequencies over 30 Hz (gamma) [54]), and frequencies lower than 4 Hz (delta) [55]. Furthermore, EEG activity is highly subject-specific [56, 57] and some non-motor areas have been reported to be involved with mo-tor recovery [21]. A wide range of PSD features may contain more MI-related information than PSD extracted from only pre-defined frequency bands such as alpha. Therefore, extracting PSD features from a wide frequency band and multiple electrodes is motivated since neurofeedback must relate to motor-related brain-activity to be effective [7],

8

FUI has been thoroughly studied in motor learning tasks [50], but is largely unexplored in MI learning tasks with BCIs.

Many of the current BCI experiments are inappropriate for practicing MI. Feedback is often only corrective, explaining only if the task was performed correctly or not. Contrarily, human learning principles suggest explanatory feedback by indicating what was right or wrong in the performance. Further-more, BCI tasks should be motivating and engaging, but the typical BCI is set in a non-stimulating (boring) environment to minimize non-task related brain activity and designed to record multiple repetitions, or trials, of an MI task [13, 51].

Many BCI-based rehabilitation studies uses feedback with binary control, such that a predefined feedback is given when MI is detected [6]. However, healthy subjects rate continuous feedback as more motivating and helpful dur-ing MI [43]. It would be appropriate to build a modulated, continuous feed-back that follows the variability of the brain [46]. Such feedfeed-back is more ap-propriate for practicing whole skills required in ADL [31]. The brain does not follow the binary ”all or nothing response” and neither should the feedback. MI-related feature patterns

To analyze EEG activity, the power spectral density (PSD) can be extracted using any time-frequency transform on the EEG data time series. The PSD is a signal’s oscillatory power distribution over frequency and it is one of the most common features extracted in BCI research [19]. EEG activity can be observed as event-related changes in PSD features. An event-related increase in a PSD amplitude is referred to as an event-related synchronization (ERS), and the corresponding decrease is referred to as an event-related desynchro-nization (ERD). ERD and ERS observed in multiple EEG electrodes result in an ERD/ERS pattern.

The most well-studied EEG-PSD features relating to MI activity are the ERD/ERS patterns in the alpha band (8-12 Hz) and the beta band (14-25 Hz) [52, 53]. During MI, a bilateral ERD can be observed in both alpha and beta centrally over the head, often stronger on the contralateral side, and this in-dicates increased activity in motor-related areas in the brain, [25]. MI-related PSD activity has also been found in frequencies outside the range of alpha and beta, such as frequencies over 30 Hz (gamma) [54]), and frequencies lower than 4 Hz (delta) [55]. Furthermore, EEG activity is highly subject-specific [56, 57] and some non-motor areas have been reported to be involved with mo-tor recovery [21]. A wide range of PSD features may contain more MI-related information than PSD extracted from only pre-defined frequency bands such as alpha. Therefore, extracting PSD features from a wide frequency band and multiple electrodes is motivated since neurofeedback must relate to motor-related brain-activity to be effective [7],

(24)

Multivariate Pattern analysis

Multivariate analysis, or MVA, is an analytical method which considers the in-formation carried by multiple features as a whole, determining how each fea-ture and interaction between feafea-tures contribute to a condition [17]. An exam-ple of inter-correlating features is cross-frequency coupling: when the activity of one frequency band is modulated by another [58]. In EEG research, MVA is used to analyze how individual features in ERD/ERS patterns contribute to explaining a neural response, such as the EEG activity elicited by MI. A decoding algorithm (also known as machine learning- or classifier algorithm) is often used to analyze such inter-correlating patterns of EEG activity [17]. The MVA of neural features dramatically increases sensitivity of human neu-roimaging by taking into account the full spatial pattern of brain activity [16, 59]. This is especially advantageous for EEG, which suffers from low spatial resolution due to volume conduction.

Motor Imagery

MI can be performed kinesthetically or visually. Kinesthetic MI is the mental experience of muscle tension and position, while visual MI is is the mental representation of a moving image, seen either from first or third person per-spective. Both visual and kinesthetic MI have been used in rehabilitation in-terventions and there is no obvious advantage for either method, only a weak indication that the kinesthetic MI is advantageous for stroke rehabilitation [60]. The neural correlates for both MI techniques are different: visual MI primarily activates occipital areas (responsible for vision) in the brain while kinesthetic MI primarily activates motor areas in the brain [61], although a meta-study found that visual MI can activate motor-related areas, and that vi-sual and kinesthetic MI share networks in the brain [62]. Kinesthetic MI is more effective than visual MI for improving physical task performance [63]. In a muscle-contraction practice study, kinesthetic imagery was the only im-agery technique producing significant strength increase in the muscle [64]. It was also shown that kinesthetic MI modulated corticomotor excitability, but visual MI did not [65]. These results provide converging evidence suggesting that kinesthetic imagery is advantageous for motor recovery.

Temporal dynamics of brain activity

Temporal variations during MI have been analyzed univariately [66, 67], re-vealing alpha ERD before starting the MI task (pre-motor activity) and the ERD is sustained throughout the task. When analyzing brain activity, trials are often averaged to obtain a representative estimate of neural activity and this improves signal-to-noise ratio. The average trial negates any intertrial varia-tions, effectively treating it as noise, but this type of noise could explain small variations in task performance [68]. If intertrial variations contain MI-related information, then the average MI trial may not be a good representative of typ-ical MI activity. Stokes and Spaak [20] highlight the importance of single-trial

Multivariate Pattern analysis

Multivariate analysis, or MVA, is an analytical method which considers the in-formation carried by multiple features as a whole, determining how each fea-ture and interaction between feafea-tures contribute to a condition [17]. An exam-ple of inter-correlating features is cross-frequency coupling: when the activity of one frequency band is modulated by another [58]. In EEG research, MVA is used to analyze how individual features in ERD/ERS patterns contribute to explaining a neural response, such as the EEG activity elicited by MI. A decoding algorithm (also known as machine learning- or classifier algorithm) is often used to analyze such inter-correlating patterns of EEG activity [17]. The MVA of neural features dramatically increases sensitivity of human neu-roimaging by taking into account the full spatial pattern of brain activity [16, 59]. This is especially advantageous for EEG, which suffers from low spatial resolution due to volume conduction.

Motor Imagery

MI can be performed kinesthetically or visually. Kinesthetic MI is the mental experience of muscle tension and position, while visual MI is is the mental representation of a moving image, seen either from first or third person per-spective. Both visual and kinesthetic MI have been used in rehabilitation in-terventions and there is no obvious advantage for either method, only a weak indication that the kinesthetic MI is advantageous for stroke rehabilitation [60]. The neural correlates for both MI techniques are different: visual MI primarily activates occipital areas (responsible for vision) in the brain while kinesthetic MI primarily activates motor areas in the brain [61], although a meta-study found that visual MI can activate motor-related areas, and that vi-sual and kinesthetic MI share networks in the brain [62]. Kinesthetic MI is more effective than visual MI for improving physical task performance [63]. In a muscle-contraction practice study, kinesthetic imagery was the only im-agery technique producing significant strength increase in the muscle [64]. It was also shown that kinesthetic MI modulated corticomotor excitability, but visual MI did not [65]. These results provide converging evidence suggesting that kinesthetic imagery is advantageous for motor recovery.

Temporal dynamics of brain activity

Temporal variations during MI have been analyzed univariately [66, 67], re-vealing alpha ERD before starting the MI task (pre-motor activity) and the ERD is sustained throughout the task. When analyzing brain activity, trials are often averaged to obtain a representative estimate of neural activity and this improves signal-to-noise ratio. The average trial negates any intertrial varia-tions, effectively treating it as noise, but this type of noise could explain small variations in task performance [68]. If intertrial variations contain MI-related information, then the average MI trial may not be a good representative of typ-ical MI activity. Stokes and Spaak [20] highlight the importance of single-trial

(25)

analysis of task-related brain activity and they provide theoretical evidence that transient activity that is not time-locked to an event is not accurately rep-resented by its average trial. For example, If MI was correlated with a single transient burst of alpha activity which could occur at any point during the MI task, the average trial would appear to have a sustained neural activity. This sustained neural activity is an artifact of trial-wise averaging and is not rep-resentative of a single trial of the MI task. Whether neural oscillations are in fact sustained rhythms or transient burst events (or a combination) cannot be determined when when many trials are averaged [69]. Thus, single-trial anal-ysis is required. Furthermore, online neurofeedback is based on a single time window of MI activity, and analyzing temporal variations in single trials can have huge benefits for BCI performance.

Temporal analysis can be extended with MVA, i.e. temporal decoding. A decoding algorithm trained and tested on data from many different time points can reveal how patterns of neural activity generalize over time [18]. This method is common in cognitive research on working memory and stimulus-response tasks, but not in MI research. Furthermore, the temporal decoding relies on estimations of classification accuracy, which is derived from all tri-als in the data set, which hinders single-trial analysis with this method.

10

analysis of task-related brain activity and they provide theoretical evidence that transient activity that is not time-locked to an event is not accurately rep-resented by its average trial. For example, If MI was correlated with a single transient burst of alpha activity which could occur at any point during the MI task, the average trial would appear to have a sustained neural activity. This sustained neural activity is an artifact of trial-wise averaging and is not rep-resentative of a single trial of the MI task. Whether neural oscillations are in fact sustained rhythms or transient burst events (or a combination) cannot be determined when when many trials are averaged [69]. Thus, single-trial anal-ysis is required. Furthermore, online neurofeedback is based on a single time window of MI activity, and analyzing temporal variations in single trials can have huge benefits for BCI performance.

Temporal analysis can be extended with MVA, i.e. temporal decoding. A decoding algorithm trained and tested on data from many different time points can reveal how patterns of neural activity generalize over time [18]. This method is common in cognitive research on working memory and stimulus-response tasks, but not in MI research. Furthermore, the temporal decoding relies on estimations of classification accuracy, which is derived from all tri-als in the data set, which hinders single-trial analysis with this method.

(26)

3. Research Overview

The problem formulation was written based on key points derived from the background. These key points are i) same-hand neurofeedback for rehabilita-tion, ii) single-trial analysis of MI using MVA and iii) time-critical design of BCIs.

3.1 Problem Formulation

EEG signals are known to be non-stationary, meaning that the EEG signals vary in time. Therefore, it is common practice to average EEG activity over a predefined time window to reveal EEG activity related to a specific men-tal task. Moreover, multiple repetitions (i.e. trials) of the same task must be performed and averaged to extract task-specific EEG signals from noise and other unrelated EEG-activity. Averaging across time and trials reveals general trends but hides the temporal dynamics (TDs) of the EEG activity. The TDs of EEG has been studied using univariate analysis of features by considering am-plitude changes in a single feature or cumulated group of features, but many mental tasks are better described as a pattern of EEG data rather than a sin-gle feature. Task-specific patterns can be extracted using multivariate pattern analysis of features. However, there are no commonly used methods to ana-lyze EEG patterns as a function of time and therefore, little is known about how EEG patterns vary in time. A new method is required to measure TDs of EEG patterns.

MI of a hand motion is a very common mental task for BCIs and has shown promise for motor-related rehabilitation after for example a stroke. However, the poor spatial resolution of EEG has led many BCI experiments to use right and left hand MI, thereby offering higher spatial discriminability in the EEG activity. This suits the goal of traditional BCIs: to achieve high classification accuracy of mental tasks in order to for example control an external device. However, rehabilitation depends on practicing motor skills relevant to recov-ery, preferrably whole movements related to ADL. Recording the complex functionality of a hand requires more than just discriminating its activity from the other hand. A hand task, mental or physical, is continuous from the start to the end point of the movement, thus it is conceivable that the relevant EEG features also follow some temporal variation throughout this duration. BCI rehabilitation could greatly benefit from a time-resolved estimation of MI

be-3. Research Overview

The problem formulation was written based on key points derived from the background. These key points are i) same-hand neurofeedback for rehabilita-tion, ii) single-trial analysis of MI using MVA and iii) time-critical design of BCIs.

3.1 Problem Formulation

EEG signals are known to be non-stationary, meaning that the EEG signals vary in time. Therefore, it is common practice to average EEG activity over a predefined time window to reveal EEG activity related to a specific men-tal task. Moreover, multiple repetitions (i.e. trials) of the same task must be performed and averaged to extract task-specific EEG signals from noise and other unrelated EEG-activity. Averaging across time and trials reveals general trends but hides the temporal dynamics (TDs) of the EEG activity. The TDs of EEG has been studied using univariate analysis of features by considering am-plitude changes in a single feature or cumulated group of features, but many mental tasks are better described as a pattern of EEG data rather than a sin-gle feature. Task-specific patterns can be extracted using multivariate pattern analysis of features. However, there are no commonly used methods to ana-lyze EEG patterns as a function of time and therefore, little is known about how EEG patterns vary in time. A new method is required to measure TDs of EEG patterns.

MI of a hand motion is a very common mental task for BCIs and has shown promise for motor-related rehabilitation after for example a stroke. However, the poor spatial resolution of EEG has led many BCI experiments to use right and left hand MI, thereby offering higher spatial discriminability in the EEG activity. This suits the goal of traditional BCIs: to achieve high classification accuracy of mental tasks in order to for example control an external device. However, rehabilitation depends on practicing motor skills relevant to recov-ery, preferrably whole movements related to ADL. Recording the complex functionality of a hand requires more than just discriminating its activity from the other hand. A hand task, mental or physical, is continuous from the start to the end point of the movement, thus it is conceivable that the relevant EEG features also follow some temporal variation throughout this duration. BCI rehabilitation could greatly benefit from a time-resolved estimation of MI

(27)

be-cause it would better mimic the continuous motion of physiotherapeutic prac-tice. Traditional BCIs would also benefit from a time-resolved estimation of MI, as it allows for additional control states.

Continual neurofeedback from EEG data has proven effective when sub-jects learn to control their brain activity during for example MI, but delays can be detrimental when learning a motor task. Many tools for EEG analysis were developed for offline analysis, but continual real-time feedback is emerg-ing as a method in BCI experiments. With better understandemerg-ing of the TDs in EEG patterns, increased demand on real-time requirements will follow.

3.2 Research Questions

This Licentiate thesis focuses on the TDs of MI in EEG data and how they can be analyzed. EEG data of MI has traditionally been averaged over large time windows for feature extraction and signal analysis, but this thesis aims to analyze the evolution of MI and MI EEG patterns as a function of time. Analyzing time-variant changes in a single imagined motion is closely related to classification of same-limb MI, as the first and last time point of a grip-ping motion can be interpreted as two discrete states: a neutral hand position and a fist. The temporal dynamics of MI can be conceptualized as a feature vector travelling from one point to another, or as a dynamic subset of fea-tures explaining MI. Both concepts could be valid explanations of TDs but they present different challenges for comparing temporal dynamics between subjects, due to inter-subject variability in MI activity. Furthermore, Finally, a BCI should be able to handle precise timings in order to benefit from the increased temporal resolution. Therefore, a BCI framework is developed and tested. Three research questions are formulated and explained.

1. How well can Motor Imagery of same-hand movements be dis-criminated using EEG data?

Single hand discrimination of EEG signals is challenging because EEG has poor spatial resolution, and the cortical motor area rep-resenting one hand is small, leading to low classification accuracy. However, hand function is very important for independent living, and lack of muscle control in the hand can severely limit a stroke pa-tient in day to day life. Several brain areas are known to be activated together during motor tasks, meaning that multivariate decoding of EEG features could reveal discriminable differences between differ-ent same-hand motions. Therefore, this question aims to find discrim-inable features within same-hand motion of opening and closing one hand.

2. How can multivariate decoding analysis be implemented to inves-tigate the temporal dynamics of EEG activity related to Motor Imagery?

12

cause it would better mimic the continuous motion of physiotherapeutic prac-tice. Traditional BCIs would also benefit from a time-resolved estimation of MI, as it allows for additional control states.

Continual neurofeedback from EEG data has proven effective when sub-jects learn to control their brain activity during for example MI, but delays can be detrimental when learning a motor task. Many tools for EEG analysis were developed for offline analysis, but continual real-time feedback is emerg-ing as a method in BCI experiments. With better understandemerg-ing of the TDs in EEG patterns, increased demand on real-time requirements will follow.

3.2 Research Questions

This Licentiate thesis focuses on the TDs of MI in EEG data and how they can be analyzed. EEG data of MI has traditionally been averaged over large time windows for feature extraction and signal analysis, but this thesis aims to analyze the evolution of MI and MI EEG patterns as a function of time. Analyzing time-variant changes in a single imagined motion is closely related to classification of same-limb MI, as the first and last time point of a grip-ping motion can be interpreted as two discrete states: a neutral hand position and a fist. The temporal dynamics of MI can be conceptualized as a feature vector travelling from one point to another, or as a dynamic subset of fea-tures explaining MI. Both concepts could be valid explanations of TDs but they present different challenges for comparing temporal dynamics between subjects, due to inter-subject variability in MI activity. Furthermore, Finally, a BCI should be able to handle precise timings in order to benefit from the increased temporal resolution. Therefore, a BCI framework is developed and tested. Three research questions are formulated and explained.

1. How well can Motor Imagery of same-hand movements be dis-criminated using EEG data?

Single hand discrimination of EEG signals is challenging because EEG has poor spatial resolution, and the cortical motor area rep-resenting one hand is small, leading to low classification accuracy. However, hand function is very important for independent living, and lack of muscle control in the hand can severely limit a stroke pa-tient in day to day life. Several brain areas are known to be activated together during motor tasks, meaning that multivariate decoding of EEG features could reveal discriminable differences between differ-ent same-hand motions. Therefore, this question aims to find discrim-inable features within same-hand motion of opening and closing one hand.

2. How can multivariate decoding analysis be implemented to inves-tigate the temporal dynamics of EEG activity related to Motor Imagery?

(28)

Averaging across time and trials hides the TDs of MI, but it is chal-lenging to visualize and correlate four dimensions (time, frequency and two spatial dimensions). TDs of MI activity is likely to vary be-tween subjects as well bebe-tween trials within subjects, and MI activ-ity should be quantified such that inter-trial variabilactiv-ity is comparable between subjects. In order to compare TDs of MI performance be-tween different subjects, a value must be derived from the MI pat-tern expression, or strength. This question relates to the methodolog-ical challenges of time-resolved multivariate analysis of EEG-data and aims to explore new methods for quantifying the strength of MI-related activity over time.

3. Considering both internal (i.e. brain dynamics) and external (i.e. hardware limitations) factors, how should real-time neurofeed-back be designed?

A real-time neurofeedback BCI should be designed based on the same methods used in offline analysis, but several adaptations must be made. Signal analysis requires pre-defined time windows. While longer time windows are beneficial for signal processing, they simul-taneously increase feedback delay in a real-time feedback system which may be detrimental for the feedback effectiveness. Further-more, operating systems such as Windows leads to nondeterministic feedback delays which further impacts the outcomes of a neurofeed-back experiment. A neurofeedneurofeed-back framework based on the methods used in offline analysis should be designed and empirically tested for delay and uncertainty. Then, the literature on delayed motor learning should be studied and compared to the properties of the framework in order to determine the design for a neurofeedback BCI.

Averaging across time and trials hides the TDs of MI, but it is chal-lenging to visualize and correlate four dimensions (time, frequency and two spatial dimensions). TDs of MI activity is likely to vary be-tween subjects as well bebe-tween trials within subjects, and MI activ-ity should be quantified such that inter-trial variabilactiv-ity is comparable between subjects. In order to compare TDs of MI performance be-tween different subjects, a value must be derived from the MI pat-tern expression, or strength. This question relates to the methodolog-ical challenges of time-resolved multivariate analysis of EEG-data and aims to explore new methods for quantifying the strength of MI-related activity over time.

3. Considering both internal (i.e. brain dynamics) and external (i.e. hardware limitations) factors, how should real-time neurofeed-back be designed?

A real-time neurofeedback BCI should be designed based on the same methods used in offline analysis, but several adaptations must be made. Signal analysis requires pre-defined time windows. While longer time windows are beneficial for signal processing, they simul-taneously increase feedback delay in a real-time feedback system which may be detrimental for the feedback effectiveness. Further-more, operating systems such as Windows leads to nondeterministic feedback delays which further impacts the outcomes of a neurofeed-back experiment. A neurofeedneurofeed-back framework based on the methods used in offline analysis should be designed and empirically tested for delay and uncertainty. Then, the literature on delayed motor learning should be studied and compared to the properties of the framework in order to determine the design for a neurofeedback BCI.

(29)

Temporal representation of Motor Imagery : towards improved Brain-Computer Interface-based strokerehabilitation

Brain-Computer Interface-based stroke

rehabilitation

TEMPORAL REPRESENTATION OF MOTOR IMAGERY

TEMPORAL REPRESENTATION OF MOTOR IMAGERY

Abstract

Abstract

Sammanfattning

Sammanfattning

I thank my family, friends and colleagues for encouraging me

during my studies and my supervisors for their indispensable

support.

I thank my family, friends and colleagues for encouraging me

during my studies and my supervisors for their indispensable

support.

List of Papers

List of Papers

Papers not included

Papers not included

Contents

Contents

Part I:

Thesis

Part I:

Thesis

1. Introduction

1. Introduction

2. Background

2. Background

3. Research Overview

3.1

Problem Formulation

be-3. Research Overview

3.1

Problem Formulation

3.2

Research Questions

3.2

Research Questions