Audio feedback in music: a study of experience of audio feedback in music for rehabilitation treatment for fear avoidance

(1)

IN

DEGREE PROJECT COMPUTER SCIENCE AND ENGINEERING, SECOND CYCLE, 30 CREDITS

STOCKHOLM SWEDEN 2017,

Audio feedback in music - a study of experience of audio feedback in music for rehabilitation treatment for fear avoidance

JENNY HANSDOTTER

KTH ROYAL INSTITUTE OF TECHNOLOGY

SCHOOL OF COMPUTER SCIENCE AND COMMUNICATION

(2)

Audio feedback in music - a study of experience of audio feedback in mu- sic for rehabilitation treatment for fear avoidance

JENNY HANSDOTTER

Master in Computer Science Date: December 23, 2017 Supervisor: Andre Holzapfel Examiner: Sten Ternström

Swedish title: Ljudfeedback i musik - en studie av upplevelse av ljudfeedback i musik inom rehabiliteringsbehandling för rörelserädsla School of Computer Science and Communication

(3)

i

Abstract

Rehabilitation treatments used for fear avoidance need to be further developed to be more adaptable to the different needs of patients. Fear avoidance is a condition whereby people avoid performing certain movements because they afraid of the pain they will experience or think they will experience from performing the movement. There is a need for exercises in fear-avoidance treatments to be fun, motivating and effective. This is to encourage the patients to continuously do exercises in the treatment. In this master thesis project, the experience of having audio feedback in music whilst performing exercises is investigated. A proof-of-concept prototype using one way of presenting audio feedback was built and used in experiments to investigate the experience. The prototype was built in Java and uses a camera-based motion capture system and markers to track movements. The results show that 100% of the participants thought about movements they had made when feedback was given, 60% claimed the feedback made them feel more aware of how they should move, and 70% said that the music with audio feedback was fun and/or exciting. The conclusion is that the use of music encouraged the participants to perform the exercises, and the feedback made them more conscious of the movements in the exercises and they reflect about the movements they made.

(4)

ii

Sammanfattning

Rehabiliteringsmetoder som används vid behandling av rörelserädsla behöver vidareut- vecklas för att behandlingar ska kunna anpassas bättre efter olika behov hos patienter.

Rörelserädsla är ett tillstånd där en person undviker att utföra vissa rörelser för att hon är rädda för smärtan som hon upplever eller tror hon kommer uppleva när rörelsen ut- förs. Det finns behov av att övningar som utförs i behandlingar är roliga, motiverande och effektiva i att behandla rörelserädsla. Detta för att uppmuntra patienter som lider av rörelserädsla att utföra övningar de fått i rehabiliteringsbehandlingen. Detta masterarbete undersöker upplevelsen av att ha auditiv återkoppling i musik medan övningar utförs.

En proof-of-concept prototyp som använder en typ av ljudfeedback byggdes och använ- des i experiment för att undersöka detta. Prototypen är byggd i Java och använder ett system för kamera-baserad rörelseföljning samt markörer för att följa rörelser. Resultaten visar att 100% av deltagarna tänkte på rörelser de utfört då ljudfeedback gavs i musiken, 60% tyckte att de blev mer medvetna om hur de skulle göra rörelserna och 70% sa att musik med ljudfeedback var kul och/eller spännande. Slutsatsen är att användandet av musik uppmuntrade deltagarna till att utföra övningarna samt att återkopplingen gjorde att de blev mer medvetna om rörelserna i övningarna och reflekterade kring rörelser de gjorde.

(5)

Introduction

Chronic pain is a common problem for people all over the world [1]. Low back pain af- fects people of all ages and in many cases the cause of the back pain is unknown. The treatments for low back pain that exist today are not sufficiently effective [2]. Low back pain is a big reason for people avoiding being active and being absent from the work- place [2]. In 2001 the cost of medical care for treating low back pain and lost work days was 1860 million EUR in Sweden [3]. When people suffer from pain for a long period of time they can develop a fear of performing certain movements [4, 5]. Chronic pain then means both a lower life quality for the individual due to pain and less activity and a large cost to society due to lost workdays.

Music has been used to structure interactions in gatherings of people, used for ed- ucational purposes and helps in making the learning and changing of behavioural patterns ordered and structured [6]. Sound and music used as feedback have been used in rehabilitation treatments for movement-related deficiencies such as stroke, injuries or operations on the brain. The studies have shown positive results when incorporat- ing music for supporting, motivating and inspiring patients in their recovery or training [7, 8, 9, 10, 11].

1.1 Problem statement

In view of the above background, we wished to investigate how to use audio feedback to inform the moving subject of deviating movement patterns and at the same time mo- tivate the subject to perform exercises. The exercises have been defined in collaboration with a rehabilitation expert, with the goal to resemble exercises used in rehabilitation treatments for back pain. There is an interest in finding ways of developing rehabilitation treatments for both practitioners and patients. The research question in this project is: Can one approach of audio feedback be used? And what is the experience of a motivating aspect and informative aspect for a person performing exercises?

The focus of this project is on investigating people’s experience of getting audio feedback to movements. The presentation of the feedback can be done with different ap- proaches. Only one approach of presenting audio feedback is used in the investigation.

The age or sex of participants in experiments is not a factor when collecting data. The participants do not suffer from fear of movement.

1

(8)

2 CHAPTER 1. INTRODUCTION

1.2 Purpose

The purpose of this thesis project is to investigate a way of presenting audio feedback, based on body movements, to persons. The aim is to get an idea of how audio feedback in the context of rehabilitation treatment exercises are experienced and test one way of giving audio feedback. The results of this project will be used in a project regarding audio feedback to movements in clinical exposure-based treatment of kinesiophobia.

1.3 Overview of the thesis

Chapter 2 will present what fear of movement means and studies that have been performed on treatments for fear of motion. It will also present music therapy and some areas where it is used. At last, the chapter will present sonification, biofeedback and studies using these techniques. Chapter 3 will present the method used in this study. Chapter 4 will present the results. The result and the method will be discussed in Chapter 5 and Chapter 6 presents a conclusion.

(9)

Chapter 2

Background

This chapter will give a background on what fear of movement is, the role of music and music therapy, as well as research that has been done with sonification and biofeedback in the area of rehabilitation treatment and sports.

2.1 Fear of movement

Fear of pain due to movements is also known as kinesiophobia or fear avoidance, the latter term will be used from this point in the thesis. People who suffer from chronic back pain may develop a fear of movements that cause, or they think may cause them pain. This leads to a behaviour of avoiding to perform these movements in everyday life [4, 5].

When medication is not the solution for treating a person suffering from chronic pain, rehabilitation treatments are used [12, 2] and as a last option surgery is performed [2]. In the rehabilitation treatment patients meets with a doctor, a physiotherapist and a psy- chologist. The practitioners talk with the patient to find the problem area and understand the patient’s needs from their different points of views. From that, a set of goals are formulated, and a treatment is constructed to reach the goals. The treatment could contain relaxation exercises, exposure training and behavioural therapy [13, 12]. The rehabilitation treatment is performed either in groups, individually or a combination of group and individual treatment [12].

Graded exposure in vivo (real life) and graded activity are methods of treatment which have been researched in different ways to see how they can be used to give the most beneficial results in the rehabilitation treatment [14, 15, 16]. Studies have shown that a combination of graded exposure in vivo and graded activity had a promising decrease in fear of movements, but that further and larger studies are needed to determine the long- term effect and to be able to tell if the results are generalizable [14, 15, 16, 17].

2.2 Music therapy

According to the American Music Therapy Association music therapy is defined as "...the clinical and evidence-based use of music interventions to accomplish individualized goals within a therapeutic relationship by a credentialed professional who has completed an approved music therapy program.” [18]. Within the setting of music therapy, a few studies have included therapists using music technology to help patients adjust to changes in

3

(10)

4 CHAPTER 2. BACKGROUND

physical abilities [7, 18], but also reduce pain in hospitalized patients and improve motor functions for people with Parkinson’s disease, among other things [18].

Music has been used by therapists and shown to give patients a sense of being in control, being independent, empowerment, achievement and giving them other ways to communicate [7]. Music and music therapy has also been shown to reduce stress, anxiety and tension for people preparing for surgery or who are in other hospital environments due to some illness or condition. Music has also been shown to help reduce pain after a surgery. [9]

2.3 Sonification and biofeedback in rehabilitation treatment and sports

Sonification has been defined by Hermann as “A technique that uses data as input and generates sound signals (eventually in response to optional additional excitation or trig- gering) may be called sonification, if and only if . . . ” four criteria apply. The criteria are i) that the sound produced is reflecting objective properties of the input data, ii) that the transformation from data to sound is systematic, iii) the transformation is reproducible, meaning that the same transformation should be produced from the same data, and iv) the system should be general enough to be able to use on different input data [19].

Biofeedback is a technique which uses biological information from a person and re- lays this information back to the person in real time. The information is either presented unprocessed from what was collected, or it is processed and presented in the form of a visual display, audio or haptic feedback [8].

Sonification as biofeedback has been used in research on rehabilitation treatments [20, 10, 11] and music technology that uses sonification of body movements have been used in therapy [7]. These studies have dealt with different types of deficiencies such as stroke [11, 6] and people whose needs have changed due to injuries or surgery [7, 6].

The quick feedback of sound being produced by movements is an indicator that works well in informing the initiator of the performance made. For stroke patients, it has been shown that adding training with a MIDI instrument to a regular rehabilitation treatment had a positive impact on the mobility of the patients, compared to receiving only con- ventional treatment [11]. For patients whose needs have changed due to an injury or surgery the ability to produce sound with instruments, both physical and electrical trig- gered by the movements of the body, was shown giving the patients a new way to communicate, participate and find a sense of accomplishment [7]. The use of biofeedback in research has indicated to be more effective when the exercises are goal oriented and contains dynamic movements, something to be accomplished by performing the exercise and re-education of muscle control [21].

Other studies have investigated how sonification could be used to aid in training or doing exercises. Greef et al. [20] designed a sonification system to ease the training of learning how to manoeuvre a wheelchair, Newbold et al. [10] researched how to use sonification of movements to help to perform stretch exercises correctly and Hermann and Zehe [22] used sonification to convey movements for visually impaired to enable them to participate in aerobics training.

(11)

Chapter 3

Method

In this project, a prototype was built to be able to perform experiments with audio feedback to movements used in rehabilitation exercises. The prototype was used in experiments with healthy subjects to collect data. Section 3.1 will describe the prototype and section 3.2 will describe the experiment.

3.1 The prototype

A proof-of-concept prototype was built to investigate the experience of having audio feedback to rehabilitation treatment exercises. A proof-of-concept prototype is built in cases where a certain aspect of a design, a technique, mechanic etc. should be tested [23].

This was a suitable type of prototype for this project where the use of audio feedback in an exercise context was investigated to see if it could be usable.

The prototype was designed to play a music file and manipulate the music played depending on evaluations of movement data collected from a moving subject. The manipulation of the music was done in a mirroring way. That is, the music mirrors the subject. The manipulations made were:

• subject is moving slowly - the rate of the music is decreased.

• subject is moving fast - the rate of the music is increased.

• subject hesitates (stops moving, makes a movement back and forth) - the music is pausing and playing.

• subject starts moving backwards on the track (changing the position of the whole body) - the volume sways.

Music was used to include a motivating aspect and it would be desirable to have the subjects choosing music they like. In the present setup, a specific music clip was chosen to simplify the experimental conditions. A 20-second clip [24] was looped and used as the music with which to present audio feedback. This music was chosen since it had a clear rhythm and a pace close to human gait.

Mirroring of the moving subject was chosen because of how it will change when the subject moves. If the subject moves as intended, the music will play as normal. If the subject moves in a wrong way the music playback will be manipulated. For example:

If person A is starting to slow down the movements, the rate of the music will become 5

(12)

6 CHAPTER 3. METHOD

slower. If A then speeds up again to an expected speed, the rate of the music will increase and go back to the normal rate.

The prototype was built to be able to cooperate with a program that collects data on movements made by a moving subject. The program that collects data was built by Joar Nykvist and was slightly modified by the author to make it usable for this prototype [25]. The modifications that were made are found in the branch named branch1 in the repository [25]. The program that collects data uses a motion capture system to collect the movement data.

Construct of prototype (see figure 3.1):

1. Main program starts the data collecting program in one thread.

2. Data collecting program calibrates the track and starts collecting movement data.

3. The main program starts the evaluation in another thread.

4. Movement data from the data collector is stored in a class which can be read by the evaluation program.

5. Evaluation reads data from the shared information class and uses it to evaluate movements.

Figure 3.1: Block diagram of the experimental prototype

The prototype was built in Java. Java was chosen since it provides libraries and frame- works for audio processing. TarsosDSP was the framework used in this project. Tar- sosDSP was chosen since it provides suitable ways of playing and manipulating music purely using Java. It is built for real-time audio analysis and processing [26, 27]. For a more detailed description of the prototype than the one above see Appendix B.

The movement data provided by the data collecting program was smoothed using exponential smoothing.

s₀= x₀

st= αxt+ (1 − α)st−1

Where s is the smoothed value, t is the time and x is the observed value.

(13)

CHAPTER 3. METHOD 7

3.2 Experiment

The experiment consisted of two parts. Part one consisted of tests with the prototype and part two was an interview.

3.2.1 Setup

To test the prototype the participants walked a track consisting of exercises used in rehabilitation treatment for fear avoidance. The exercises used in the experiment were defined in collaboration with a rehabilitation expert. See figure 3.2 for an image of the track where the exercises were performed and Appendix C for photos. The exercise started with the participant sitting down on a chair (a). Then standing up, walking to the step board and stepping up onto it (b). After this, the participant jumped down and walked to the box (c). Bent the knees and picked the box up. Stood up with the box and then put it back down. Then the participant turned around and walked back to the step board, stepped up, jumped down, walked to the chair and sat down.

Figure 3.2: The track

Each participant walked the track a total of 4 times. Two times with and two times without audio feedback. One of the times with and without audio feedback the participants were presented with distractions to try and create breaks in their normal movements. The distractions were also used to see if music has a stabilizing action in the condition where distractions are present. The order in which the participants walked the track was randomized by simple random sample without replacement.

The distractions were made by tossing bags of rice in the participant’s field of vision in different areas of the track. Bags of rice were used since they do not present a risk of unforeseen movements after landing.

The technical equipment used in the experiment was:

• 14 markers, 12 to place on the participant and 2 to configure the track. For placement on the participant see Appendix A.1.

• Laboratory for multimodal performance interaction (PMIL) including a modern camera-based motion capture system with passive markers (17 cameras, 1 mm spa- tial resolution).

• 2 speakers were used to play the music.

(14)

8 CHAPTER 3. METHOD

3.2.2 Interview

The second part of the experiment consisted of an interview. The aim of the interview was to collect the participant’s thoughts and experience of getting audio feedback while performing the exercises. The interview questions can be found in Appendix A.2.

3.2.3 Participants

10 people participated in the experiment. The participants did not suffer from fear avoidance. There were 5 women and 5 men participating. Ages ranged from 18 to 59. During the experiment one participant at a time was present. Each participant signed a consent form before starting the experiment.

(15)

Chapter 4

Results

Section 4.1 will present data collected from the prototype on registered activating points for audio feedback. Section 4.2 will present data from the interviews.

10 people participated in the experiments.

4.1 Data from prototype

Figure 4.1: Average number of registered instances of activating points for the participants in each condition

Figure 4.1 shows the average number of instances of activating points each participant had when walking the track with different conditions. The x-axis shows the 10 participants and the y-axis shows the average number of times an activating point occurred.

9

(16)

10 CHAPTER 4. RESULTS

Figure 4.2: Total number of registered activating points for all participants

Figure 4.2 shows the total number of activating points during the four different walks made by the participants. +M means with music, -M without music, +D with distractions and -D without distractions. The total number of activating points given when there was music playing and no distractions (+M, -D) was 56. The total number of activating points during all three other conditions was 58.

Figure 4.3: Total number of registered activating points for all participants with instances caused by technical problem removed

Figure 4.3 shows the total number of activating points during the four different walks made by the participants with the activating points caused by technical problems removed. +M means with music, -M without music, +D with distractions and -D without distractions. The total number of activating points given when there was music playing and no distractions (+M, -D) was 51. The total number of activating points given when there was music playing and distractions (+M, +D) was 52. The total number of activating points given when there was no music playing and no distractions (-M, -D) was 51.

The total number of activating points given when there was no music playing and dis-

(17)

CHAPTER 4. RESULTS 11

tractions (-M, +D) was 49.

4.2 ANOVA test

4.2.1 Total number of activating points

A one-way ANOVA test on the result of the number of activating points during the different setups for walking the track was performed.

Setup for ANOVA test:

H0: The means are equal. H1: The means are not equal.

The critical value was 2.87, with df1= 3 and df2 = 36.

α = 0.05

Gained F value in test was 0.02128. H0 cannot be rejected. This means that it cannot be said that there is a significant difference in the number of activating points during the different conditions of the four walks the participants made.

4.2.2 Total number of activating points, with activating points caused by tech- nical problems removed

A one-way ANOVA test on the result of the number of activating points during the different setups for walking the track was performed.

Setup for ANOVA test:

H0: The means are equal. H1: The means are not equal.

The critical value was 2.87, with df1= 3 and df2 = 36.

α = 0.05

Gained F value in test was 0.0605. H0 cannot be rejected. This means that it cannot be said that there is a significant difference in the number of activating points during the different conditions of the four walks the participants made.

4.3 Data from interviews

Figure 4.4: Helpful - How helpful the feedback was to the participants

(18)

12 CHAPTER 4. RESULTS

Figure 4.4 shows how the participants graded the experience of having audio feedback. The x-axis represents the grade, where 1 is not helpful, 2 is a little helpful, 3 is pretty helpful, 4 is helpful and 5 is very helpful. The y-axis represents the number of people who gave a certain grade.

Motivations for grade Motivations

Expressed that music and feedback made it fun and/or exciting 70%

Expressed that they felt disturbed by changes in the audio 40%

Table 4.1: Feelings felt when receiving feedback

Table 4.1 shows thoughts of how the participants felt during the experiment. These thoughts were given when they motivated the grade they gave.

Mirroring or non mirroring preferred

Preferred feedback

Mirroring 50%

Non mirroring 50%

Table 4.2: Preferred way of getting feedback

Thoughts of movements Thought of movements

Conscious of movements made causing the feedback (during experiment) 100%

Gave suggestions of movements causing feedback (during interview) 70%

Consciously changed movement based on feedback 20%

Felt that they became more aware 60%

Table 4.3: Thoughts of movements when receiving feedback

Table 4.3 shows in what different ways participants thought of the movements performed.

Thoughts on the changes made to the music Thought of changes

Change of music rate disturbed the movements 20%

Pausing and playing disturbed the movements 60%

Table 4.4: Thoughts of the changes in the music when feedback was given

(19)

CHAPTER 4. RESULTS 13

Reactions to distractions

Reaction

Reacted physically 30%

Talked or laughed 60%

Ignored the bags 10%

Table 4.5: Reactions to the distractions

Table 4.5 shows how participants reacted to the bags of rice being tossed in their field of vision.

(20)

Chapter 5

Discussion and evaluation

The first part of this chapter will discuss the results of the experiment, the second will discuss the advantages and problems of the method used and the third will present future work.

5.1 The results

In this section, the results from the data gathered with the prototype will be discussed first. Then the answers from the interviews and motivations given to the answers will be discussed.

5.1.1 Data collected from the prototype

The data collected from the prototype on each participants time they walked the track shows that the participants seem to have reacted differently to the given conditions; how- ever, the number of observations per participant is too small to rule out a random effect (see figure 4.1).

The participants’ performance cannot be said to have been improved with the feedback in the music. This is shown in the ANOVA test (section 4.2.1) which shows that

there is no significant difference between the conditions (with/without music and with/without distractions). This is also supported by the results in figure 4.2. A reason why there is

not enough difference between the different conditions for walking the track could be that none of the participants suffer from fear avoidance. Also, the distractions did not have the same effect on all participants. This means that some participants reacted to the distraction in such a way that feedback was given, but some participants did not react much or at all (see 5.2.4 where the distractions are discussed in more detail).

The feedback that was given also depended on how the prototype was constructed and how the data was gathered. See section 5.2.1 and 5.2.3 for discussion of the impact these had on the result.

5.1.2 Interview responses

This section will discuss the different parts of the interview. First, the aspect of helpfulness is discussed. Second, the way the feedback was presented is discussed and lastly the thoughts about the movements made.

14

(21)

CHAPTER 5. DISCUSSION AND EVALUATION 15

Helpful

The participants graded how helpful it was to get audio feedback on their movements (see figure 4.4). Motivations given by participants who gave a value 1-3 were that the changes in the feedback were not happening in real time. That the timing of the feedback did not happen in real time made it difficult for the participants to know what movement the feedback referred to and thus made it difficult to understand what they had done wrong. They also found that the changes in the music that informed the participant of hesitant movements were disturbing the movements overall. Grades 3 and above were given with motivations that to get audio feedback was informative, worked as a re- minder of how to do the movements and reminded the participant to be conscious of the movements.

The participants talked about helpfulness in general (see table 4.1). 70% mentioned that to walk the track with music playing and audio feedback made it more fun and/or exciting. (The last 30% did not make such a comparison). 40% mentioned that they felt that all changes in the audio disrupted the rhythm in their movement. These comments suggest that music brings motivation to the participants. It also suggests that the feedback presents a challenge where the music is more than something to have in the background whilst doing exercises. This is also seen in previous studies where the use of music gave patients a feeling of achievement [7] and goal orientation in the exercises has been indicated to be effective [21].

In this project, the changes that occurred in the music when feedback was given probably needed to be more clearly explained at the beginning. This is because people use music in different ways in everyday life; for example, in dancing and for studying, music is used differently. The use of music as applied in the experiment was new to the participants.

Mirroring type of audio feedback

The audio feedback was done in a mirroring way (see table 4.2). The participants who were positive about having the audio feedback mirroring their movements said that they felt that the feedback made them understand how they had moved. The participants who felt they would have preferred another way of getting feedback gave descriptions of music or sounds that they could adapt the movements to, i.e., a leading type of feedback.

A mirroring type of feedback was chosen because of the qualities it has when choosing and manipulating the audio. A mirroring type of feedback is possible to use on a bigger variety of audio than a leading way because of the way the music would change due to movements (see section 3.1). Mirroring type of feedback also allows the possibil- ity for letting the user choose the music.

Depending on the audio used, leading audio feedback could be a choice to consider.

It was one of the options considered by the author when deciding on the method to use in this project. But for music, audio feedback in a leading way could present some additional challenges concerning letting the user hear the music they chose in the way the user expects it to sound. For example: If person A is starting to slow down a movement in the exercise, the rate of the music will increase to encourage the user to move faster. If A then moves faster, the music would keep that speed to keep A moving at the correct pace. There could then be a problem with the music being played at the wrong pace and

(22)

16 CHAPTER 5. DISCUSSION AND EVALUATION

A’s expectation on how the music they chose should sound. Because if the rate of the music goes back to normal when A increased the pace of the movement there is a chance that A follows the music and slows down again.

With the music that was used in the experiment for this thesis both a mirroring or leading type of feedback could have been applied. This might have influenced the way the participants reasoned about the way the feedback was given during the experiment.

Thoughts of movements

Participants felt more conscious and reflected on movements when feedback was given.

All participants became more conscious of movements they had made when feedback was given (see table 4.3). There were participants who gave examples of movements that could have caused the feedback, participants who consciously changed their movements based on the information received from the audio feedback and participants who felt more aware of the movements. That the participants felt more conscious and reflected on movements suggests that even though there is no support for the movements being affected by the feedback (see section 4.2), the feedback made them think about how they were moving. This could mean that even the participants who did not feel more aware when performing different movements still reflected on the movement after the feedback was given. This could have effects in situations where a person performs the exercises as part of a rehabilitation treatment. In such a situation, reflecting of movements one day could have an impact on the exercises done the following day.

5.1.3 Social, economic, ecological sustainability and ethical aspects

The problem for people who suffer from fear avoidance is that their quality of life is affected since they avoid performing certain movements, which make them avoid participating in certain activities or social events that include or could include doing the movement they are avoiding [2]. Fear avoidance could also affect the person’s ability to work.

Depending on the work they do, they might not be able to do all tasks needed for the work, which leads to missed work days [2] and costs for treatments to deal with the pain [3].

The treatments that address fear avoidance need to be developed to be more efficient and to give practitioners more choices in how to design the treatments for patients. With more effective treatments people could overcome or get tools to deal with the fear avoidance and not need to avoid participating in activities. It could also help them not to be absent from work.

Patients could get more options of eco-friendly alternatives to travel if treatments were more effective. Since activities they might have avoided could include walking or riding a bicycle, they might have travelled by car or other vehicles even short distances.

From an ethical point of view, better methods in fear-avoidance treatment are relevant because it helps patients get their life back. It also helps society as a whole since the people suffering from fear avoidance will not be absent for work and they will not in- cur as many medical costs (which in Sweden is not only a cost for the person who gets the treatment). The consequences of developing the treatments are positive for those who need treatment to work with fear avoidance.

Because of this, the use of audio feedback in combination with music is of interest as a potential way to develop treatments. The results in this thesis suggest that the use of

(23)

audio feedback in music has the potential to be a motivating factor and encourages the patients to reflect on movements as feedback is given. Also see section 5.3 for suggestions on further development of what has been done in this project.

5.2 The method

In this section, motivation for the chosen method will be presented and the different parts of the method will be discussed and evaluated. The first part will present the motivation. The second part will discuss the process of building the prototype. The third part will discuss the chosen audio and its use to give feedback. The fourth part will discuss the technique used. The fifth part will discuss the experiment.

5.2.1 Motivation for chosen method

To be able to gather information on the experience of performing movements and receiving audio feedback based on the movements, a prototype presents a way to be able to try this mechanic without having a finished product. A proof-of-concept prototype was chosen since it provides a suitable level of the prototype. Since the aspect of having audio feedback based on movements was to be investigated.

To be able to gather data on what the experience is for people who perform movements and receive audio feedback interviews were chosen. Interviews provide a form of information gathering where the interviewed gets preestablished questions, but can use their own words to answer and explain the answer to the questions. This is suitable since in this project information of the experience, motivation and information received from the audio feedback was of interest.

The data on number of activation points for audio feedback was used to get information on if the feedback affected the participants movements. This to be able to see what effects the feedback could have on the participants in comparison to when they walked the track in silence.

5.2.2 The prototype

This section will discuss the different parts of the prototype. First, the building of the prototype is discussed. Second, the way evaluation was made and lastly the language used.

The building process

The prototype’s evaluation functionality was built using movement data from one person. The reason data from one person was used was that it does not present a problem for constructing a proof-of-concept prototype and investigate the problem at hand. Fur- thermore, the time it took to collect relevant data made it problematic to schedule many people. This meant that the prototype could make incorrect evaluations on the participant’s data due to the length or body type being different from the data collected during development. This could result in manipulation of the audio played, even though the participant was not making an incorrect movement. This consequently also influenced the data collected by the prototype.

(24)

The focus of this project was to investigate the experience of using music as the car- rier of audio feedback. This was achieved with this prototype since the participants got audio feedback on which they could offer their opinions. They could also comment on the experience of having music and audio feedback compared to the experience of walking the track in silence.

Evaluation

The evaluations made by the prototype was not considering different heights or body types of people. To have made the prototype more adaptable to different types of people, data on many people and the incorporation of machine learning could have been a choice. Then the prototype would be closer to a finished product.

The language

The language used was Java, with the TarsosDSP framework used to process audio. The framework used has the drawback that real-time low-latency for the audio pipeline is difficult to achieve without optimization [26]. This together with the threads in the prototype resulted in a slight delay of manipulations of the audio. This was not a problem for the prototype and the experiments. But it had an impact on the participants’ experience of the audio feedback. 70% mentioned that the delayed audio changes were disruptive because it made it more difficult to interpret what movement it referred to since the audio did not return to normal when they had finished one movement but could be active during the next as well. (An example of a movement is lifting the leg to step up on the step board and the next movement is to stand on top of the step board). This could also have influenced the judgment of how helpful they found the audio feedback.

5.2.3 The audio

Music was chosen as sound feedback to fulfil the need of making the exercises motivating to perform. The discussions in the larger project to which this thesis project is con- nected had the motivating part as an important factor to consider in rehabilitation treatments.

The manipulations were designed to mirror the movements of the moving subject.

The choice of mirroring was made with the aim of giving the user the goal of keeping the music playing as it should by performing the exercises correctly. Also with mirroring, as described in the example in section 3.1, the correction of an incorrect movement pattern would restore the music to its original state. The music used in this project was a rhythmic loop of music and the music was unknown to the participants. This type of music could have led the participants to have a different kind of opinion on the feedback than if it had been music they knew from before.

The participants commented on the changes made in the audio. The ways chosen to inform of the incorrect movements were, as mentioned in 3.1, decrease the rate of music, increase the rate of music, pause/play effect and sway with volume. Only the three first of the manipulations occurred during the experiments. The participants understood what the changes in the rate of the music meant. A few participants expressed that they found that changes in the rate of playback was disturbing (see table 4.4). These were people who mentioned they used the music to keep a steady pace when they walked the track.

(25)

The pause and play effect was hard to understand what it meant and 60% found that it was disturbing their movements. To inform of a hesitating movement another way of manipulation could have been used. For example, jumping back a bit in the music, play it from there and repeat this until the hesitant movement was corrected or using pitch shifting in some way.

5.2.4 Impact of technological environment

During the testing of the prototype a technical problem sometimes occurred. It was no- ticed that the collected data could show values far outside of reasonable values for short moments. An example would be the value of the knee angle which was used to evaluate movements. A person standing still showed usually an angle between 150 and 180 degrees, but suddenly the angle could be showing values below 70 degrees without the person moving the legs. This was discussed with the person who developed the data collecting program. The reason could be that the number of markers used, 14, is quite many markers for the camera system to keep track of.

As seen in figure 4.3, there was 27 of the 230 (11.7%) instances of feedback that were based on data that was outside the range of reasonable values for the movement.

• 5 out of 56 for the condition of having music playing and no distractions.

• 6 out of 58 for the condition of having music playing and distractions.

• 7 out of 58 for the condition of walking in silence and having no distractions.

• 9 out of 58 for the condition of walking in silence and having distractions.

The ANOVA test (see section 4.2.2) shows that even with the activating points based on unreasonable data values removed, there is no significant difference between having walked the track with different conditions (with/without music and with/without distractions). The conclusion is the same as mentioned in section 5.1.1. This suggests that the values outside of the reasonable range did not influence the outcome of the results for the collected data.

To avoid this problem fewer markers could be tried to use. For this project, the markers of one arm, one of the shoulders and one of the hips could probably have been removed. This would mean to do more changes in the program that collects movement data. Due to the time constraints of the project the extended changes to the data collecting program were not made.

For the participants, the activating points based on unreasonable data was only no- ticed when the music was playing. For the condition +M -D there were 4 participants who got feedback based on unreasonable values. 5 participants got feedback based on unreasonable values during the condition +M +D. These were 6 different participants who had instances of unreasonable data whilst walking. None of them had more than two instances of feedback given based on unreasonable values of data. There is not a pattern of these participants expressing their experiences in a more negative or positive way. This suggests that the feedback given based on unreasonable data values did not affect the participants experience in a specific way.

(26)

5.2.5 The experiment

Distractions in the form of tossing bags of rice were used to create breaks in the movement of the participants. 30% of the participants had physical reactions to the first and second bag being tossed. Their reaction was to stand still for a little while. 60% said something or laughed at the first being tossed and 10% ignored them (see table 4.5). Be- cause of this, the distractions presented seemed to not have been enough to give distinct physical reactions for most of the participants. This could be a reason why there was not a noticeable difference between having distractions or not in the data collected from the prototype (see figure 4.2).

5.3 Future work

A similar study needs to be done using music more similar to music that could be chosen by a person. Since the experience of manipulations could be different with music which has more variations in the same song.

The prototype could be improved by using a data collecting program which uses fewer markers. This could help in avoid collecting incorrect data.

A second prototype should use a language which is better suited for real time reactions without extensive optimization to be done.

Lastly, the prototype may benefit from being more adaptable to be used on people of different heights and body types. One way of doing this could be to use machine learning when constructing the prototype. This would help in avoiding incorrect evaluations of the movements.

(27)

Chapter 6

Conclusion

In this project, the focus was to investigate the experience of having audio feedback in music, with the aim of being used in rehabilitation treatments for fear avoidance. In these treatments, an important factor is to have motivating exercises where movements that patients are afraid of doing are worked with.

In this project, a proof-of-concept prototype was built to investigate the experience of having music with feedback incorporated in it. The feedback was based on movements made by a moving subject. The results show that 70% of the participants thought it was fun and/or exciting to have feedback incorporated in music. Which suggests that the use of music and feedback had a motivating impact and provided a positive experience. The participants thought of movements they made in the different areas of the track when presented with feedback. This was not affected by if they liked or disliked the mirroring feedback, all participants reflected on movements they made. Which is shown by 100%

of the participants being conscious of movements they had made during the experiment and 60% became more aware of the movements in the exercises. But further studies are needed to investigate the experience of getting audio feedback when using other type of music than the one used in this experiment and to find other ways of presenting feedback.

21

(28)

Bibliography

[1] Daniel S. Goldberg and Summer J. McGee. Pain as a global public health priority.

BMC Public Health, 11(1), 2011.

[2] Warren Kaplan, Veronika J. Wirtz, Aukje Mantel-Teeuwisse, Pieter Stolk, Béatrice Duthey, and Richard Laing. Priority medicines for europe and the world 2013 up- date. Technical report, World Health Organization, 2013.

[3] Mattias Ekman, Olof Johnell, and Lars Lidgren. The economic cost of low back pain in Sweden in 2001. Acta Orthopaedica, 76(2):275–284, 2005.

[4] Mari Lundberg. Rörelserädsla - ett centralt begrepp för sjukgymnaster.

http://www.fysioterapi.se/admin/filer/Fysioterapi_nr7-13_

forskningpagarNY.pdf, 2013.

[5] Back Pain Solutions Online. Addressing fear of movement necessary in treatment of chronic back pain. http://backpainsolutionsonline.com/

announcements-and-releases/back-pain-therapy/low-back-pain- treatment/addressing-fear-of-movement-necessary-in-treatment- of-chronic-back-pain, 2012.

[6] Michael H. Thaut. Rythm, music and the brain. Routledge, 2008.

[7] Karen Burland and Wendy Magee. Developing identities using music technology in therapeutic settings. Psychology of Music, 42(2):177–189, 2014.

[8] Oonagh M. Giggins, Ulrik McCarthy Persson, and Brian Caulfield. Biofeedback in rehabilitation. Journal of NeuroEngineering and Rehabilitation, 10(1), 2013.

[9] Kathi J. Kemper and Suzanne C. Danhauer. Music as therapy. Southern Medical Jour- nal, 98(3):282–288, 2005.

[10] Joseph W. Newbold, Nadia Bianchi-Berthouze, Nicolas E. Gold, Ana Tajadura- Jiménez, and Amanda CdC Williams. Musically informed sonification for chronic pain rehabilitation: Facilitating progress & avoiding over-doing. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, pages 5698–5703, 2016.

[11] Sabine Schneider, Paul W. Schönle, Eckart Altenmüller, and Thomas F. Münte. Using musical instruments to improve motor skill recovery following a stroke. Journal of neurology, 254(10):1339–1346, 2007.

[12] Arbetsgrupp Smärta. Rekommendationer för behandling av långvarig icke malign smärta utöver läkemedelsbehandling. Technical report, Örebro läns landsting, 2014.

22

(29)

BIBLIOGRAPHY 23

[13] Birgitta Helmerson Ackelman, Ingegerd Johansson, and Adrianne Levy Berg. Vård- program för fysioterapeutisk intervention - flödesschema psykosomatik. Technical report, Karolinska Universitetssjukhus, 2014.

[14] Steven J. Linton, Katja Boersma, Markus Jansson, Thomas Overmeer, Karin Lind- blom, and Johan W.S. Vlayen. A randomized controlled trial of exposure in vivo for patients with spinal pain reporting fear of work-related activities. European Journal of Pain, 12(6):722–730, 2008.

[15] Steven J. Linton, Thomas Overmeer, Markus Janson, Johan W.S. Vlaeyen, and Jeroen R. de Jong. Graded in vivo exposure treatment for fear-avoidant pain patients with functional disability: A case study. Cognitive Behaviour Therapy, 31(2):49–

58, 2002.

[16] Johan W.S. Vlaeyen, Jeroen de Jong, Mario Geilen, Peter H.T.G. Heuts, and Gerard van Breukelen. Graded exposure in vivo in the treatment of pain-related fear: a replicated single-case experimental design in four patients with chronic low back pain. Behaviour Research and Therapy, 39(2):151–166, 2001.

[17] Johan W.S. Vlaeyen, Jeroen R. de Jong, Patrick Onghena, Maria Kerckhoffs-Hanssen, and Ank M.J. Kole-Snijders. Can pain-related fear be reduced? the application of cognitive-behavioural exposure in vivo. Pain Res Manage, 7(3):144–153, 2002.

[18] American Music Therapy Association. Definition and quotes about music therapy.

http://www.musictherapy.org/about/quotes/.

[19] Thomas Hermann. Taxonomy and definitions for sonification and auditory display.

In Proceedings of the 14th International Conference on Auditory Display (ICAD 2008), 2008.

[20] Andreas Almqvist Greef, Ludvig Elblaus, and Kjetil Falkenberg Hansen. Sonification as catalyst in training manual wheelchair operation for sports and everyday life. In Proceedings of the Sound and Music Computing Conference, SMC 2016, 2016.

[21] He Huang, Steven L. Wolf, and Jiping He. Recent developments in biofeedback for neuromotor rehabilitation. Journal of NeuroEngineering and Rehabilitation, 3(1), 2006.

[22] Thomas Hermann and Sebastian Zehe. Sonified aerobics - interactive sonification of coordinated body movements. In The 17th International Conference on Auditory Display (ICAD-2011), 2011.

[23] 3D PROTOTYPE DESIGNER. Proof-of-principle prototype. http://3d-printing- expert.com/proof-of-principle-prototype/, 2015.

[24] laurentwirz. Punchy kick groove funk 98 bpm. https://www.looperman.

com/loops/detail/108755/punchy-kick-groove-funk-98-bpm-by- laurentwirz-free-98bpm-funk-drum-loop, May 2017.

[25] Joar Nykvist. https://github.com/jn01674361/mck, 2017.

[26] Joren Six, Olmo Cornelis, and Marc Leman. Tarsosdsp, a real-time audio processing framework in java. In AES 53RD INTERNATIONAL CONFERENCE, 2014.

[27] Joren Six. TarsosDSP. https://github.com/JorenSix/TarsosDSP.

(30)

24 BIBLIOGRAPHY

[28] OpenClipart-Vectors. boy-2025115_1280. https://pixabay.com/sv/pojke- m\%C3\%A4nskliga-manliga-mannen-2025115/, 2017.

(31)

Appendix A

Information related to experiment

A.1 Placement of markers

Marker placement:

• One marker just above each knee, front of the leg.

• One marker on the ankle, on the outside of the leg.

• One marker on each hip.

• One marker on each shoulder.

• One marker on each wrist.

• One marker just above the elbow, on the outside of the arm.

Placement of markers. Silhouette [28].

25

(32)

26 APPENDIX A. INFORMATION RELATED TO EXPERIMENT

A.2 Interview questions

1. Did you notice the sound changing at any point while you walked the obstacle course? If yes, what changed?

2. At the time you heard the sound change, what did you think caused it? How did you interpret the change in sound?

3. What movement do you think induced the change in sound, now afterwards?

4. Did the change of the sound make you aware of how you were moving?

5. How helpful did you find the sound feedback? (Helpful in the meaning that it made you aware of the movements you made) On a scale 1-5 (1 is lowers, 5 high- est)

6. Why did you give the above rating (on 5)?

7. How did the sound modifications feel to use? (What other modifications would have felt good)

8. Other comments about the prototype.

(33)

Appendix B

Structure of prototype

The structure of the whole prototype:

• Main: Main interacts with the user and starts threads to run the data collecting pro- gram and Player (the evaluation program).

• Data collector: Data collecting program (named MCK in [25]) calibrates the track and starts collecting movement data. Movement data from the data collector is stored in the class SharedInformation which can be read by the evaluation program Player.

• Shared information: Contains data collected by the data collector. This is data on what position on the track the person is in and what movements of legs, arms and torso are. This class has a smoothing function to smooth the values that are collected before using them in evaluations. An alpha of 0.7 was used. The reason for that was to have the previous value have some effect on the new value but not too much. But with a low alpha the previous value got too much influence and movements that were made got lost. This was because new movement data was collected every 0.5th second and this makes fast movements disappear and slow

27

(34)

28 APPENDIX B. STRUCTURE OF PROTOTYPE

movements could be judged as standing still and hesitant movements disappeared in most cases.

• Player: This program plays and manipulates the music depending on evaluations made from the movement data. It evaluates to find fast and slow movements, hesitant movements and how the position of the person changes. This class uses Hes- itation checker to evaluate if a movement is done hesitantly and it uses Person to store the current information regarding the person on the track.

If a movement is evaluated to be fast, slow, hesitant or the position changes in a wrong way the music is manipulated in different ways (see section 3.1).

Fast and slow movements are evaluated in situations where the knees are bending (when a person interacts with the chair, the step board or the box). The fast/slow movements are found by looking at the changes in knee angles from the movement data. When the person has a large change in the angle between two data instances it is evaluated to be fast. When the angle change is small between three instances the movements is evaluated to be slow. Evaluation of hesitant movements are men- tioned in the bullet point Hesitation checker below. The position changes are used in looking for hesitant movements and for checking if a person moves in the direction opposite of what they should. To look for position changes data instances collected on the position of the person is used.

• Person: When Player is created and started it creates a Person object. This object describes the person on the track. It stores:

– What position on the track the person is in (this is the exact position on the track).

– What area the person is in. See figure 3.2 for an image of track and bullet pint Area below.

– What area the person was in previously.

– What state the person is in – What state it was in previously.

– What direction the person is moving in, with the chair as the starting point.

– Which leg is being lifted in area b (see figure 3.2).

– How long the person has been standing still in one position.

– How long the person has been standing still and not bent the legs.

• Area: The person can be positioned in different areas (see figure 3.2). The areas are:

– Area a: the chair area.

– Area b: the step board area.

– Area c: the box area.

– The area between a - b and b - c the gait area. Where the person walks between areas.

• Person state: The person can be in different states when moving and performing the exercises on the track.

(35)

APPENDIX B. STRUCTURE OF PROTOTYPE 29

– In the chair area, the person is either sitting or standing/walking.

– In the gait area, the person is in a state of standing/walking.

– In the step board area, the person is either in the process of lifting the leg to step up, the person is on top of the step board or has jumped down and is standing/walking.

– In the box area, the person is either in the process of bending down to pick up or put down the box or in the process of rising up with or without the box.

• Hesitation checker: This class is used by Player to look for hesitant movements when the person is in the different areas. It looks for movements done back and forth or if the person stops for a long time (meaning the person stops and does not perform the movement or stops moving forward). These are looked at in the different areas a person can be in.

(36)

Appendix C

Pictures of experiment environment

Figure C.1: The course .

30

(37)

APPENDIX C. PICTURES OF EXPERIMENT ENVIRONMENT 31

Figure C.2: The course - chairs and step board .

Figure C.3: The course - box and step board .

(38)

www.kth.se

Audio feedback in music: a study of experience of audio feedback in music for rehabilitation treatment for fear avoidance

Audio feedback in music - a study of experience of audio feedback in music for rehabilitation treatment for fear avoidance

JENNY HANSDOTTER

Audio feedback in music - a study of experience of audio feedback in mu- sic for rehabilitation treatment for fear avoidance

Abstract

Sammanfattning

Contents

Chapter 1

Introduction

1.1 Problem statement

1.2 Purpose

1.3 Overview of the thesis

Chapter 2

Background

2.1 Fear of movement

2.2 Music therapy

2.3 Sonification and biofeedback in rehabilitation treatment and sports

Chapter 3

Method

3.1 The prototype

3.2 Experiment

Chapter 4

Results

4.1 Data from prototype

4.2 ANOVA test

4.3 Data from interviews

Chapter 5

Discussion and evaluation

5.1 The results

5.2 The method

5.3 Future work

Chapter 6

Conclusion

Bibliography

Appendix A

Information related to experiment

A.1 Placement of markers

A.2 Interview questions

Appendix B

Structure of prototype

Appendix C

Pictures of experiment environment