Bachelor of Science in Computer Science May 2021
Using Gamification to Improve User Experience and Health Effects in
Mobile Applications
Jonathan Andersson
Faculty of Computing, Blekinge Institute of Technology, 371 79 Karlskrona, Sweden
This thesis is submitted to the Faculty of Computing at Blekinge Institute of Technology in partial fulfilment of the requirements for the degree of Bachelor of Science in Computer Science.
The thesis is equivalent to 10 weeks of full time studies.
The authors declare that they are the sole authors of this thesis and that they have not used any sources other than those listed in the bibliography and identified as references. They further declare that they have not submitted this thesis at any other institution to obtain a degree.
Contact Information:
Author(s):
Jonathan Andersson
E-mail: jobv18@student.bth.se
University advisor:
Associate Senior Lecturer Yan Hu
Department of Computer Science (DIDA)
Faculty of Computing Internet : www.bth.se
Blekinge Institute of Technology Phone : +46 455 38 50 00 SE–371 79 Karlskrona, Sweden Fax : +46 455 38 50 57
Abstract
Background. According to the World Health Organization, over 264 million people suffer from depression. A recent trend to treat and combat depression is e-health application like Headspace with the help of mindfulness or meditation. The rise of new treatment methods based on these concepts are seen as a promising alternative to traditional methods like cognitive behavioural therapy and medication.
Objectives. The objectives of this study is to make a new mobile application, in the form of a mobile e-health prototype. The application, called MindBud, is designed to help the user reduce depressive thoughts. This is done by using a daily schedule to plan your day and in turn, reduce depressive thoughts and procrastination through structure. Then, the study seeks to compare two versions of this application, one ver- sion will have gamification elements and one will be without them. The comparison will measure overall user experience through a test called the system usability scale, and in addition measure the effectiveness of the application on depressive thoughts.
Methods. Two versions of MindBud were implemented, one basic app and one with gamification elements added to it. The applications were then tested by per- forming an experiment with sixteen participants. Each of the participants tested both versions of the application, and then answered a questionnaire about the app.
The answers of the questionnaire were used to compare test scores between the two versions of the application, to see if gamification had any impact on overall user ex- perience and to see which gamification elements could be used to reduce depressive thoughts through the application.
Results. The results show a slight increase in score in regards to overall user expe- rience when comparing the gamified app with the basic one. Most notable increases came in questions about frequency of use, and complexity of the application. Addi- tionally, the gamified application scored significantly better when participants were asked how much they thought the app version would reduce depressive thoughts.
Conclusions. The gamification elements added were found to increase overall user experience, and also help reduce depressive thoughts more than the basic version.
The used gamification elements were an in-game avatar, a reward system and an experience and level system.
Keywords: gamification; mobile applications; usability; mental health; e-health
Acknowledgments
I would like to thank my supervisor Yan for always responding and helping when it was needed. I would also like to thank my family and my partner Sofia for always being there whenever I needed them.
Another big thanks goes to all the people who took part in the experiment.
iii
Contents
Abstract i
Acknowledgments iii
1 Introduction 1
1.1 Aim and objectives . . . . 2
1.1.1 Objectives and goals . . . . 2
1.1.2 Research questions . . . . 3
1.2 Background . . . . 3
1.2.1 Key Terminology . . . . 4
1.3 Scope . . . . 4
1.4 Outline . . . . 5
2 Related Work 7 2.1 Gamification in E-Health . . . . 7
2.2 Gamification of Meditation . . . . 7
2.3 Using Gamification in Medical Assessment . . . . 8
2.4 Gamification in Software Engineering . . . . 9
3 Method 11 3.1 Methodology . . . . 11
3.2 Implementation . . . . 12
3.3 Experiment . . . . 16
3.4 Other Alternatives . . . . 17
3.5 Data Collection . . . . 17
3.6 Working Conditions . . . . 18
3.7 Ethical, Societal and Sustainability Aspects . . . . 18
4 Results and Analysis 21 4.1 SUS Score . . . . 21
4.2 Depressive Thoughts . . . . 22
4.3 Additional Features . . . . 24
4.3.1 Reducing depressive thoughts even more - Basic Version . . . 24
4.3.2 Improving overall user experience - Basic Version . . . . 25
4.3.3 Reducing depressive thoughts more - Gamified Version . . . . 26
4.3.4 Improving overall user experience - Gamified Version . . . . . 26
4.4 Significance Testing . . . . 28
4.5 Summary . . . . 29 v
5 Discussion 31 5.1 SUS Test . . . . 31 5.2 Depressive Thoughts . . . . 32 5.3 Limitations . . . . 33
6 Conclusions and Future Work 35
6.1 Future Work . . . . 36
vi
List of Figures
3.1 Main menu of basic version. . . . 12
3.2 Adding an activity in the basic version. . . . 13
3.3 Schedule in the basic version. . . . 13
3.4 Main menu of the gamified version. . . . 14
3.5 Adding an activity in the gamified version. . . . 14
3.6 Showing the experience bar and the in-game avatar in the the gamified version. . . . . 15
3.7 Shows the "activity complete"-screen in the gamified version. . . . 15
4.1 Comparison of overall SUS score of both the basic and gamified version 22 4.2 Comparison of detailed SUS score of both the basic and gamified version 23 4.3 Difference of detailed SUS score between the basic and gamified version 23 4.4 Comparison of average score in the question about depressive thoughts of both the basic and gamified version . . . . 24
4.5 Comparison of average score in the question about reducing depressive thoughts even more of both the basic and gamified version . . . . 25
4.6 Detailed scores by individual answers in regards to reduction of de- pressive thoughts in basic version . . . . 27
4.7 Detailed scores by individual answers in regards to if depressive thoughts could be reduced even more for basic version . . . . 27
4.8 Detailed scores by individual answers in regards to reduction of de- pressive thoughts in gamified version . . . . 28
4.9 Detailed scores by individual answers in regards to if depressive thoughts could be reduced even more for gamified version . . . . 28
vii
Chapter 1
Introduction
Gamification is a concept where video game elements are used in various applications to increase enjoyment or engagement in otherwise mundane or boring tasks. It differs from games in the sense that there are no play elements in gamification, only game related elements. Gamification can be very beneficial in areas concerning health and education, where engagement might be low otherwise. The term has been on the rise, and recently more companies than ever has sought to adopt it into their applications, programs or solutions.[5] One specific application for gamification in recent years is using it to improve mental health, more specifically depression. Various apps like Stop, Breathe and Think or Headspace seek to use gamification in combination with mindfulness and meditation to combat the user’s depression or depressive thoughts.
Depression is a mental disorder which is common, according to the World Health Organization (WHO), more than 264 million people suffer from it around the world.
Symptoms include sadness, disinterest and lack of pleasure in activities previously found interesting or rewarding. Common problems faced are lack of sleep and ap- petite, tiredness and concentration problems.[1] Another problem identified by the WHO is that mental health related problems are on the rise world wide, and de- pression is one of the most common causes of disability.[2][7]. In addition, according to the Swedish Folkhälsomyndigheten, two studies performed in Great Britain and in the Czech Republic found that during the COVID-19 pandemic, depression and depressive symptoms increased. This presents a promising opportunity to explore gamified apps that improve depressive symptoms in people with depressive thoughts.
With mental health issues on the rise, and mobile phones being used by most people, gamification for mental health and more specifically depressive thoughts can be favourably developed in mobile applications. This ensures that the solution is accessible and discreet. Using mobile applications as a solution is also an area that needs more exploration according to Ming Lau et al.[10] Further, Fleming et al.[8]
identify the need for more accessible solutions and include the statement in their conclusion that more trials with comparisons between game-based and non-game- based applications should be used. In addition, Sardi et al.[12] also state the need for more research on integrating gamification into mobile applications.
To summarize, the evident gap in this field of study is gamification in mobile applications, and solutions that are accessible to users. This presents goals in which mobile apps with gamification can be an overlapping solution.
The main problem this thesis will deal with is the use of gamification in mobile applications to reduce depressive thoughts in users. This represents a tangible gap in the research of this field of study, and should be able to address some of the
1
2 Chapter 1. Introduction overall limitations identified in the aforementioned works. The mobile app solution was chosen to solve the accessibility problem.
The problem in question is highly relevant to this field, as previously stated, mental health issues are on the rise world wide and mental health applications and user-friendly self care is a real, proven solution.[7] Integrating this in mobile apps and applying gamification should provide an option and supplement to more traditional care in the form of medication and therapy.
1.1 Aim and objectives
The aims of this thesis is to develop a mobile application with gamification elements to help reduce depressive thoughts in its users, and to measure the user experience of the application. The app will be developed with one version using gamification and one version without gamification. The two different versions will then be used to test the participants. The participants will answer an online survey in which they will outline their user experience with the two different app versions, and if they felt that the app helped to reduce their depressive thoughts. The app will be a prototype of a health application, and thus will only have a simple system of adding activities to a daily schedule. These activities will consist of a name, a start and end time and a user-inputted description. Lastly, the user data collected will then be used to answer the two research questions outlined in section 1.1.2.
1.1.1 Objectives and goals
The objectives and goals of this thesis are as follows:
• Study the field of human-computer interactions in addition to psychology, and assess how to develop a gamified application that could reduce depres- sive thoughts in people that use it.
• Plan and implement the mobile applications and its gamification elements, making sure that the gamification design principles are followed. The app will be implemented with the following functions: a menu, functions to start activities and a schedule to show your current activities, a point and level system, an in game avatar and a reward system.
• Construct an online questionnaire in accordance with the System Usability Scale(SUS), and add some other questions asking about the users’ depressive thoughts and how the app affected those.
• Distribute the application and questionnaire to test subjects and have them test the app versions and answer the questions about user experience and depressive thoughts.
• Analyze the data and use it to make a conclusion about how gamification affects user experience, and some qualitative conclusion should be able to be drawn about how mobile applications can affect depressive thoughts.
1.2. Background 3
1.1.2 Research questions
For this thesis, the following research questions will be used:
RQ 1: How can gamification elements be implemented in a mobile application that aims to reduce the users’ depressive thoughts?
RQ 2: What is the difference in the user experience in a mobile application ac- cording to the system usability scale when gamification is and is not used?
1.2 Background
As a concept, gamification has seen a widespread adoption throughout academia and the software industry in the last ten years.[6] It’s even seen rise in other areas as well, such as health applications and grocery store bonus discount programmes.[3]
Gamification can be defined as "Gamification is the use of game design elements in non-game contexts", as said by Deterding et al. Gamification exists to address problems with products or processes in which the users might not be motivated to participate or use the product. In these circumstances, gamification has found its use in boosting the user interaction or usage of said products or processes. Some examples include exercise applications where gamification exists to increase motivation in users.
When gamification is applied, the users might want to strive towards certain goals, or collect badges for accomplishing certain tasks. Sometimes the gamification even falls back on itself, since some activities are already games of sort, but can then be gamified again through the internet. For example, in the mobile health application
"Samsung Health" there exists a program where you are tasked with walking. When you walk, the application logs your steps and then compares it to other users around the world. You can even challenge other people in the application.
However, not all uses of gamification can relate themselves to actual games and sports. Another use today that is ever so common among Swedish grocery stores is the so called "bonus system". Several different grocery stores in Sweden use these programmes, which incentivice users to spend more. As every Swedish krona spent is then added as points in their bonus system, which in turn grants the user new discounts.
A relatively unexplored field of gamification is mental health and mobile applica- tions. As previously mentioned, a number of authors and works highlight the need for more studies conducted on gamification in mobile applications, and also the need for more accessible solution. These two statements fit very well together as mobiles are accessible by virtually everyone today, and as such a mobile application was seen as a good solution for this thesis.
MindBud, as the application is called, is a mobile application used to write down a daily schedule by its user. The schedule can then be viewed at any time, and cleared when the day is over. This application works by making the user write down daily activities and then feeling accomplishment when the activities are done. It is developed as a prototype to help reduce procrastination and depressive thoughts in the user. In this project, two version of the application were developed. Firstly the aforementioned application with merely the daily schedule. Secondly a new version of the application which contains several gamification elements such as: an in game
4 Chapter 1. Introduction avatar, a leveling system, in-game rewards and experience gain. The second version of the application checks for any completed activities, and then rewards the user with experience points and thus levels. When new levels are reached, the in game avatar changes color. This is the reward system.
In this thesis, the System Usability Scale(SUS) will be used to measure the overall user experience of the application. This scale is used to rate the user experience of software in a standardised way. The scale has ten questions that are presented to the users, and they rate the questions from one to five. When the answers are collected, the scores are calculated into a final score. Depending on the score, the software can be compared to other software. The score goes from 0 to 100, where a 68 is considered average.
1.2.1 Key Terminology
In terms of gamification design, there exists three main aspects: Mechanics, Dynam- ics and Aestethics. Mechanics relate to the way data is presented to the user when gamification is used, for example points, badges or achievements. Dynamics are the behaviours of said mechanics relating to the user’s actions in the app. For example choices and finishing tasks. Lastly aesthetics relate to the emotions evoked when the users interact with the system. Examples include challenges in the system or being in a community.[5].
Some additional key concepts are the game elements commonly applied in gam- ification. The concepts are: Feedback, Goals, Badges, Point System, Leaderboard and User Levels. Basten defines them as follows: Feedback is the instant notification that makes sure that users understand what is happening, whether it is success or failure. Goals are the desired goals that the user is supposed to challenge the user to complete tasks. Badges are optional rewards and goals that are not related to the main goals of the system. Point system is numerical score that is added when the user completes tasks. Leaderboards track and display the users tasks to motivate the user to complete them. User levels is another system to keep the user motivated, through display of the overall score of the user.
1.3 Scope
As the main focus of this project is to explore the benefits on user experience when using gamification, the health-aspects of the application will be made very simple.
The application will only be a prototype of a health application, and the main focus of the implementation will be on the gamification of the application. The application will be developed in the Unity Engine, and will be made from scratch. In this thesis, the overall difference between the applications, after gamification elements have been added is observed. However no observations can be made about individual game elements. In addition, the main aspect of this observation is the user experience with the two different applications, and secondly how they impact the depressive thoughts of the user.
Some limitation to be considered are mainly time and testing. Implementing mobile applications take time, and even with the Unity Engine to speed things up,
1.4. Outline 5 not too many gamification elements can be added nor can a very advanced application be made. As such the implementation was kept simple, and gamification elements were added with time in mind. Some notable elements that did not make it because of time constraints were leader boards and badges. These were deemed to be too time-consuming to implement. Secondly testing, this goes hand in hand with time and also the availability of test environments. Only one mobile device was available to test on, and there was limited time to test. As such some bugs might be expected.
1.4 Outline
This thesis begins with the Introduction, this contains the aim, objectives, goals, research questions, background, key terminology and scope of the thesis. Secondly is the chapter called Related Work. Following is the Method chapter, which outlines how this study was performed, how data was collected and how the experiment and implementations were designed. It also describes the limitations and how risks were managed. After that chapter, the Results and Analysis chapter follows. This contains the results from the survey and the scores of the System Usability Scale, it also has the analysis of the results. The part after that is a Discussion about the results and finally conclusions are made and future work is outlined in the last Conclusions and Future Work chapter.
Chapter 2
Related Work
2.1 Gamification in E-Health
Gamification as a concept has become more and more popular ever since its adoption around ten years ago. As such there exists a number of studies done on gamification, and even some done on the effects of gamification when applied to health or mental health. In a literature review carried out in 2017, the authors Lamyae Sardi et al.
stated that the healthcare domain has seen a quick increase in adoption of gamifica- tion. More specifically in terms of health self-monitoring and management.[12]
In the review, 46 studies were considered and investigated. The review shows that a majority of the papers selected had gamification and serious gaming related specifically to chronic disease rehabilitation, physical activity and mental health. In relation as to what gamification elements were used, the results of the review show that a vast majority of the studies used feedback/rewards in their gamification. Other common methods include a progress bar, social connections and challenges/quests.
The review also found some answers as to what the benefits of using gamified e- Health applications were. They state that "the gameplay afforded in these e-Health interventions may positively affect users’ emotional experiences and foster their sat- isfaction and self-esteem.", another important advantage of gamification is "that is was perceived to highly motivate users to change their health behaviours and stay engaged with the application". Finally they found another advantage in the form of entertainment and ease of use. They state that another advantage found when using gamification was that it promoted game play, entertainment, ease of use and intelligibility.[12]
However, it also states that only some evidence exists to suggest that gamification could be helpful in the field of digital health. It highlights the need for more valid empirical evidence in the area of gamification.[12]
2.2 Gamification of Meditation
In a study carried out in 2019 by authors Matthew T. Fish and Amelia D. Saul, the aim was to find if the use of a mindfulness mobile app to help the participants to meditate, could improve the users severity of depression. The paper states that roughly 35% of college students report depression as a significant concern. The study explores the validity of using alternative methods to treat depression to the more traditional methods of medication and psychotherapy. In the study, they question
7
8 Chapter 2. Related Work the validity given to traditional methods currently and seek to explore meditation as an alternative to these (often more expensive and less accessible) methods.[7]
The application used for the experiment is called Headspace, it is a gamified mindfulness-meditation application. According to the authors, it has been found to reduce stress, irritability, aggression, and occupational stress. While increasing focus, compassion, and well-being. The application provides the user with guided mindful- ness meditation sessions. The application in itself is designed to fulfill the eight core drives that form the Octalysis Framework, created in 2015 by gamification expert Chou. The framework consists of the core drives: meaning, accomplishment, empow- erment, social influence, ownership, avoidance and unpredictability. The study used two groups of participants, one using the app for two weeks, and the control group not using the app at all. The two groups answered a questionnaire on depression called PHQ-9 (Patient Health Questionnaire - 9) before and after the two week trial.
The questionnaire is comprised of nine questions pertaining to depressive symptoms, and if the person taking the test has a score above a threshold, the user can be said to have some degree of depression. The overall results of the study showed that the experimental group showed significant changes in the form of improvements in relation to the control group. [7]
2.3 Using Gamification in Medical Assessment
In a Master’s thesis carried out last year (2020), authors Fredrik Carlsson and Se- bastian Vusak explored the use of gamification in a computerized test battery called CoGNIT. The test is used for cognitive assessment in diagnosing and treating "pa- tients with idiopathic normal pressure hydrocephalus (INPH)". Symptoms of the disease include dementia among other things. The basis for the study is that the CoGNIT test is a very tedious task. Each test takes about 40 minutes to complete, and it has to be completed several times. Gamification was to be used in a new version of the test to increase the quality of the data gathered from the test. If a person taking the test is weary or tired towards the end of the test, the data may be affected negatively. [6]
In the study, the participants were asked to take both the non-gamified test and the gamified test, and then the results were compared. Because of the COVID- 19 pandemic, no real subjects with the disease could participate in the study, and healthy individuals were instead asked to partake.
As for the implementation of the gamification elements, the elements were picked specifically to not hinder or interfere with people having the symptoms of the dis- ease. The following symptoms are typical for INPH patiens: "memory impairment, slow ideation, attention deficits, apathy, depression and aggressiveness". The game- elements that did make it into the test were avatars and rewards as they were found to not interfere with the test subjects’ disorders. The results of the study show that the participants showed better test results in the gamified version when it came to all tests except for the tests regarding memory. In that test, a worse score was recorded in the gamified version. [6]
Finally however, the authors state that since they could not perform the test with real subjects which had symptoms of mild dementia, the applicability of the
2.4. Gamification in Software Engineering 9 conclusions drawn from the experience are affected. However, the conclusions and result are still applicable to healthy adults. [6]
2.4 Gamification in Software Engineering
In a systematic review carried out last year (2020), authors Daniel de Paula Porto et al. seek to characterize how gamification has been used in non-educational contexts of software engineering. The study examined 103 different sources published up to January 2020, and sought to answer a number of research questions, among others:
"How do software engineering activities benefit from gamification?" and "How is gamification inserted into software engineering activities?". The study also shows the overall increase of publications related to gamification, which sprung into popularity after 2011, when it was first mentioned in a work by Deterding et al. The number of publications on gamification peaked in 2018, with a small decrease in 2019. As the study was conducted in early 2020, only part of the data is collected for that year.[11]
The study found that the 3 most popular gamification elements used in the studies were: points, leaderboards and badges. With levels, rewards and challenges, social graphs, avatars, voting and betting also being used to some extent. The study found numerous benefits to using gamification in software engineering, some of them being engagement, motivation, encouragement but also improvements in the quality of work performed and better team integration.[11]
Lastly, the authors state some questions in regards to future work in the subject.
They ask if other elements than the ones stated could be used. Additionally they ask if the ones that were not as common could be better explored. Finally they ask if the gamification elements are not used because they are harder to implement.[11]
Chapter 3
Method
3.1 Methodology
In this thesis, a mobile application will be developed as a prototype of a real e- health application. The purpose of the application will be to help reduce depressive thoughts in its users. This will be done by having the user make a daily schedule to plan their activities. Having a daily schedule has been shown to help reduce stress and help people improve their mental health.[9] However it should be stated that daily routines will differ in their impact on mental health.[9]
As the primary purpose in this study is to assess the effect of gamification on the user experience in mobile health applications, the application will be developed in two versions. One version will only contain the main basic features. This includes a menu system with a help panel to guide users as to how to use the application, the primary function of the application: the activity and schedule system. This allows the user to input activities with a chosen name, description and start and end times. The users can then view their activities in a schedule panel. Lastly users can also clear the activities at the end of the day. The other version of the application will contain all of the basic features of the first version, however it will also contain some added gamification elements. In addition to evaluating the main aspects of gamification design, the added elements were chosen due to their regular usage in gamification, and also with time constraints in mind. The chosen gamification elements were: an experience and level system, an in game avatar and lastly a reward system.
With this design process, both versions of the application were implemented.
The chosen research methods were experiment and questionnaire. The experiment consisted of participants testing both version of the application, and data collection was done by having the participants answering a survey after they had tested the applications. As such, both the experiment and the questionnaire were used to- gether to answer both RQ1 and RQ2. However, RQ1 was mostly answered by the experiment, with the addition of some questions in the survey about what different elements or features could have been added to the application. The majority of the questionnaire asked the users about the overall user experience however, and there the System Usability Scale (SUS) was used.
For the study, a small literature review was done mainly to research what game elements are usually used for gamification. The literature review showed that some of the most common elements used are: feedback/rewards, a progress bar, a social connection, challenges/quests, badges and point systems. As previously mentioned, some of these were excluded in this study because of the scope. The final game
11
12 Chapter 3. Method elements chosen were, an in-game avatar, an experience bar, a level system and rewards. These can be seen in Fig. 3.5, 3.6 and 3.7 respectively. The rewards in the application is that when the user levels up, the bird changes colour.
3.2 Implementation
The implementation of the application will be done in two phases, the first phase entails just the basic application with the menu, schedule and ability to make, plan and complete activities. This will be done on a basic level to keep the implementation simple. The simplicity is to make sure that the implementation can be completed in the time frame available. Another measure taken to ensure that the time plan is upheld is the use of the Unity Engine to implement the application. Using the Unity Engine will remove the need to implement the application from scratch, and thus speed up the development.
In the second phase of development, gamification elements will be added to the application. These include a player level system, in addition to a point system (in which you can gain levels), a reward system, which will reward the user beyond just the basic points, and lastly a player-avatar.
Figure 3.1: Main menu of basic version.
Design
In terms of design, the three main aspects of gamification design will be considered.
The point and level system is related to the mechanics aspect, as it will show the overall user progress in the app. This is used to further motivate the user to continue
3.2. Implementation 13
Figure 3.2: Adding an activity in the basic version.
Figure 3.3: Schedule in the basic version.
its progress and thus continue doing its day to day activities, reducing procrastination and thus reducing the depressive thoughts. The second aspect, dynamics is the way in which the user can gain these points and levels, which will be through the planning
14 Chapter 3. Method
Figure 3.4: Main menu of the gamified version.
Figure 3.5: Adding an activity in the gamified version.
and completion of custom made tasks in the application. The customization will be in terms of what the user is supposed to do to complete the task and how long it takes. This means that the user can specify what should be done for each task, and
3.2. Implementation 15
Figure 3.6: Showing the experience bar and the in-game avatar in the the gamified version.
Figure 3.7: Shows the "activity complete"-screen in the gamified version.
thus customize their own experience. The final aspect, aesthetics relates to the users’
emotion. This will be invoked through the choice of colors and through the reward
16 Chapter 3. Method system and additionally the player avatar. The reward system will exist to invoke feelings of joy and achievement when completing certain milestones, for example using the app two days in a row, or completing all the tasks planned for an entire day. The player avatar will be designed to feel welcoming and relaxing. This will be done by using a cartoon-style design and through using the avatar to convey happy and relaxed emotions.[5]
3.3 Experiment
As previously stated, the experiment will be performed through an online survey.
The test subjects will answer the survey, the survey will ask about both versions of the app. The online survey will consist of the SUS-questions in addition to some questions regarding if the users thought the app helped reduced their depressive thoughts. The SUS-questions will be used to answer research question two, and the additional questions will be used to answer question one. The planned number of test subjects are 15-20, though if a number higher than this is interested in trying the app, that does not make any difference. The test subjects should all be over the age of 18 to participate in the study. The planned number of participants is because of the believed difficulty to find people who will enter the study. The 18 years of age requirement is in accordance with ethics recommendations.
When using the SUS, the users are asked to score the following ten items with one of five responses that range from Strongly Agree to Strongly Disagree.
1. I think that I would like to use this system frequently.
2. I found the system unnecessarily complex.
3. I thought the system was easy to use.
4. I think that I would need the support of a technical person to be able to use this system.
5. I found the various functions in this system were well integrated.
6. I thought there was too much inconsistency in this system.
7. I would imagine that most people would learn to use this system very quickly.
8. I found the system very cumbersome to use.
9. I felt very confident using the system.
10. I needed to learn a lot of things before I could get going with this system.
The System Usability Scale (SUS) was chosen because of its ease of use, reliable results even when used on small sample sizes and its validity. When the tests have been administered and scored by the users, the final score of 0-40 will be converted to a number between 0-100, according to usability.gov, a score of 68 would be considered average, and anything below that is considered below average. [4]
3.4. Other Alternatives 17 In addition to the SUS-questions, the users’ will be asked to score some statements about the effect of the application on their depressive thoughts, and what they think the application could use to further the effects of the gamification elements presently found in the application. These statements will also be scored from Strongly Agree to Strongly Disagree. Lastly, an open text question will ask the users if they have any additional features they think would help them reduce depressive thoughts, or improve the overall user experience of the application. The statements and questions include:
1. I think that the application helped me reduce my depressive thoughts.
2. If the application was further developed with more features I think it could help me reduce depressive thoughts even more.
3. To better help me reduce my depressive thoughts, the application should add...
4. To improve the user experience of the application, the application should add...
3.4 Other Alternatives
In the beginning of this project, a different version of the experiment was discussed.
In the first version, one experimental group and one control group would be used to test the applications. The experimental group would test the gamified version of the application, and the control group would test the non-gamified basic version. The two results from the different groups would then be compared, side by side.
However, it was deemed to great a risk to not be able to get enough people for both an experimental group and a control group, and thus the experiment was changed into its current version. Had the study been greater in scope and time, perhaps a control group could have been used. It should be said however that the final version of the experiment was not deemed to be worse than the first proposition.
The final version was chosen to make the results as valid as possible, through greater numbers of testers of both version of the application.
3.5 Data Collection
The data collection of this study was done through an online survey taken after the participants had finished testing the second application. After all of the answers had been collected, the scores of the SUS-test could be used to calculate the SUS-score of the system. The score determines the overall grade of the system. The SUS-score has both a letter grade that goes from "A" to "F", and additionally an adjective rating which goes from "Excellent" to "Awful".[13]
The score is calculated from the answers the participants entered in the survey.
The test has ten different statements which have to be scored from one to five.
Where one means Strongly Disagree and five means Strongly Agree. The statements can be either positive or negative, and all of the odd-numbered questions are positive statements while all of the even-numbered questions are negative statements. Once
18 Chapter 3. Method the results are in, the scores can be calculated. This is done by taking the user score from the odd statements and subtracting one from each of them. This will convert them from one to five to zero to four. With the even statements, you take the score and subtract it from five. This will convert all of the answers into the zero to four format. Once this is done, you add up all of the scores from all participants and then multiply the final score by 2,5. This final calculation converts the score from 0-40 to 0-100. The score then has to be normalized for the number of participants (which in this study was 16.[13]
After the SUS-questions, the participants were asked to answer four additional questions for both version of the applications. These answers were in relation to RQ1, and will be presented separately to the SUS-score, which will answer RQ2.
3.6 Working Conditions
Because of the COVID-19 pandemic, all work has been done from home. Meetings with the supervisor have been virtual, and there has been no physical contact be- tween author, supervisor and participants. All communication between author and participants has also been remotely, and there has been no supervision during the experiment except for contact if participants had questions about the experiment.
3.7 Ethical, Societal and Sustainability Aspects
In regards to ethical questions, the user-study is planned to observe if users feel like gamification improved their experience of a mobile app and if said mobile application reduced their depressive thoughts. As such, care should be taken to not adversely effect the users. It should also be stated that it does not in any way, shape or form attempt to treat any medical health issues. The application will only ever be experimental, and proper care should be sought if needed. This was addressed in the information letter provided together with the access to the applications.
As previously mentioned, the study contains two parts, firstly the development of a prototype app to reduce depressive thoughts in the users. Secondly, the collection of data from users using an online-questionnaire. Because the questionnaire will be online, there will be no contact with the applicants that are testing the app.
Both the app and the questionnaire will be delivered through the internet (e-mail or download).
The collected data will be stored securely, and will only be able to be accessed by the author and the supervisor, and no data will be linked to. No individual data will be accessed or analyzed, only the collection of data as a whole, and the questionnaire will be anonymous, to ensure that no data can connected to any individual user. In terms of society, as mentioned earlier, depression, depressive thoughts and mental health problems are rising worldwide, and depression has seen some rise during the COVID-19 pandemic. As such the research of solutions to these problems could be of great use to society, to help individuals improve their symptoms.
As previously stated, globally mental health issues are on the rise, and thus it seems that the need for both more options regarding care and more accessible care
3.7. Ethical, Societal and Sustainability Aspects 19 is obvious. Thus research and development in this topic should be of great interest to society.
Chapter 4
Results and Analysis
In the following section, the results of this thesis will be presented, visualized and lastly used to answer both of the research questions asked.
4.1 SUS Score
In this study, a total of 16 people took part in the experiment. All of the participants completed the entire study and answered all of the survey questions. However two of the participants contacted the author about mistakes they made during the test, and to correct the scores, the anonymity of their results had to be waived to correct their mistakes. One of the mistakes was one of the SUS-questions where a word’s meaning was mistaken for the opposite and thus the scoring of the question was reversed.
Another reported mistake that was corrected was one participant who reversed the answers in two of the free-text questions. The participant thought the question was about the gamified version of the application, but it was about the basic version.
These answers have been reversed in the data, to make them correct.
For both the basic version and the gamified version, the SUS score had to be calculated from the raw data in accordance with the SUS guidelines. The raw data is represented as a score of one to five for each of the ten statements in the SUS questionnaire, where a one means Strongly Disagree, a three means Indifferent and a five means Strongly Agree. After the scores were collected, they were either taken and subtracted by one if they were one of the odd-numbered statements, or if they were one of the even-numbered statements the score was subtracted from five instead.
After this is done, the scores now correctly represent the statements in a positive manner, even if the statement itself is negative. For example, the second statement reads "I found the system unnecessarily complex." which is a negative statement.
Because the scores are recalculated, a higher score on this statement means the system wasn’t considered very complex. The odd-numbered questions are simply converted to a score between zero and four instead of one to five.
After the scores have been converted to one through fours, they are all added together for all participants. After the scores are summarized, they are multiplied with 2,5. This is done to convert the score from 0-40 to 0-100 instead. Of course, in this study there was 16 participants, and as such the score had to be normalized to represent a percentage between 0 and 100, this was done by dividing the score by 1,6. Lastly, after the score from each statement had been calculated and normalized, the scores were added together and divided by the number of questions to find the overall SUS score of both the basic and gamified version. The overall SUS score was
21
22 Chapter 4. Results and Analysis
Figure 4.1: Comparison of overall SUS score of both the basic and gamified version
78,7 for the basic version, and 82.1 in the gamified version as shown in Fig 4.1, this is a difference of +3,4.
In figure 4.2, the scores of each individual test statements are shown. The test shows improvement in the gamified version in statements one, two, nine and ten, while having the same scores in statement three, four, five and six. Lastly the figure shows a decrease in score in statement seven and eight.
Statements where the gamified version had improved test results relate to usage, complexity, confidence and difficulty to learn the system. Meanwhile, statements where the two version were identical include how easy it was to use the system, if the person needing it would need help from a technical person, integration of the functions in the system and lastly inconsistency. Finally, the statements where the basic version scored higher than the gamified version are, if a person imagines other people would learn the system quickly and if the system was cumbersome to use.
Another observation that can be made is that while the basic version did outscore the gamified version in some areas, it is only by a small margin, the largest point difference in favor of the basic version is 4,7 units of score. The gamified version however wins out by quite big margins in some of the areas where it is scored higher.
This is made more clear by Fig 4.3.
4.2 Depressive Thoughts
In the following section, the scores for the questions about depressive thoughts will be shown. The participants first had to score the following two statements for each of the app versions: think that the application helped me reduce my depressive thoughts.
4.2. Depressive Thoughts 23
Figure 4.2: Comparison of detailed SUS score of both the basic and gamified version
Figure 4.3: Difference of detailed SUS score between the basic and gamified version
24 Chapter 4. Results and Analysis
Figure 4.4: Comparison of average score in the question about depressive thoughts of both the basic and gamified version
and If the application was further developed with more features I think it could help me reduce depressive thoughts even more.. These statements were rated one to five with the same scoring system as previously. As can be seen in Fig 4.4, users thought that the gamified version would reduce their depressive thoughts substantially more than the basic version. The gamified version also scored higher in regards to making additions to reduce depressive thoughts even more. This is shown in Fig 4.5. Both figures show the average scores of respective statements.
4.3 Additional Features
Lastly, the participants could suggest additions to the two version of the application.
First in terms of reducing depressive thoughts, and secondly to improve the overall user experience. If the users did not feel like they had anything to add, they were instructed to write nothing. Out of the people who wrote something to reduce depressive thoughts even more in the basic version of the application, they mostly suggested additional features closely linked to motivation.
4.3.1 Reducing depressive thoughts even more - Basic Version
1. Things to look forward to.
2. Further motivations to make me want to complete activities.
3. A rating system for how the activity went.
4.3. Additional Features 25
Figure 4.5: Comparison of average score in the question about reducing depressive thoughts even more of both the basic and gamified version
4. Something to further motivate me to use the application.
5. Rewards
6. Suggestions for activities to do.
4.3.2 Improving overall user experience - Basic Version
In regards to improvements to the overall user experience, participants were more inclined to mention bugs or annoyances. Some mentioned new features as well. Some of the suggestions include:
1. Changing the clock (the user found it messy).
2. Mention of bugs when adding activities.
3. Changing the system for adding activities, needing to click confirm and save is cumbersome.
4. Adding option to plan different days than the current.
5. Alarm function to help remember activities.
26 Chapter 4. Results and Analysis
4.3.3 Reducing depressive thoughts more - Gamified Version
As for the gamified version, the answers pertaining to reducing depressive thoughts even more had more answers that were "Nothing" or "Don’t know". The answers that did suggest changes were very similar to the basic version, and some even said that the same things they suggested in the basic version should be done for gamified version as well. The answers that wanted the same changes, wanted the clock changed because it felt clunky to use, another response that a user felt should be added to both versions was an option to write down your current mood. Another user also wanted things to look forward to, and a "best day of the week" for both versions of the application.
For the gamified version to further reduce depressive thoughts, users suggested the following additions or changes:
1. Adding a friend system, where you could add other users as friends. This would help by enabling group activities.
2. Having a more personalized experience, such as adding a username or choosing your own avatar
3. Choosing which animal the player avatar should be.
4. Animations for the bird.
4.3.4 Improving overall user experience - Gamified Version
As for improving the overall user experience, users suggested the following:
1. Planning your days ahead of time, option to be reminded of activities.
2. Adding different categories to give more or less points to different types of activities. This could be done by asking some personal questions when opening the application for the first time.
3. Improving the user interface and smoothness of the application.
4. More animations to bring "life" to the application.
5. An overview of the rewards, maybe a leader-board.
Lastly we can observe the details of the scores for the depressive thoughts, as observing the details can be interesting later when discussing the results. In Fig 4.6 and 4.7 we see the detailed scores for the basic version of the application.
We can observe that in the first question, half of the users felt indifferent towards reduction in depressive thoughts, with only three people feeling that the application could help them reduce depressive thoughts. As for the second question, a majority of the users answered some form of agreement that the application could be made to help them reduce depressive thoughts even more.
As for the gamified version, in the first question we see a shift from strongly disagree to disagree (a one to four and five), while the people who felt indifferent
4.3. Additional Features 27
Figure 4.6: Detailed scores by individual answers in regards to reduction of depressive thoughts in basic version
Figure 4.7: Detailed scores by individual answers in regards to if depressive thoughts could be reduced even more for basic version
remained the same. In the second question, we see that seven people answered strongly agree, while the rest either felt indifferent or agreed.
28 Chapter 4. Results and Analysis
Figure 4.8: Detailed scores by individual answers in regards to reduction of depressive thoughts in gamified version
Figure 4.9: Detailed scores by individual answers in regards to if depressive thoughts could be reduced even more for gamified version
4.4 Significance Testing
In this study, a simple T test was performed to calculate the P-value. After perform- ing significance testing on the results of the SUS survey, the P-value was found to
4.5. Summary 29 be 0.3662. This is not considered to be statistically significant. This could perhaps be explained by the low number of participants, and is a limitation of this study.
However, arguments will still be made that it is significant, because of the clear in- crease in score. As can be seen in Fig. 4.1. To summarize, it is significant, just not statistically significant.
4.5 Summary
This section will provide a summary of the chapter and try to answer the research questions with the results from the SUS-questionnaire and the additional questions posed by the experiment. In the following chapters after this one, the results will be used for discussion and conclusion.
The following research questions were asked in this thesis:
RQ1: How can gamification elements be implemented in a mobile application that aims to reduce the users’ depressive thoughts?
RQ2: What is the difference in the user experience in a mobile application when gamification is and is not used?
RQ1: To answer the first research question, we can observe both the chosen gam- ification elements in this study, and the overall impact that they had in reducing the users depressive thoughts. We can also observe what new features the participants suggested, to further the elements that could have been used as well. The following gamification elements were used in the application: a user level and experience sys- tem, a reward system and an in game avatar. If we look at Fig. 4.4, we can clearly see an improvement in the gamified version in regards to reducing depressive thoughts.
However, we can also observe Fig. 4.5 and see that even though participants felt like the gamified version would reduce their symptoms more, they also felt that it could reduce them even more if additions were made. They even felt that it could be improved more than the basic version.
In regards to what additions can be made, the users suggested some various functions and improvements. Many of the suggestions in the basic version can be heavily related to gamification, and some of them are even outright gamification elements. In the gamified version we saw even more advanced gamification elements suggested, such as adding social aspects and more customization.
RQ2: When answering the second research question, we can directly observe the test results from the SUS questionnaire. In Fig. 4.1 we can clearly see that the gamified version scored slightly better overall in the SUS scoring system. If we look at Fig 4.2 we can observe the individual scores in the different statements.
In the figure we can see that the gamified version improved in statements about inclination to use the system frequently, complexity in the system, confidence when using the system and overall difficulty of learning the system. However, the system also scored slightly worse in some statements, these being about people learning the system quickly and the system being cumbersome to use. To show the differences even clearer, another figure was produced which only shows the difference in scores between the two versions, this can be seen in Fig. 4.3. When we observe this figure, we can quickly deduct that the gamified version had larger improvements than it had deterioration. The largest improvement was in the first statement which is directly
30 Chapter 4. Results and Analysis related to motivation to use the system.
A small part of the users experienced bugs in the gamified version, however these do not seem to have been widespread enough to affect the result in any major way.
