Evaluating the user experience in mobile games using session recording tools

Department of Science and Technology Institutionen för teknik och naturvetenskap

LIU-ITN-TEK-A--15/034--SE

Utvärdering av

användarupplevelsen av

mobilspel med hjälp av

sessionsinspelningsverktyg

Veronica Börjesson

Karolin Jonsson

2015-06-12

LIU-ITN-TEK-A--15/034--SE

Utvärdering av

användarupplevelsen av

mobilspel med hjälp av

sessionsinspelningsverktyg

Examensarbete utfört i Medieteknik

vid Tekniska högskolan vid

Linköpings universitet

Veronica Börjesson

Karolin Jonsson

Handledare Camilla Forsell

Examinator Katerina Vrotsou

Upphovsrätt

Detta dokument hålls tillgängligt på Internet – eller dess framtida ersättare –

under en längre tid från publiceringsdatum under förutsättning att inga

extra-ordinära omständigheter uppstår.

Tillgång till dokumentet innebär tillstånd för var och en att läsa, ladda ner,

skriva ut enstaka kopior för enskilt bruk och att använda det oförändrat för

ickekommersiell forskning och för undervisning. Överföring av upphovsrätten

vid en senare tidpunkt kan inte upphäva detta tillstånd. All annan användning av

dokumentet kräver upphovsmannens medgivande. För att garantera äktheten,

säkerheten och tillgängligheten finns det lösningar av teknisk och administrativ

art.

Upphovsmannens ideella rätt innefattar rätt att bli nämnd som upphovsman i

den omfattning som god sed kräver vid användning av dokumentet på ovan

beskrivna sätt samt skydd mot att dokumentet ändras eller presenteras i sådan

form eller i sådant sammanhang som är kränkande för upphovsmannens litterära

eller konstnärliga anseende eller egenart.

För ytterligare information om Linköping University Electronic Press se

förlagets hemsida

Copyright

The publishers will keep this document online on the Internet - or its possible

replacement - for a considerable time from the date of publication barring

exceptional circumstances.

The online availability of the document implies a permanent permission for

anyone to read, to download, to print out single copies for your own use and to

use it unchanged for any non-commercial research and educational purpose.

Subsequent transfers of copyright cannot revoke this permission. All other uses

of the document are conditional on the consent of the copyright owner. The

publisher has taken technical and administrative measures to assure authenticity,

security and accessibility.

According to intellectual property law the author has the right to be

mentioned when his/her work is accessed as described above and to be protected

against infringement.

For additional information about the Linköping University Electronic Press

and its procedures for publication and for assurance of document integrity,

Evaluating the user experience in mobile

games using session recording tools

A thesis presented for the degree of Master of Science

in Media Technology and Engineering

Link¨oping University, Sweden

Veronica B¨

orjesson

Karolin Jonsson

Supervisor: Camilla Forsell

Examiner: Katerina Vrotsou

Abstract

This thesis work examines how the user experience of mobile games can be evaluated with the use of session recording tools. The aim is to produce a workflow for user testing with session recording tools for mobile devices. In order to evaluate the tools and services, and to develop the workflow, several user tests have been conducted.

When using mobile session recording tools, it is possible to record the screen of the device and the microphone input while the user is playing the game. In some tools it is also possible to record the input from the front camera of the device, making it possible to capture the user’s facial expressions and reactions during the test session. Recording the test session makes it easier to understand and evaluate the player experience of the game, and also to identify issues such as difficulties with the navigation in the application or annoyance due to non intuitive interaction patterns. It is also a good way to get feedback about what the user likes and dislikes in the application. The fact that no additional equipment is needed for recording the test session, and that the user can perform the test comfortably on their own device in their own home, increases the chances for the test itself to have a minimal impact on the user experience, since the user can complete the test in their natural environment. Session recording tools are appropriate when conducting remote user testing since the users and the user experience researcher do not have to be at the same location. It is also a flexible approach since the testing does not have to be carried out in real-time. The test users can perform the test when they have time and even simultaneously, while the user experience researcher can watch and analyse the recordings afterwards. When conducting user testing with session recording tools, there are also other parts necessary besides the actual tool. The test has to be set up (instructions, tasks, questions etc.) and both the test and the game application containing the integrated session recording tool need to be distributed to the test user in some way. The test users need to be recruited from somewhere, and they have to match the desired target group for the test session. There are test services which provide all this; test set up, recruitment of test users, distribution of test and game application, and also some which even provide analysis of the recordings. When not using a test service, the test facilitator needs to take care of recruitment of test participants, test set up, distribution and analysis of test data by him or herself. During this study, methods for conducting user testing using session recording tools both with and without test services have been tested and evaluated. The mobile game Ruzzle, developed by MAG Interactive, has been used as test object. This thesis also covers how the user experience in mobile games differs from other software, and it also investigates how the user experience can be analysed from the session recordings, i.e. how the user’s emotions can be read from the recorded screen, voice and face. As a part of the thesis work, a testing workflow has been developed for the commissioning company MAG Interactive. It contains guidelines for how to conduct user testing with session recording tools and which parts that are necessary in order to carry out a successful test process. Tables with information regarding the tools and test services are also presented, in order to facilitate the decision on which tool/service that is most suitable for the specific test objective.

Key words:

Sammanfattning

Detta examensarbete unders¨oker hur anv¨andarupplevelsen i mobilspel kan utv¨arderas genom anv¨andning av sessionsinspelningsverktyg. Syftet ¨ar att producera riktlinjer, i form av ett arbetsfl¨ode (p˚a engelska ¨aven kallat workflow), f¨or hur anv¨andartester med sessionsinspel-ningsverktyg f¨or mobila enheter kan genomf¨oras. F¨or att kunna utv¨ardera verktyg och testtj¨anster, samt f¨orv¨arva de kunskaper och den erfarenhet som kr¨avs f¨or att kunna utveckla detta ar-betsfl¨ode, har flertalet anv¨andartester genomf¨orts.

Mobila sessionsinspelningsverktyg m¨ojligg¨or inspelning av mobilenhetens sk¨arm medan anv¨andaren spelar mobilspelet. Vissa verktyg har ¨aven inspelning av mobilenhetens frontkam-era vilket g¨or det m¨ojligt att spela in anv¨andarens ansiktsuttryck under testsessionen. Detta underl¨attar vid f¨ors¨ok att f¨orst˚a anv¨andarupplevelsen och vid utv¨ardering av spelupplevelsen, samt vid identifiering av problem som till exempel sv˚arigheter att navigera i applikationen eller irritation ¨over icke intuitiva interaktionsm¨onster. Det ¨ar ¨aven ett bra s¨att att f˚a ˚aterkoppling om vad anv¨andaren gillar och ogillar med applikationen.

Det faktum att ingen extra utrustning beh¨ovs f¨or att kunna spela in testsessionen, samt att anv¨andaren kan utf¨ora testet p˚a en enhet som de k¨anner sig bekv¨am med, i sitt eget hem, ¨okar chanserna f¨or att testet skall ha en minimal inverkan p˚a anv¨andarupplevelsen, efter-som anv¨andaren kan utf¨ora testet i sin vardagsmilj¨o. Sessionsinspelningsverktyg passar bra vid utf¨orandet av anv¨andartester p˚a distans eftersom anv¨andare och testanordnare ej beh¨over befinna sig p˚a samma geografiska plats. Det ¨ar ¨aven ett flexibelt tillv¨agag˚angss¨att d˚a testerna ej beh¨over utf¨oras i realtid. Testanv¨andarna kan utf¨ora testet n¨ar de har tid, till och med samtidigt, medan testanordnaren kan analysera inspelningarna efter˚at. Vid genomf¨orandet av anv¨andartester med sessionsinspelningsverktyg, finns ¨aven andra essentiella delar f¨orutom sj¨alva verktyget. Det m˚aste finnas en metod f¨or att skapa ett test (inneh˚allandes bl.a. testinstruk-tioner, uppgifter och fr˚agor) och m¨ojlighet att distribuera testet till anv¨andarna. Anv¨andarna skall ¨aven rekryteras n˚agonstans ifr˚an, och de m˚aste ¨overensst¨amma med den ¨onskade m˚algruppen f¨or testsessionen. Dessutom beh¨over applikationen, med det integrerade verktyget, distribueras till anv¨andarna s˚a att de kan ladda ned och installera applikationen f¨or att kunna delta i testet. Det finns testtj¨anster som erbjuder b˚ade skapande av test, rekrytering av testanv¨andare, distribuering av test och applikation, och vissa erbjuder ¨aven analys av sessionsinspelning. D˚a ingen testtj¨anst anv¨ands ¨ar det upp till testanordnaren sj¨alv att rekrytera testanv¨andare, skapa testet, distribuera test och applikation, samt analysera inspelningar. I denna studie har metoder f¨or att genomf¨ora anv¨andartester med sessionsinspelningsverktyg, b˚ade med och utan testtj¨anst, testats och utv¨arderats. Mobilspelsapplikationen Ruzzle, utvecklad av MAG Interac-tive, har anv¨ants som testobjekt. Detta examensarbete omfattar ¨aven hur anv¨andarupplevelsen i mobilspel skiljer sig fr˚an anv¨andarupplevelsen i vanlig mjukvara och det unders¨oks ¨aven hur anv¨andarupplevelsen kan analyseras fr˚an sessionsinspelningar, d.v.s. hur anv¨andarens k¨anslor kan utl¨asas fr˚an inspelad sk¨arm, r¨ost och ansikte. Som en del i examensarbetet har ett ar-betsfl¨ode f¨or testning producerats f¨or f¨oretaget MAG Interactive. Detta inneh˚aller riktlinjer f¨or genomf¨orandet av anv¨andartester med sessionsinspelningsverktyg och vilka delar som ¨ar n¨odv¨andiga f¨or att genomf¨ora en framg˚angsrik testprocess. Tabeller med information g¨allande olika verktyg och testtj¨anster presenteras f¨or att underl¨atta valet av sessionsinspelningsverk-tyg/testtj¨anst.

Nyckelord:

Acknowledgements

We would like to thank our families and friends for their love and support, through all of our lives which has led up to this moment. A big thank you to our supervisor Camilla who has been an inspiration and given us lots of good feedback and advice. We would also like to thank all of the participating companies who has taken the time to answer all of our questions and letting us test their tools and services.

Finally, a big thank you to MAG Interactive, and especially to the Ruzzle team, for giving us this opportunity and making us a part of the team. We have learned so much and gained valuable experiences that we will carry with us for the rest of our lives. It has been super fun and we are so glad for getting the opportunity to get to know all of the amazing people at MAG Interactive, it has truly been a pleasure.

Contents

1.4 The Test Object: Ruzzle . . . 2

1.5 Disposition . . . 3

1.6 Limitations . . . 3

2 Theory 4 2.1 User Experience . . . 4

2.2 Digital Games . . . 5

2.2.1 Relevant Genres of Digital Games . . . 6

2.2.1.1 Social Games . . . 7

2.2.1.2 Casual Games . . . 7

2.2.1.3 Mobile Games . . . 7

2.3 User Testing . . . 8

2.3.1 Different Testing Methods . . . 8

2.3.2 Testing Methods . . . 9

2.3.3 Remote User Testing . . . 10

2.3.4 Post Test Questionnaire . . . 11

2.3.5 Testing of Digital Games . . . 12

2.3.6 Testing of Mobile Games . . . 14

2.3.7 Test Users . . . 14

2.4 Session Recording Tool . . . 15

2.4.1 Metrics . . . 16

2.4.2 Facial Reactions . . . 17

2.4.3 Audio Recordings . . . 17

2.5 Workflows for User Testing . . . 18

2.5.1 Remote Usability Testing . . . 18

2.5.2 Mobile Applications . . . 19

2.5.3 Mobile Games . . . 19

2.5.4 Session Recording Tools . . . 21

3 Approach 22 3.1 Production of Testing Workflow . . . 22

3.2 Materials . . . 22

3.3 Research . . . 22

3.4 Evaluation of Session Recording Tools . . . 23

3.5 Integration . . . 23

3.6 Finding Test Users . . . 23

3.7 Creating the Test Plan . . . 24

3.8 Distribution . . . 24

3.9 Execution of User Tests . . . 24

3.10 Analysis of Session Recordings . . . 25

3.12 Final Evaluation of Session Recording Tools and Test Services . . . 26

4 Results 27 4.1 Test Services and Session Recording Tools Initially Investigated . . . 27

4.2 Tested Session Recording Tools . . . 30

4.2.1 Lookback . . . 31

4.2.2 UXCam . . . 33

4.2.3 PlaytestCloud . . . 34

4.2.4 UserTesting . . . 36

4.3 Distribution and Test Set Up Services . . . 37

4.3.1 Beta Family . . . 38

4.3.2 PlaytestCloud . . . 41

4.3.3 UserTesting . . . 43

4.3.4 Distribution Without Test Set Up Service . . . 45

4.4 Comparison of Test Services and Session Recording Tools . . . 46

4.5 Outcome of Test Session Analysis . . . 46

4.5.1 Questionnaire Results . . . 47

4.5.2 Insights Gained from Screen, Facial and Voice Recordings . . . 51

4.5.3 The Test Object: Ruzzle . . . 51

4.6 Resulting Workflow . . . 51

4.6.1 Test Plan . . . 51

4.6.2 Test Objective . . . 52

4.6.3 Test Users . . . 52

4.6.4 Tool and Test Service . . . 52

4.6.4.1 Session Recording Tool . . . 53

4.6.4.2 Distribution and Test Set Up . . . 54

4.6.5 Time Plan . . . 56

4.6.6 Prepare Test Details . . . 57

4.6.6.1 Preparations . . . 57

4.6.6.2 Introduction . . . 57

4.6.6.3 Instructions . . . 58

4.6.6.4 Screener . . . 58

4.6.6.5 Pre Gameplay Questionnaire . . . 58

4.6.6.6 Tasks . . . 59

4.6.6.7 Post Gameplay Questionnaire . . . 59

4.6.7 Perform Test . . . 59

4.6.7.1 Pilot Test . . . 60

4.6.7.2 Actual User Test . . . 60

4.6.8 Analysis . . . 60

4.6.9 Summarise Results and Share with the Team . . . 61

5 Discussion 62 5.1 User Testing of Mobile Games . . . 62

5.1.1 Remote Testing . . . 62

5.1.2 Test Users . . . 64

5.1.3 Test Method and Procedure . . . 66

5.1.4 Social Aspects . . . 67

5.1.5 Post Gameplay Questionnaire . . . 68

5.2.1 Evaluation of tools and features . . . 69

5.2.2 Grading of SRTs and Test Services . . . 71

5.2.2.1 Website . . . 71

5.2.2.2 Easy to integrate . . . 72

5.2.2.3 Easy to set up test . . . 72

5.2.2.4 Customise test . . . 72 5.2.2.5 Demographics Specification . . . 72 5.2.2.6 Profile Information . . . 73 5.2.2.7 Researcher Environment . . . 73 5.3 Analysis of Recordings . . . 74 5.3.1 Voice . . . 74 5.3.2 Facial . . . 75 5.3.3 Read Emotions . . . 75 5.4 Workflow . . . 76

5.4.1 Planning the Test and Writing Instructions . . . 76

5.4.2 Pilot Testing . . . 77

5.4.3 Deciding on a Session Recording Tool . . . 78

5.4.4 Deciding on Recruitment, Distribution and Test Set Up . . . 80

5.5 Further Research . . . 81

6 Conclusion 82 References 84 Appendix A - Initial Test Instructions 89 Appendix B - Declaration of Informed Consent 90 Appendix C - Test Procedure: Lookback 91 Appendix D - Test Procedure: UXCam 93 Appendix E - Pre Gameplay Questionnaire 94 Appendix F - Post Gameplay Questionnaire 95 Appendix G - Session Recording Tool Survey 96 Appendix H - Questions for the Pilot Test 97 Appendix I - Final Workflow 98

List of Figures

2 Illustration of the flow concept developed by Mihaly Csikszentmihalyi [6] (adapted from an illustration by Senia Maymin [32]). . . 6

3 Questions answered by different UX research methods (adapted from [48] and [15]). . . 9

5 UXCam’s UX researcher environment. . . 33

6 PlaytestCloud’s UX researcher environment. . . 35

7 UserTesting’s UX researcher environment. . . 36

8 Beta Family’s test set up service. . . 39

9 PlaytestCloud’s test set up service. . . 41

10 UserTesting’s test set up service. . . 43

11 Age and gender distribution from a total of 26 post gameplay questionnaire respondents. . . 47

12 Test users preferences to start and stop the recording manually or automatically and where they would have preferred to conduct the user test. There was a total of 15 survey participants, out of which 7 had completed the test session using UXCam and 8 using Lookback. . . 49

13 The test users’ preferences regarding preview functionality in the session record-ing tool. There was a total of 15 survey participants, of which 7 had completed the test session using UXCam and 8 using Lookback. . . 50

14 Properties for test set up. . . 57

15 Using Lookback with the setting to show camera input in the lower right corner set to on. . . 70

List of Tables

2 Benefits and challenges with remote usability testing. . . 11

3 Session recording tools included in the initial investigation. . . 28

4 Test services which also provide SRTs and were included in the initial investigation. 29 5 Features available in the UX researcher environment (where recordings can be watched and annotated) for the respective services. . . 30

6 Properties for the SRTs. . . 31

7 Features for test set up and distribution services. . . 38

8 Grading of tools and services. . . 46

9 Choice of session recording tool. . . 53

Abbreviations

FACS Facial Acting Coding System GEQ Game Experience Questionnaire GUI Graphical User Interface

HCI Human Computer Interaction

ISO International Standards Organization NDA Non-Disclosure Agreement

NPS Net Promoter Score PX Player Experience QA Quality Assurance

SDK Software Development Kit SRT Session Recording Tool UI User Interface

UX User Experience

VIS Voice Interaction System

Definitions

Attitudinal methods: The attitudinal approach aims to collect data about “what people say”.

Behavioural methods: The behavioral approach aims to answer “what people do”.

Dashboard: In this thesis, the term dashboard refers to the session recording tool’s website where the recordings are uploaded and organised and where it is possible to view the user tests. Facial Acting Coding System: Facial Acting Coding System (FACS) is a guide which pro-vides categorisation of the movement of facial muscles by assigning all facial muscles a number which is modified when the muscles move.

Gameplay: Gameplay is created by the game developer and the player together. It is the model developed through game rules, interaction with the player, challenges, skills to overcome these challenges, themes and immersion.

Graphical User Interface: A User Interface (see UI below) that includes graphical elements such as windows, icons and buttons [10].

Heuristic: A heuristic is a method or procedure where experience is used in order to learn and improve [14].

Heuristic Evaluation: A couple of experts review and examine the UI and decide how well it conforms to recognised usability principles called “heuristics”.

Net Promoter Score: Metric for customer loyalty. The primary purpose is to evaluate the loyalty of the customers, and the net promoter score (NPS) is based on the question ”How

likely is it that you would recommend [company X] to a friend or colleague?”

Non-disclosure AgreementA legal contract through which the parties agree to not disclose any information covered by the agreement. This can cover for example confidential material, information or knowledge.

Playability: How fun the game is to play, how usable it is and how well the interaction style and plot-quality is. Playability is affected by the quality of the storyline, responsiveness, pace, usability, customisability, control, intensity of interaction, intricacy, strategy, degree of realism and quality of graphics and sound.

Player Experience: The experience a player has when playing a game, the UX of games. PX is targeting the player and the interaction between the player and the game.

Screener: A pre-defined first question in the test setup, to prevent people who are not in the target group to become a test user and continue with the test. The persons who give the correct answer can continue with the test while the others are denied, the users do not know what the correct answer is beforehand.

Session Recording Tool: In this thesis a session recording tool refers to a digital recording tool for mobile devices, using the built in camera of the device.

Test Service: In addition to session recording, this term also includes recruitment of test users, test set up and distribution of the test and the application. Some test services also offer analysis of the recordings.

Think aloud: Users vocally explain their thoughts while using the product.

Quantitative methods: Quantitative methods are good for answering questions like “how many” and “ how much”.

Qualitative methods: Qualitative studies collect data about behavior or attitudes through direct observations.

User Interface: The interface features through which users interact with the hardware and software of computers and other electronic devices.

Usability: How effective, efficient and satisfactory a user can achieve a specific goal in a prod-uct.

User Experience: The perception a user gets from using a product, or the anticipation of using it. This also includes the experience afterwards and subjective emotions.

UX Researcher Environment: In this thesis, the UX researcher environment refers to the session recording tool’s online environment where it is possible to analyse a recording, i.e. watch a video and add annotations.

1

Background

The background chapter contains six sections, where the first two introduces the study 1.1 and motivates why it has been conducted 1.2. The third section 1.3 explains the aim of the study as well as the objectives, and the fourth section 1.4 introduces and describes the test object. Finally, the disposition of the thesis 1.5 as well as the limitations of the study 1.6 are described.

1.1

Introduction

This thesis is written as a part of the Master’s program in Media Technology and Engineering at the Department of Science and Technology, Link¨oping University during spring 2015. The thesis work has been carried out in association with the mobile games company MAG Interactive at their headquarter in Stockholm, Sweden.

The aim of this thesis work is to evaluate different tools for testing the user experience (UX) in mobile games and to produce a workflow for how to conduct user testing with session recording tools (SRTs). The workflow will be used as guidelines for MAG Interactive, describing how the process of user testing should be conducted. When performing user testing with a SRT, there are other necessary parts besides the SRT. It is important to find suitable methods for recruiting the test users and creating a test plan. Also, both the application and the test instructions need to be distributed to the test users if conducting remote user testing. Methods for all of these parts will be discussed in the thesis and suitable services will be evaluated. Since proper user tests, including analysis, has to be carried out in order to evaluate the tools, it has been decided that the user tests should also generate valuable feedback about the game which can be used in future iterations.

1.2

Motivation

Currently, there is little research available regarding the use of SRTs in UX evaluation of mobile games. Therefore, the aim of this thesis work is to compare and evaluate easily accessible tools and produce guidelines which can be applied during user testing with these tools. Mobile games development is a rapidly growing and changing area, but it is difficult to make accurate evaluations of the UX and to perform user tests which will yield in reliable results. It is generally preferable to test the UX in the player’s natural environment. However, this usually means that the result can only be based on subsequent feedback, which can be problematic due to memory limitations of the player. Other factors, like personal preferences and difficulties explaining the experience in detail, can also affect the result. It is possible to conduct observational tests in focus groups (a moderated test session where a group of players discuss the game), but this involves placing the player in an unnatural environment which might affect the performance and behavior of the player. Additionally, focus groups typically only test new players and not old ones which might be desirable. Another method to test the UX, and the player’s understanding of the interface, is to use mobile SRTs. These tools can be used to record taps on a touch display along with everything the user sees on the screen, and in some cases even facial expressions and sound. The use of SRTs allows for tests to be performed in the player’s natural environment.

1.3

Aim

The aim of this thesis project is to produce a workflow for how to conduct user tests of mobile games using a SRT. Various mobile SRTs will be evaluated and compared against each other,

thence the top candidates will be integrated into the mobile game application Ruzzle. User tests will be carried out in order to investigate the on-boarding process of the game, as well as to establish which tool (if any) is the most appropriate in the context. The resulting workflow will contain general guidelines for how to conduct user testing.

1.3.1 Objectives

The thesis aims to answer the following questions: • How can UX be tested using mobile SRTs?

• Why is remote session recording a suitable approach and which available tools are the most appropriate for mobile games?

• How can recorded test data from SRTs be interpreted into information that can be used to address UX and usability issues?

1.4

The Test Object: Ruzzle

Figure 1: Ruzzle, a mobile game developed by MAG Interactive.

The test object, which the users have tested in the user tests during this study, is a mo-bile game called Ruzzle. Ruzzle is a social player vs. player word game developed by the Swedish mobile games company MAG Inter-active. The players can choose to challenge friends or strangers and the board is consti-tuted of 16 letters in a 4x4 grid (see figure 1). The game was inspired by the board game Boggle. The aim of the game is to find as many words as possible in two minutes. A word is formed by dragging a finger between adjacent letters on the board, where one word has to consist of a minimum of two letters. It is not possible to use the same letter-box more than one time in a word, and each word will only be awarded points once per round. One game is constituted of three rounds, each of the rounds are two minutes in duration and the total score determines the winner. Ad-ditionally, the different letters awards differ-ent points and the goal is to collect more points than your opponent before the game finishes. As you gain experience your level will increase. There is also a tournament mode where the player, after reaching level six, takes part in weekly tournaments. The goal in the tournaments is to get an as high

score as possible, and every player can play as many rounds as they like. Every week, the player competes against 19 opponents with the scores from the best round played. Ruzzle requires a network connection and is available on iOS, Android, Windows phone and Facebook. The test

object is an established game which has been downloaded around 60 million times, according to MAG Interactive’s observations in May 2015.

1.5

Disposition

In order to gain a profound understanding of what is needed when evaluating the UX of mobile games, this report begins with a theoretical part in chapter 2. This chapter contains various theories and definitions, ranging from UX to Player Experience (PX) and finally addresses mobile games and testing workflows. “Approach” in section 3 introduces the method and steps for conducting the study, while the chapter “Results” (section 4) demonstrates the results of the study. In chapter 5, “Discussion”, the results are discussed and evaluated based on the theoretical section and also based on knowledge gained from conducting the user tests. Finally, chapter 6, “Conclusion”, aims to answer the initial objectives presented in the introduction (chapter 1.1).

1.6

Limitations

This research aims to find a testing workflow suitable for the specific target game and similar games, hence the aim is not to declare an entirely general method that will work for every type of game. The thesis project is conducted during the course of 20 weeks, which is the time limit for the master’s thesis. When this study was initiated, several of the relevant testing tools were only available for iOS, hence iOS became the choice of platform for this study.

Since session recording of user testing using mobile devices is a relatively new field, especially within the area of mobile games, many of the investigated test services and SRTs are under development. They are frequently being updated and new features and tools are becoming available. The tables and other details regarding the tools and test services in this report has been collected with the information available when writing the report. Therefore, we reserve the right for eventual printing errors and outdated information.

2

Theory

The theoretical chapter consists of five sections. Section 2.1 explains what UX is, how it can be defined and what separates UX from usability. Section 2.2 defines what a digital game is and how the UX in games differentiates from regular software. Section 2.2.1.3 focuses on UX in mobile games and what separates it from UX in other digital games. Section 2.3 is about user testing; what it is and what different methods that exist including what parts they consist of. Section 2.4 covers SRTs; what it is and how it can be used in UX testing. Finally section 2.5 deals with workflows and what guidelines to follow when conducting user testing.

2.1

User Experience

The International Organization for Standardization (ISO) defines UX as: ”a person’s percep-tions and responses resulting from the use and/or anticipated use of a product, system or service” [19]. This includes the user’s emotions, beliefs, preferences, perceptions, physical and psychological responses, behaviours and accomplishments before, during and after use of the product. UX is also a consequence of interaction with a product or system, and the internal and physical state of the user as a result from prior experiences, attitudes, skills, personality and context of use [19]. According to the ISO, usability can be defined as: “The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.” (ISO 9241-11) [56]. The usability criterias can be used to assess aspects of UX, for example functionality, system performance, interactivity, assistance from the system, etc [19]. Usability refers to implementing an application in a way so that the user can perform a task effectively and efficiently, without having to put too much effort into it. The result should be of good quality and the time it takes to perform the task should be minimised, i.e the main focus is productivity [5]. Usability Partners [56] describes UX as a subjective extension of usability, which focuses on how satisfied the user is. In order to evaluate the UX it is important to go beyond the aspect of usability and evaluate the whole experience. UX is an important part of human computer interaction (HCI), but initially the term was not associated with games development [2]. Nowadays, the field of HCI and game research are learning from each other and HCI UX evaluation methods are used in games devel-opment for improving the UX of the game [2]. Due to the ”intentional challenge and emotion” in games, many HCI methods can not be used in the same way as when evaluating the usabil-ity of productivusabil-ity applications [29]. Creating a good (or bad) UX in games depends on the aspects: immersion, fun, enjoyment, usability, aesthetic, pleasure and flow. It is also important how all of these aspects influence the user [5]. Therefore, these factors have to be considered when evaluating the UX. In HCI, focus is often on the end result of the experience. The current UX evaluation methods often offer insights about the experience, but not objective knowledge. As stated by Calvillo et. al; “experience is both the process and outcome of the interaction” [5]. When interacting with an application, the user should feel that all “elements of the experience acted in symphony” [5], which will give rise to a positive experience. Hence Calvillo et al. [5] means that by evaluating the elements present in the interaction process, also the experience can be evaluated. An experience is both personal and subjective, but from a scientific point of view an evaluation of the general UX is needed. Even though the experience is personal, it is not often unique. It is possible to share an experience with others and to empathise with it. Even if an action is performed by an individual and gives rise to a personal experience, the same process of interaction when completing the task is used by many individuals. In the same way, the experience is many times the same or similar [5] which makes it possible to get an

idea about the general perception of the experience by observing or asking some of the users.

2.2

Digital Games

All games available on digital platforms such as PCs, consoles or mobile devices are digital games. A digital game can be distributed both online and offline, and can be available as both singleplayer and multiplayer. Games can be described as an activity one is participating in for fun. Entertainment can, however, be difficult to define since it is objective and depends on what the player is experiencing as fun [20]. For a game to be fun, it needs to be motivating, have rules and be engaging. It also needs to have a purpose and a feeling of presence. Rules can be displayed by success and failure rates in the graphical user interface (GUI), or be made up during the game process. Isbister states that specific usability measures are necessary for digital games. This is due to the fact that a game is a complex software with different goals and area of use, compared to traditional task-orientated software, which currently most usability evaluation methods are targeting [17] (page 8).

Researchers have different point of views, standpoints, methods and terminologies for de-veloping good UX in games. According to Isbister et al. [17] (page 5), there is a difference in testing the UX and the player experience (PX). The PX is the experience a player has during game play, i.e. the UX of the game. Testing the UX covers what it is like to interact with the software, including how engaging the experience is, regardless of the end goals. The focus of PX testing is to determine if the game is enjoyable and fun to play, but also to find out where the players might encounter problems and get stuck. The playability determines the quality of the game, i.e. playability is a kind of usability measurement for games, including the UX quality and how enjoyable the game is. The definition of playability is: “.. the degree to which a game is fun to play and usable, with an emphasis on the interaction style and plot-quality of the game; the quality of the gameplay..” [55]. Playability is affected by the quality of the storyline, responsiveness, pace, usability, customizability, control, intensity of interaction, intri-cacy, strategy, degree of realism and quality of graphics and sound. Since games are supposed to give rich and meaningful experiences, where also the gamers personal feelings are involved, the study of PX requires additional methods besides the usability methods used in the field of HCI. When playing a game, the player continuously evaluates his/her own performance in the game. This can be done both consciously and subconsciously. Is the player able to per-form, meet challenges and attain the desired goals? After reaching the goals, the player will experience positive feelings and perceive him- or herself to be competent [51]. Immersion, fun, presence, involvement, engagement and flow are concepts that has been used to describe PX, and these terms are often broadly defined. The concepts can be related to various psychological compartments, i.e the concepts are overlapping each other making it more difficult to measure and understand them. By using psychologically valid metrics when evaluating games, it makes it easier to measure the experience [51].

Becerra from AnswerLab and Smith from Electronic Arts state that “we don’t use games, we play them” [3]. Therefore, the PX should be measured from a playability, and not a usability, perspective (see table 1). Becerra and Smith say that if a game would be usable, it would be boring, since the players would understand everything instantly. Without the challenge, a game would not be exciting. However, menus, navigation and such still have to be usable in order to make it possible to play the game. Becerra and Smith explain that there are two different types of motivations for playing a game; task oriented and fun orientated.

Playability Usability Challenges are good Challenges are bad Surprising or unclear elements can

be positive and enjoyable

Localisation and understanding of all main elements should be instant Motivation is focused on the fun Motivation is focused on the tasks Fun is a big factor of success Ease of use is a big factor of success

Table 1: Clarification of the differences between playability and usability according to Becerra and Smith [3].

Figure 2: Illustration of the flow concept devel-oped by Mihaly Csikszentmihalyi [6] (adapted from an illustration by Senia Maymin [32]). Csikszentmihalyi came up with the

con-cept “Flow” [6], displayed in figure 2. It is about finding the balance between boredom and anxiety and can be applied on mobile games. If the player has good gaming skills, and the game is found to be too easy, it also gets boring. If the player on the other hand has bad gaming skills, the game can be too challenging which leads to anxiety. According to Lemay et al. [28], the flow-model can help designers grasp the answer to a fundamental question about their work: “What motivates people to play?”. Lazzaro studied why some players prefer action games, while others pre-fer word puzzles. The answer is that people play different games for different reasons, they have various goals and motivations. Lazzaro defined the Four keys of fun [26] which rep-resent the four most important feelings that a game can generate. The first key is called hard fun and acts on the emotions frustration and relief, while the second key; easy fun,

fo-cuses on curiosity and an easy going approach. The next key is called serious fun which provides relaxation and excitement. The last key, people fun, provides amusement and is enjoyable since it focuses on social bonding and interactions with other people. Lazzaro defines PX as how the player interaction creates emotion, and how well the game provides and satisfies the sort of fun that the players want [27]. The most popular games utilize all of these emotions in order to draw attention and motivate the player.

2.2.1 Relevant Genres of Digital Games

Digital games can be divided into different kinds of genres. Below follows information about genres of digital games which are of relevance in this study.

2.2.1.1 Social Games

A social game is where more than one person is simultaneously engaged actively in a game [18]. When evaluating the UX of a game, the social aspects of the experience also has to be taken into account. Isbister states that most games, both singleplayer and multiplayer, are usually played in a social context. It has been concluded that social play leads to more positive effects than solo play. Social games also provide more competence, while requiring less effort and inducing less frustration [18]. Isbister states that in order to thoroughly understand the user experience of games it is important to also consider the social nature of it. By adding people to the play session the player/players will have an entirely different end experience. Isbister also points out that social games should be tested in a social context [18].

2.2.1.2 Casual Games

Furtugno describes casual games as games that can be played by anyone regardless of age or gender [8]. They are available on all kinds of platforms, even game consoles (such as Nintento Wii) which has previously been used mainly by hardcore gamers. The success of casual games has made games reach a wider audience, but it has also generated new design issues. Since there is no restricted target audience, designing a game that appeals to everyone can be difficult. There are various definitions of casual games in different parts of the games industry, depending on the game content, game medium and play length or the current market. All of these are factors in the design of casual games, but Furtugno claims that the most important factor is to start thinking about who is intended to play the game. Proceeding from this, decisions and assumptions can be made regarding expectations, experiences and what they consider intuitive, challenging and fun.

2.2.1.3 Mobile Games

A mobile game is a digital game and as the name of the genre implies, it is played on a specific platform. Gaming has become mainstream, not only hardcore gamers are playing games as leisure activities anymore. The use of smartphones and tablets have made games more acces-sible and also made them available to a low cost, which has increased the number of so called casual players (players who occasionally or frequently plays easily accessible and easy to play games just for fun or in order to relax). Game designers are standing before the challenge to create appealing, accessible and usable games for players who are not typical hardcore gamers [28]. It is important to consider the general preferences of the targeted audience, Lemay inves-tigated if the typical hardcore gamer and the casual gamer experience games differently and concluded that different audiences are drawn to games. There are no universal guidelines for what constitutes a good gaming experience that can be applied to all groups of players [28]. When analysing a mobile game, it is important to find a balance between flow and challenge [52]. The problem is to find a good difficulty level so that the game is not too easy since that usually becomes boring, but also not too difficult since that becomes frustrating for the player. Regardless of how good the graphics quality is in the game, these kind of problems can lower the quality of the entire PX. When analysing a game it is also necessary to examine and find a balance between the playability, mobility and usability, as well as mobile device and touch screen properties. Korhonen discusses that usability of mobile games should be evaluated dif-ferently than other digital games since they are mobile and are used in different contexts than other digital games. Mobile games can also be used at various locations with differing lightning conditions and noise levels etc. and the player might need to focus their attention on other

things in their surroundings from time to time. Additionally, it is not possible to measure playability with the same heuristics as those developed for task-orientated software, since it is a game many different parameters, paths and stories are created by each player and hence the scenarios will differ from player to player [21]. ¨Ulger also states that mobile games may encounter several playability issues. One of the issues is handling interruptions, for example when receiving a phone call during a game session. Additionally, control of sounds and environ-ment can be restricted on the device. Another difficulty can be to help the player understand the game. Having a simpler user interface (UI) than in PC or video games is necessary since the screen resolution and size is more restricted. The character status (if the game contains a character) and the game goals need to be evidently visible to the player. Touchscreen dis-plays also present some specific issues, such as distribution of game items on the screen. The distribution of items should suit both right-handed and left-handed people. Since the mobile device is portable and can be used everywhere, the environment, noise and light may vary. Distinctive is also the fact that traditional mobile devices have rather small screens, insufficient audio capabilities, limited processing power and battery limitations [52].

2.3

User Testing

User testing, and measurements of the user experience, is conducted in order to improve the UX of existing UIs [1]. It is also conducted throughout the development process in order to collect feedback on concept ideas, game mechanics, design decisions, etc. which can be used to ensure that a project is heading in the right direction.

2.3.1 Different Testing Methods

There is a wide range of UX research methods, some are older and some are more modern. Rohrer states that it is nearly always best to use a combination of multiple methods, providing different insights. The methods can be separated into behavioural vs. attitudinal, and quanti-tative vs. qualiquanti-tative (see figure 3). The attitudinal approach aims to collect data about “what people say”, while the behavioral approach aims to answer “what people do”. Usability studies are in-between the attitudinal and behavioral methods, mixing the best from both areas by combining self-reported information with behavioral observations. According to Rohrer, it is generally recommended to go closer to the behavioral side when performing usability studies [48].

Figure 3: Questions answered by different UX research methods (adapted from [48] and [15]).

Qualitative studies aim to collect data about behavior or attitudes through direct observations. Qualitative methods are recommended to answer why an issue occur and how to fix it [48]. In quantitative studies, data is gathered indirectly using measurements, analytics tools or surveys. Quantitative methods are better for answering questions like “how many” and “ how much”. These methods can be used to grade issues according to severity, by concluding which issues are the most crucial ones.

2.3.2 Testing Methods

“When studying natural use of the product, the goal is to minimise interference from the study in order to understand behavior or attitudes as close to reality as possible” [48]. The natural use approach provides a greater validity to the study, but it also offers less control for the facilitator. If the main focus of the study is to investigate what people do, why they do it and how to fix potential problems; the study will focus on qualitative behavioral methods, observing the players in their natural environment and their everyday use. Heuristic evaluation is a method that consists of having a few experts review and examine a UI and thereafter decide how well it conforms to recognised usability principles or guidelines called “heuristics” [39]. Observational methods for usability testing are used to gather information about the user’s behavior. This can be face and body expressions, focus level, preferences, actions while performing tasks as well as when encountering errors, and also opinions and comments from the test participant. There are three different observational testing approaches; test monitoring, direct recording and think-aloud. In test monitoring, the observer directly observes, records and comments on the participant’s behavior throughout a test session. Typically, the practitioner follows a data sheet with guidelines containing checklists of tasks, test time, comments from the practitioner and the participant, as well as an explanatory statement of the event. It is also possible to use one-way mirrors which means that the test user is observed through a one-way mirror from an adjacent room. The test monitoring approach is the most common method where only a few test users are needed. Direct recording is suitable when having many test participants and when there is a need to eliminate potential bias from the test observer. There are different recording alternatives available, such as audio and video taping and screen recording. In the

think-aloud approach, the respondents are asked to verbally explain what they are doing while using the product, and they are reminded to regularly verbalise their thoughts throughout the test session [53]. There is also a usability evaluation method called focus groups, where a group of users discuss a product while being moderated by a test facilitator. The benefits of this method is that the users can discuss their experiences and results, which can lead to many useful ideas and feedback. This is also a cheap testing method, especially if being carried out before developing a product. The drawback is that it is an unnatural environment for the test users and an experience in a focus group could usually not be compared to a real experience in the users everyday lives[9].

User tests can be carried out in different contexts and environments. There is natural use, where the test participant is carrying out the test in his or her normal environment, like they would during everyday activity. There are also scripted user tests, where the participant is following a script. In some studies, a mixture of different contexts is used, which is called a hybrid. A testing method which can be conducted from the user’s natural environment is called remote usability testing, which will be further discussed below in section 2.3.3.

According to Krug and Sharon [24], it is important to ask the whole team (developers, designers, UX researchers etc.) what questions they want to have answered as a result of the testing process. They also point out that these questions are not necessarily the ones being asked to the users but rather what you want to know as a result of the study. They suggest that tests should be planned together with the entire team. The whole team should be involved in creating the test, choosing participants and they should be encouraged to provide feedback regarding the test process.

2.3.3 Remote User Testing

An alternative to regular observation of test sessions is remote user testing. The main difference between remote user testing and regular user testing, is that the test participants and the test facilitators are at different physical locations. This approach makes it possible for the participants to carry out the test in their everyday environment, without being distracted by the facilitator or disturbing equipment. The test process is perceived to be more natural for the test user, while the UX expert can still watch and analyse the test procedure from a remote location [49]. In a webinar (web-based seminar) at UserTesting.com [24], Krug and Sharon recommend performing a pilot test on one person in order to test the test before using it for user testing. This should be done in order to make sure that everything is working, and that the users will understand the tasks and the questions given to them in the test. Sharon insists on the importance of pilot testing and states that he has never regretted doing a pilot test, only regretted not doing one. One drawback with the remote testing method is that the possibility to interpret the user’s body language is essentially lost. The possibility to ask follow up questions in the middle of the test process is also ruled out (if having an unmoderated test session). On the other hand, remote testing does not require any traveling, neither for the facilitators nor for the participants. This saves time and also makes it possible to have geographically dispersed user groups without requiring a larger budget. Some advantages with remote user testing are that tests can be carried out with a limited budget, strict time constraints and without the use of additional testing facilities. It also offers a faster and easier recruitment process. The test group does not have to be collocated, and the test can be performed in a natural environment where the test participants feel comfortable [49]. There are both moderated and unmoderated remote user testing. In a moderated test session, the facilitator and the user are both online when the test is being performed and they can interact with each other during the test process. When using the unmoderated test method, there is no real-time interaction between the test

participants and the facilitators, and the test can be carried out asynchronously. Unmoderated user test studies provide flexibility, and the user can complete the test session when they have time and want to go through with it. Another advantage is that all test users can perform the test simultaneously, since the test data will be analysed retrospectively [49]. When carrying out unmoderated remote user testing, it is extra important with clear and specific instructions. The test facilitator can not assume that the test user will understand how everything works without thorough and easily interpretable instructions. When performing moderated user tests, the moderator can make sure that the user stays on the task, hence unmoderated tests place greater demands on instructions and preparations [24].

Usability.gov [54] summarises the main benefits and challenges with remote usability testing in table 2.

Benefits Challenges

• Eliminates the need for a lab as well as the need to place the test participant in an unnatural environment, which is usually the case when performing tests in a lab.

• Supports a larger group of test users. • Typically, less expensive than in-lab

testing.

• Possible to run multiple tests simulta-neously.

• Unmoderated testing allows for the test user to perform the test at any time suitable, increasing the possibility for all participants to complete the test. • Usually possible to test a larger number

of users than in a lab environment.

• Security could be compromised if the testing information is sensitive (since data might leak out).

• Restricted view of user’s body lan-guage.

• Technical problems are likely if the test users:

– Are not comfortable with the technology (which can be likely if the target group is not gamers). – Have conflicting software or

equipment installed on their device.

– Have a slow Internet connection.

Table 2: Benefits and challenges with remote usability testing.

2.3.4 Post Test Questionnaire

A post test questionnaire is a survey consisting of questions that the user should answer after testing the product. When performing online surveys, Nielsen recommends short and well written surveys which are easy to answer. This will yield in a higher response rate while avoiding misleading results [43]. According to Nielsen, the most important aspect regarding questionnaires is to maximise the response rate. Low response rates can be misleading since there is a chance they are based on a biased group of very committed users, hence the result could not be viewed as a representation of how most users experience it. In order to get as many people as possible to respond to the surveys, they should be ”quick and painless” [43]. The best

way to achieve this is, according to Nielsen, to reduce the number of questions. The questions also have to be easy to understand and the survey easy to operate, in order to avoid misleading answers due to misunderstandings. The questions should only address the core needs, Nielsen refers to Reichheld’s article ”The One Number You Need to Grow” where it is stated that only one question is needed in order to get insight into the customer-satisfaction. From this question the Net Promoter Score (NPS) can be calculated. NPS is a metric for customer loyalty which was developed by Satmetrix, Bain & Company and Fred Reichheld [38]. The primary purpose of the NPS methodology is not to evaluate the customers satisfaction with the product, but to evaluate the loyalty of the customers towards the brand or the company. Reichheld researched the link between survey responses and actual customer behaviour which resulted in one direct question ”How likely is it that you would recommend [company X] to a friend or colleague?” [47] (company X can be replaced by for example a product or a game). Based on their replies, the respondents are divided into different categories where some are considered to be beneficial for the company and some can affect the brand negatively [37].

Nielsen also suggests that an alternative approach to having all users participate in the same short survey, is to ask different questions to different users. In this way more questions can be included in the study while it is still being kept short for the individual survey taker. This might provide more insight into the UX [43]. Since digital games can differ a lot regarding application area, target players and game experiences, there is no universal measurement approach which fits all. This has lead to an absence of coherent tools to measure entertainment experiences in a reliable manner [16]. In an attempt to develop a generally applicable measurement, IJsselsteijn et al. developed the Game Experience Questionnaire (GEQ). GEQ is a post play questionnaire that takes into consideration seven dimensions of player experience: sensory and imaginative immersion, tension, competence, flow, negative affect, positive affect and challenge [16].

2.3.5 Testing of Digital Games

User testing of games can be performed using a range of different methods. They can be small one-to-one sessions or larger test sessions involving groups of people. The procedure can be everything from think-aloud, questionnaires and post-play interviews to an automated recording system which collects data about the player. Some user tests are performed in testing labs with special equipment, where the players can be observed and their actions and reactions documented. This is however a rather expensive procedure, which only makes it available to bigger companies with a large testing budget [4]. The data gathered during the test sessions focuses on both usability and playability issues. By observing the player playing the game, not only usability issues like problems with navigation, menus and level difficulties etc. can be deduced, it can also be established if the player had fun, and if so when and where in the game, as well as how fun it was [4]. Reviews and game forums relying on user feedback are common post-launch evaluation methods in games development. However, these do typically not generate detailed information and it can be difficult to deduce the cause of problems [4]. Brown claims that the most powerful UX evaluation tools “offer insight not just into the user experience but into the exact elements influencing the experience“ [4].

The disadvantages of user testing of digital games are that it takes time and costs money, while the reward is the ability to create games that the players want to play. This generates more players and more money for the business [35]. Even though digital games are different from medical devices, they are both using a UI and hence similar testing methods can be used. Oliveira et al. [45] studied usability testing of a respiratory UI in a medical environment, and investigated the possibility to use computer screen and facial expression recordings in usability testing of the UI. They say that a suggestion is to use computer screen recordings, instead of

other qualitative methods like interviews and questionnaires, when performing UX evaluation of clinical information systems. This is because it provides further objectivity while capturing UX problems in real-time. Similarly, the use of questionnaires and post-play interviews have been questioned in the gaming industry. Since questionnaires and post-play interviews are carried out after the game experience, both methods have been criticised for not being able to capture the user state during their engagement with the game. After the game, the test participants have to focus their attention on the evaluation, instead of the experience he or she just had and is supposed to evaluate [34]. Another source of criticism is the fact that emotional experiences can be difficult to describe in words since they are not primarily based on spoken or written language. The use of screen recording tools alone will however not give enough insight into the UX, the user’s emotions also have to be taken into account. In user testing of games, usability issues can be detected from analysing screen recordings, but the player’s emotions when playing the game can not be deduced only from this. To reveal the player’s feelings; physiological measurements can be recorded, survey methods can be used or facial recordings can be collected. Collecting physiological measurements will provide high resolution data, but requires complex and expensive technology and also a high level of expertise. As already concluded, surveys provide limited information. Recordings of facial expressions on the other hand, can be collected even with a smaller budget [45] and can provide clues of the players true emotions. As Darwin puts it in The Expression of the Emotions in Man and Animals: “They [the movements of expressions in the face and body] reveal the thoughts and intentions of others more truly than do words, which may be falsified.” [7]. Displaying facial expressions is a visible and distinctive way of communicating emotions [45] and by observing the users facial expressions during the test session, additional information about the UX can be deduced. Oliveira et al. concludes in their study that the combination of (computer) screen recordings and recordings of facial expressions can improve the evaluation of user interaction tests. Their study aims to improve user assessment during usability tests in order to improve efficiency and effectiveness in health information systems. As stated by Oliveira et al. the combination of screen recordings and facial expression recordings can be used to determine the emotions of the test participator [45]. Since an essential part of a good gaming experience depends on the emotional response of the player (see section 2.2), one can assume that the approach could work well also in user testing of games.

Isbister states that the games development process consists of five stages [18] (page 347) which are tested and evaluated using different usability methods and tools:

1. The first step is before the project has even started. The product will not be tested, but it is important to make time for user research in the development process and decide on what research to conduct. It is also necessary to add time to the planning schedule for addressing the usability issues discovered. A testing workflow needs to be developed, making it possible for everyone in the team to test consistently.

2. The second step is the concept phase. Once the target audience, genre and platform have been identified, it is easier to make decisions about the usability testing and specifically heuristic evaluation. The aim is also to design a game which is fun and to recognise which social and psychological aspects that matter.

3. In the third step, called the pre-production stage, it might be desirable to use expert evaluation in the team, to ensure that the UX objectives are fulfilled.

4. The fourth step is the production phase, where mainly classic usability methods, such as think-aloud, are used. If time and resources are available, physiological measures can be

used to verify the emotions evoked by the game.

5. In the post-production phase, continuous post-launch and playability testing will be needed if updates and new features are released.

2.3.6 Testing of Mobile Games

According to Isbister, it is preferable to perform user tests on mobile or handheld devices in a setting where the “players might engage in game play embedded in their daily lives” [18]. It is also important to test the game on a demographically appropriate group. For example, the people in the test group should have the same relationship to each other, as the people in the target group has to each other in their natural environment. Both environmental and contextual factors are important for achieving a natural gaming experience also during the testing process [18]. Testing the experience that a mobile game provides to the players can not be done using a computer [12]. Dragging fingers on a touchscreen display generates a different UX compared to computer mouse clicks. It is important to test the game on a real device in order to get realistic performance results. Real occurring events such as disruption, battery consumption, memory restrictions or charging of the mobile device have a large impact on the overall UX and playability [13]. Therefore, the best way to understand the UX of a mobile game is to test on a real device and not on a simulator. It is also important to test on many different devices, since most of the end users do not use the same high quality products as the developing company might have access to. Isbister states that video recording and screen capture makes it possible to do rich analyses of gameplay [18].

Inokon, producer at a mobile games company, was interviewed by Isbister and Schaffer [17] (page 161) and he highly recommends usability testing and describes it as ”removing the blin-ders”. Often the game developers get so close to the game that they lose the player perspective. The usability tests can in many cases reveal issues previously unnoticed, and it might be diffi-cult for the developer to recognise these after spending time and energy working on the game, but it can be these adjustments that help your game to become a market success. Inokon also explains that “usability can be lethal to a project if not used properly”, and lists his most important pieces of advice [17]:

• Take time to thoroughly observe the players and grasp the context of the notes. • Make time in the schedule for solving emerged issues.

• Not every issue will be addressed, choose to fix the most important changes for the game vision.

• A game is not a snapshot, the game is constantly changing so don’t procrastinate the evaluation for too long. Test it when the game is in alpha stage and functional.

2.3.7 Test Users

Isbister investigates the question ”If the developers are also players, why can they not test the games themselves?”, and the answer is: because developers are not typical gamers [17] (page 350). They already have a previous experience and knowledge of the game, resulting in biased opinions. It is necessary to test with both professional game testers and end users in order to get a valid player perspective. Recruiting users is the most difficult but also the most important part of user testing [42]. It is important that the game is tested by test users with a similar demography as the users who will play the game, in order to guarantee that it meets their

requirements. Despite this, not many companies have a procedure for regularly gathering test users. The traditional ways of finding test users are to recruit colleagues, family members and friends, or persuade random people in the streets or caf´es to participate. It is also possible to reach out to players through online communities or social media.

Another approach is to use an independent, specialised recruitment agency that finds test users for you, or to use an online test service that provides a community of test users. There are online services which offer both SRTs and user testing (such as PlaytestCloud which is further discussed below in section 4.2.3). The criterias for the test users should match the target audience regarding demographics such as age, gender, game expertise (casual or hardcore) and income [42]. However, Krug and Sharon insist that it is not a requirement to test on the exact target group, since valuable insights will be gained anyway [24]. Test users who are recruited, except for colleagues or family members, are often offered payment as compensation [42].

A common problem when performing user tests is that not all test users show up. According to Nielsen, the average show rate is 11 %, which is almost one in nine testers [42]. The no-show rates can be higher for remote user test studies, than for studies carried out in person. Additionally, unmoderated sessions may vary greatly in quality, therefore Schade recommends test facilitators to recruit some extra test users just in case [49]. Test sessions should be carried out throughout the development process, and the number of test users in one session differs depending on the purpose. Nielsen claims that it is enough to test with only five users in one session when performing qualitative usability testing [41]. Using more than five users is a waste of resources, it is better to run as many minor tests as possible with few test users. Nielsen and Landauer studied the number of usability problems found through usability testing [41], and discovered that about 80% of the usability problems can be found using five test users from the same target group. Testing the quality metrics in quantitative testing of a game, such as learning time, efficiency, memorability, user errors and satisfaction, requires four times as many users as qualitative user testing does [44]. Nielsen concludes that 20 is the optimal number of test users for quantitative studies.

The result from the 2013 UX Industry Survey [59] shows that most companies (40% of the respondents) uses an average of 6-10 test users per study while the report from 2014 [58] declares that 1-5 users is a more common test group size. According to the result from the 2014 survey, now 40% of the participating companies uses test groups consisting of 1-5 users, and the percentage who uses 6-10 users has slightly decreased. However, according to Krug and Nielsen, testing on only one person is better than not user testing at all. Krug claims that even a bad test with the wrong test user will reveal possibilities to make important improvements [23] [41].

2.4

Session Recording Tool

A Session Recording Tool (SRT) is a software which records the participant’s screen while using the product during a test session. This can be done remotely or at a specific location, for example from the user’s home or at the office. Session recording methods have been used for many years, traditionally using additional equipment (such as cameras and sleds) for recording the user. However, a new type of session recording, where software is used, has become available and popular in recent years. This software is called a SRT and can be integrated into an application, the UX can then be recorded directly in the mobile device or computer without additional equipment. There are a wide range of SRTs, each with their own benefits and drawbacks. Since the aim of this thesis is to evaluate the UX of mobile games, the focus will be on SRT’s with support for mobile applications. Some of the tools support facial and