HOW THE CHOICE OF VIRTUAL JOYSTICK AFFECTS USABILITY IN MOBILE FIGHTING GAMES

(1)

HOW THE CHOICE OF VIRTUAL JOYSTICK AFFECTS USABILITY IN MOBILE FIGHTING GAMES

Bachelor Degree Project in Informatics

30 ECTS

Spring term 2020

Tomas Granlund

Marcus Karåker Gustafsson Supervisor: Henrik Engström Examiner: Mikael Johannesson

(2)

Abstract

As the mobile games market keeps growing, game interfaces for mobile games also warrant further development. One area that especially has room for improvement is the touchscreen joystick. This dissertation focuses on the differences in usability for different touchscreen joysticks. A mobile fighting game prototype was developed for this purpose, and a user study was conducted in order to evaluate the difference in usability between two joysticks. Data was gathered and demonstrated, and an analysis of the logged data was conducted, showing no significant difference between the two joysticks.

The authors hope that the comparison methodology and measurements demonstrated may be used as a benchmark and framework for future research in mobile joysticks.

Keywords: mobile games, touchscreen joystick, fighting games

(3)

1 Introduction

Mobile gaming now constitutes more than half of the video game market (Torok et al. 2018a), and as mobile games grow in number and the industry becomes more competitive it is important for mobile games to constantly develop themselves. One of the big challenges that mobile game developers face is choosing and designing input interfaces. Virtual gamepads are greatly inspired by their analog counterparts (Kurabayashi 2019; Kim & Lee 2015), but limiting oneself to the control schemes of other consoles might be an unnecessary constraint.

There are many unique input interfaces for the mobile platform that could make games more fun and easier to interact with.

A mobile game prototype was developed for this study at the Japanese games company Cygames. Using the prototype in a user study, we explore whether touchscreen joysticks of different sizes show a difference in usability. The study uses usability as defined by Brown et al. (2010), with a focus on effectiveness and satisfaction. Effectiveness is measured using in- game quantitative data, while satisfaction is evaluated from questionnaire answers (Appendix B).

Originally the study was meant to investigate how the Kinetics joystick (Kurabayashi 2019), developed by Cygames, would compare to a conventional joystick in a mobile fighting game environment. However, due to the COVID-19 crisis, the internship at Cygames was cut short and the Kinetics technology could not leave the company grounds. This led to a restructure of the study, instead investigating different joystick sizes.

Even though no conclusions could be made from the results, a rather thorough analysis of the collected data was performed and presented. The results obtained from this study could be used as a benchmark for similar studies in the future, or to give further context.

(6)

2 Background

2.1 Touchscreen Input

The touchscreen as an input interface works distinctly differently from other human-computer interaction (HCI) techniques because it allows direct touch input on the output display. There are several defined ways to interact with a touchscreen, namely: tap, double tap, long tap, drag, multi-touch and flick (Kim & Lee 2015). Together these basic input schemes can offer the same functionality as most external input controllers like mice, tv-controllers and keyboards by providing the proper user interface displayed on the screen. In this sense, touchscreens allow for a type of dynamic interface (Torok et al. 2018b), meaning they can change their input interface depending on the situation. If the user needs to write, the display will show a keyboard. If the user needs to browse the web, the user can simply tap buttons and fields.

When it comes to video games, however, the interface solutions are often less obvious. As smartphone processing power grows, more and more mobile games are directly ported from other devices, consoles and machines, like PC and Playstation (Sony Interactive Entertainment 2013). These consoles often rely on many buttons and interface configurations (Halloran & Minaeva 2019; Torok et al. 2018a) that are not easily replaced by a touchscreen interface.

2.1.1 Touchscreen Gamepads

The two main features of a physical gamepad are simple buttons for gameplay actions, and an interface for directional input (i.e. a joystick). Baldauf et al. (2015) investigate four different touchscreen gamepads that resemble this pattern: directional buttons, eight-way D-pad, gestural tilt control, and virtual joystick.

“Directional buttons” refers to four separate buttons for up, down left and right, like one can find on the Nintendo 64 controller (Nintendo 1998), usually in yellow. The D-pad or directional pad is an eight-directional button pad in the shape of a cross, similar to those on XBOX 360 (Microsoft 2005) controllers.

Tilt control is when embedded technology, like a gyroscope, is used to allow the user to tilt the controller for directional input. While tilt control is present in physical game controllers like the Wii U controller (Nintendo 2012), it has seen heavy use in smartphone games as well.

Finally, the virtual joystick resembles the layout where a constrained knob is dragged in 360^o directions, usually returning to its origin position when not manipulated. Touchscreen gamepads also allow for a dynamic interface where buttons can be added or removed depending on the situation (Torok et al. 2018b). This is something a physical controller cannot do.

When touchscreen gamepads occupy the same visual space as the game content, it brings a host of new problems and questions to answer (Torok et al. 2018a), but also benefits. If the interface is on the same display as what is controlled, the distance that the eye has to travel between interface and output display is greatly minimized or in some cases even removed completely since some may view the interface through the corner of their eyes (Baldauf et al.

2015). As for the problems, they often have to do with size and positioning. According to Kim

& Lee (2015), most studies on displayed touchscreen interfaces before 2015 focused on the optimal size of virtual buttons and the preferred distance between them. The problem stems

(7)

from the inevitability that the virtual input interface will be rendered on top of anything else that is displayed. Additionally, buttons cannot be much smaller than a finger if correct and intended input from the user is to be collected. In the case of portable devices this is a problem because the display cannot be made too big before its size interferes with its portability (Kim

& Lee 2015). This has led to some studies in new and different input schemes for mobile games (Kim & Lee 2015; Kurabayashi 2019).

2.1.2 New Touchscreen Gamepads

Touchscreen gamepads have limitations that makes it difficult for them to reproduce the experience of playing with a physical gamepad. For example, they lack tactile feedback (Baldauf et al. 2015). Rather than trying to replicate physical controllers, it may be a better idea to design new controllers specifically for the touchscreen. Furthermore, according to Gerling, Klauser and Niesenhaus (2011) in a study with 45 video game players, those transitioning from their comfort platform to a new platform with the same game had an equally positive experience but felt more challenged and experienced more usability issues. It might not always be necessary for mobile input interfaces to imitate physical and conventional controllers since the only consequence of having a completely new interface in a video game is that users have to get familiar with using it. For example, the game Skullgirls Mobile (Autumn Games 2018) takes advantage of this by completely replacing the conventional directional input with flick-gestures, a unique input interface for the touchscreen.

2.1.3 Defining Virtual Joystick

As discussed previously, there are many types of virtual gamepads. However, this term is very broad and includes all different kinds of input interfaces for mobile games. Virtual gamepads refer to the previously mentioned: directional buttons, eight-way d-pad, gestural tilt control, and the virtual joystick, and also gamepads that utilize the unique input gestures of a touchscreen: multi-touch and flick. In order to conduct a measurable study and output clear data we want to limit the study to specifically investigating the virtual joystick. We define the virtual joystick as a virtual touchscreen gamepad where continuous input is gathered in a defined area on the screen and full 360^o output can be generated. This is in contrast to the d- pad that can only output a limited number of directions.

2.2 Game Design and Game Interface

Cummings (2007) studies how the interface through which the user interacts with the game has had a vital role in how games have been designed. The game controller’s design and capability determine the character and possible complexity of interactions between player and game software. Incorporating an “analog joystick”, capable of expressing velocity and direction, was first introduced to home-console gaming with the release of the Nintendo 64 (Nintendo 1998) console. The joystick made navigation in 3D-environments easier, and so video game home consoles for 3D, released after the Nintendo 64, followed suit.

With the introduction of smartphones to the gaming market, things have changed dramatically. Instead of inventing new hardware that aims to improve the gaming experience, the sheer market size of the mobile gaming industry (Torok et al. 2018a) instead encourages game developers to invent software on the new and pervasive hardware. The virtual gamepads, however, lack the tactile feedback of physical controllers (Baldauf et al. 2015) and this affects user game performance and player enjoyment (Zaman, Natapov & Teather 2010; Brown et al.

(8)

2.3 Game Controllers and HCI

This previous research by Brown et al. (2010), includes a user study on 12 users who play a racing game using three different game controllers. The study is evaluated on three measurable factors of HCI research that Brown et al. (2010) adapts to research on game controller design: functionality, usability and experience.

Functionality refers to if the controller can technically be used to achieve the given video game tasks. This is distanced from any user context and can be answered by technical analysis of the controller in relation to the game.

Usability is dependent on four factors: effectiveness, efficiency, satisfaction and context of use, Brown et al. (2010) explain.

Effectiveness refers to the ability of the user to perform the task using the technology. This is different from the previously discussed functionality as the potential for the user to complete the task must also be taken into account: a controller that technically can complete a task is not of much use if not operable by a user. In their user study, effectiveness was measured using the race lap completion time as a variable.

Efficiency refers to how demanding the task is on various resources, for example mental effort, physical effort and time. In their user study they measured efficiency with a questionnaire on mental effort.

Satisfaction refers to how comfortable and satisfied the players are with using the game controller. This was measured with a questionnaire for evaluating user satisfaction with electronic consumer products.

Finally, context of use refers to how the environment and user differences affect the usability of the controller. This could concern the hardware and software that the controller is designed to control, as well as any physical or social factors that differ from user to user.

It is important to note that when studying game controllers, satisfaction is a completely subjective concept that must be measured by asking the user directly, and it is common that context of use directly affects attempts to measure satisfaction due to social contexts, physical environment or hardware capability.

Experience is “the psychological and social impact technology has on users” (Brown et al.

2010, p.215). The user experience, although hard to measure, was evaluated through a questionnaire where users got to define their three most positive and three most negative experiences with the controller. The users were also asked to rate which controller they liked best in different situations.

2.4 Examples of Control Schemes for the Virtual Joystick

In order to decide upon which kind of mobile game would be the most interesting environment to investigate virtual joysticks, several games representing different genres were tested and considered. The following games were chosen due to their status as popular mobile games that make use of a virtual joystick as a major part of the interface. A recurring pattern is that these games use the virtual joystick almost exclusively for character movement.

(9)

2.4.1 PUBG Mobile

PUBG Mobile (PUBG Corporation 2018) is a third-person shooter game for mobile touchscreen-based devices. It uses a fully dynamic control scheme where the interface can be customized. The primary use for the joystick is to move the character. Aiming in PUBG Mobile is done by dragging your finger anywhere on the screen. Since this is not always optimal for fine-tuning your aim, there is also the option of moving your entire device as if using an AR camera. This relies on the mobile device supporting tilt-functionality.

Figure 1. The customizable interface of PUBG Mobile.

2.4.2 Honor of Kings

Honor of Kings (Tencent Games 2015) is a chinese “MOBA” game played on mobile. You control your character’s movement using a single joystick. Precise controls are important to this game due to its competitive multiplayer aspect. Aiming abilities is done by dragging your finger after pressing an ability-button, meaning that the joystick is only for movement. Camera control is not necessary as the camera is always centred on your own character.

2.4.3 Honkai Impact 3rd

Honkai Impact 3^rd (MihoYo 2016) is a Hack-n-Slash action game rendered in 3D and viewed from a third-person perspective. The character movement is controlled using a virtual joystick.

There is the option of letting the game automatically control the camera, and at the same time which opponent to attack, or of manually controlling the camera. In manual camera control, the space that is not occupied by other controls like buttons is used for camera control, like in PUBG Mobile.

(10)

Figure 2. Gameplay screenshot of Honor of Kings.

Figure 3. Gameplay screenshot of Honkai Impact 3

^rd.

2.4.4 Street Fighter IV: Champion Edition

Street Fighter IV: Champion Edition (Capcom 2017) is a mobile fighting game that uses a virtual joystick in the same way that conventional fighting games would use a real joystick.

This is a design choice that some other mobile fighting games like Skullgirls Mobile (Autumn Games 2018) and Flappy Fighter (AAPCOM 2018) have chosen to avoid, presumably due to the lack of haptic feedback on the joystick in combination with the expectation on players to perform very complex joystick inputs to trigger certain special attacks.

(11)

Figure 4. Gameplay screenshot of Street Fighter IV: Champion Edition.

Figure 5. Gameplay screenshot of Dead Trigger 2.

2.4.5 Dead Trigger 2

Dead Trigger 2 (Madfinger games 2013) is a first-person shooter game available for touchscreen-based devices. The controls are in many ways similar to PUBG Mobile: a virtual joystick is provided for moving around your character and aiming is primarily done by dragging your finger across the screen. However, the character you control is never seen on camera due to the first-person perspective.

(12)

2.5 Previous studies 2.5.1 Touch and Play

Halloran and Minaeva (2019) performed a user study involving 14 testers playing the same game genres, using three different control methods: PC, Console and touchscreen. Four different genres were tested: puzzle, platformer, racer and FPS. For the racer and FPS genres, the results showed a significant preference towards console and PC, over the touchscreen- based controls. But for the platformer genre, the touchscreen scored almost evenly with console, and for the puzzle genre, touchscreen was about as good as any other control method.

They conclude that the games are natively designed with a platform in mind. Therefore, they are not inter-substitutable. Directly porting console or PC games to the mobile platform requires careful consideration and is a dangerous design hazard, because an emulated game on a different platform is just seen as inferior to the original platform it evokes. These results strongly encourage game developers for mobile games to play on the strengths of the native platform (Halloran & Minaeva, 2019).

2.5.2 Touchscreens vs. Traditional Controllers

Zaman, Natapov and Teather (2010) present a small study comparing the use of physical controls on a physical portable gaming console with touchscreen controls on a mobile device.

Their study focuses on level completion time and player deaths, and come to the conclusion that physical controls are more efficient.

During the analysis they encounter the problem that player deaths per level and completion time per level is correlated: dying means starting over at a checkpoint, which means that the player’s completion time is affected.

2.5.3 Game Controller Impact on Immersion

A study on the impact that mobile game controllers have on immersion was performed by Cairns et al. (2014). Two studies were conducted, both using the questionnaire provided by Jennet et al. (2008), comparing the use of the touchscreen to the use of other built-in interface mechanisms present in a mobile device, like accelerometers for tilt-based controls. The results for the first study on a mobile racing game indicated that the tilt controls were more immersive than the touch controls, but in the second study on a platforming game the results showed that touch controls were perceived as more immersive.

2.5.4 Game Controllers and HCI

Brown et al. (2010) also performed a study where 12 users played a racing game using a keyboard, gamepad and steering wheel. They conclude that the steering wheel, which is a controller specifically designed for the game genre, was difficult to use for inexperienced users.

However, they claim it was clearly superior in functionality, and that while less skilled users failed to make use of its functionality, more skilled users would be able to play the game better by using it. This seems to point towards an interesting aspect of game controllers: namely, the balance between their potential usefulness and understandability. The study was done with a very large scope, attempting to capture many different aspects of HCI all at once. This led to some compromised results (Brown et al. 2010).

(13)

3 Problem

The aim of the study is to investigate how the choice of virtual joystick affects usability in mobile fighting games. Usability is however a broad term and can be divided into four specific areas: effectiveness, efficiency, satisfaction and context of use (Brown et al. 2010).

Effectiveness refers to the ability of the user to perform the task using the technology, which in this case is the joystick. This can be tested by gathering data about user performance.

Efficiency is how demanding the task is on resources, like mental effort. This might be more difficult to measure as performance data: subjects might be using a lot or little effort but still be performing well. Satisfaction is completely subjective and is in the study by Brown et al.

(2010) measured with a questionnaire. Context of use refers to how the environment and user differences affect the usability of the controller. For this study, a focus on effectiveness was chosen since it can be measured by collecting runtime data in-game. It was also decided that it would be interesting to measure the satisfaction of the users when it comes to the joystick.

This means asking them how comfortable and satisfied the users are with using the joystick, much like Brown et al. (2010), in a questionnaire.

This study is limited to only consider how the size of the virtual joystick affects usability in mobile fighting games. In this case, a small joystick and a big joystick are considered as two different types of joysticks. This limitation was applied after the study had been restructured to accommodate for the fact that the Kinetics joystick could no longer be used and tested. This was due to the time in Japan being cut short because of the COVID-19 crisis and it was not allowed to take the Kinetics technology outside of Cygames.

3.1 Method 3.1.1 Hypothesis

The hypothesis is that the size of the virtual joystick will affect usability (Brown et al. 2010).

3.1.2 Evaluation

To answer the hypothesis a user study was conducted where subjects played a mobile fighting game prototype including two challenges, one for a small joystick and one for a larger joystick.

Each challenge prompted the subject to do twelve attacks in a seemingly random order: 3 of each of the 4 attacks. If the experimenter noticed that the subject could not finish all the designated attacks, the subject was shown how they could skip the challenge by holding three fingers on the screen for 5 seconds.

As mentioned earlier, two specific parts of usability were studied: effectiveness and satisfaction. The effectiveness component can be evaluated using purely quantitative methods, which in this case was done using the logs recorded during the user study. Conclusions regarding satisfaction must be entirely drawn from the qualitative questionnaires.

To measure the effectiveness of the game controller, the concept of effectiveness introduced by Brown et al. (2010) was used. The game used for the evaluation was a mobile fighting game prototype developed by the authors of this study. Two variables were compared:

 time between successful attacks (interval) and

(14)

The difference between the two joysticks in mean interval and attempts was calculated for each subject and the probability of a significant difference was tested using a paired within-subject T-test as well as a comparison of mean values.

Satisfaction was measured by having the subject answer a questionnaire (Appendix B) after having finished the two challenges. This questionnaire included questions about gaming habits as well as questions about which joystick the subject preferred.

Due to time constraints and COVID-19, subjects were chosen from friends or relatives that were available for virtual or real-life meetings. Although all of the subjects still had minimal knowledge of the experiment, the fact that they had some kind of relationship with the researchers could affect the study results, especially the questionnaire answers. Before beginning the test, the subject was shown the informed consent document (Appendix A) and was asked to sign it. Then a short introduction in how to play the prototype was given, the majority of times through a demonstrative video, but also by physical demonstration. Each subject played with both joysticks, and exactly half of the subjects started the experiment with one joystick while the other half started with the other. After playing the mobile game prototype, they filled in the questionnaire (Appendix B) via Google Forms.

3.2 Method Discussion 3.2.1 Determining the Scope

In reality, virtual joysticks are not limited to a certain application, but extend to the way that application is implemented in the game. For example, the mobile version of Terraria (505 Games, 2013) uses a joystick that is invisible until the player touches the screen, upon which it appears at the touch point. In this study we chose to test on what appears to be a very common way to implement a joystick for mobile fighting games: having a static joystick at a corner of the screen.

Due to the limitations of the study it was decided to focus on the two most simple aspects of usability – effectiveness and satisfaction. The two remaining aspects, effectivity and context of use, were deemed too time-consuming to include in the analysis for the scope of this study.

3.2.2 Previous Experience

If some subjects have more experience using one of the joysticks over the other, the data may become biased. While this may be mitigated by filtering test subjects on certain criteria, bias can still emerge from other factors. For example, testing only on subjects without previous experience in mobile games might still result in biases from subjects who have played more games in general, although these biases would arguably be weaker. The study benefits from subjects who are familiar with gaming because it makes its results more useful to the field of game studies, but subjects’ previous experience with gaming was still collected in the questionnaire as additional information.

3.2.3 Experiment Ethics

Only the game’s log and the questionnaire answers were collected during each experiment, and no cameras or recording devices other than the game’s logging software was used, to ensure the subject’s privacy. The subject was not identifiable within the collected data, and during the experiment the subject was able to choose whether they wanted to do the experiment in private or not.

(15)

Before the experiment was started, we presented the subject with an informed consent document (Appendix A). The document detailed the purpose of the experiment, the general procedure, the information that was gathered and how long that information was stored. This is in accordance with the guidelines from the Swedish Research Council (2020) and the Act concerning the Ethical Review of Research Involving Humans (SFS 2003:460) for experiments involving human test subjects. Only if the subject read the document and chose to give their consent, the experiment could proceed. The subject was also given the right to withdraw their consent and/or cancel the experiment at any time.

3.2.4 Questionnaire

Some data could not be captured in-game alone, since the actual experience and subjective thoughts of the subjects might not always directly translate to their effectiveness (Brown et al.

2010) and performance in the game. To accommodate for this, a questionnaire was used. With the questionnaire we aimed to ask questions about the subject’s thoughts on the two joysticks, as it would be interesting to compare how well they performed practically with which joystick they preferred subjectively, in line with Brown et al. (2010)’s Satisfaction component. The questionnaire, from here on also called the joystick questionnaire (Appendix B), was therefore used to find how both the subjects' subjective thoughts and gaming habits aligned with their effectiveness (Brown et al. 2010) in the mobile fighting game.

(16)

4 Implementation

4.1 The Choice of Joystick 4.1.1 Conventional Joystick

The Free Joystick Pack (Fenerax Studios 2019), henceforth simply “conventional joystick”, is a freeware virtual joystick available in the Unity Asset Store. It supports two-dimensional vector output in 360^o and comes with UI and Unity scripts that are completely open and customizable. It has several pre-defined templates for joysticks. One is the static template, that always stays in the same place on the screen. Then there is the floating template, that follows the finger of the user once the handle is dragged beyond the joystick’s radius. Finally, there is also a dynamic template that makes the joystick appear whenever the user touches the screen in a certain area, making that touch point the center point of the joystick until the user releases their finger again. Because of a dominating appearance in the mobile fighting games that were field tested by the authors, a static virtual joystick at the bottom corner of the screen was selected for the conventional joystick.

4.1.2 Kinetics

As previously mentioned in the background, there are a few studies that aim to find alternative gamepad interfaces to the conventional ones based on physical controllers. One of the new systems proposed is Kinetics by Kurabayashi (2019). By itself it is an engine for recognizing detailed movement from a very large touch area, like a whole thumb pressed against the screen (Kurabayashi 2019). The idea is to use this engine to emulate a joystick that does not require the user to drag their finger across the screen, but rather keep their finger static on the screen and enable directional input through slight tilting finger movements.

The Kinetics algorithm works by continuously sampling points from touch input. These points are fed into a buffer that keeps track of new points and older points for comparison. By analyzing the history of the data points the algorithm outputs a vector direction that can be used directly for gameplay purposes (Kurabayashi 2019).

4.1.3 The Differences

The main benefit of Kinetics over the conventional joystick is that it can recognize very slight tilting movements of a large finger area accurately. Therefore, it is potentially superior in emulating a touchscreen joystick meant to feel as though the user is tilting a real joystick

“beneath” the touchscreen (Kurabayashi 2019). However, the lack of user interface and haptic feedback means that it can be difficult for the player to understand what inputs are being interpreted by their finger movements. This means that Kinetics is, at first, difficult to learn how to use without some kind of visual aid. But once the player has learned how to use the joystick, it is possible that the visual aid can be omitted.

As for the size of the UI, while the conventional joystick can vary in size depending on the requirements of the game, Kinetics needs about 60x60 pixels of space on the display (Kurabayashi 2019). The conventional gamepads usually feature several buttons and according to Kim and Lee (2015) it is not recommended to decrease virtual buttons sizes below 7x7 mm, which translates to about 30x30 pixels. This amounts to the UI taking up a considerable amount of space on the display, and smartphone touchscreens usually have very

(17)

little space to spare. Kinetics, while functionally taking up a space of 60x60 pixels, visually only takes up the space of the finger placed on the screen.

It could be argued that Kinetics further benefits from avoiding the need for the user to use large dragging motions or keep track of the center of the joystick (Kurabayashi 2019).

However, it should be noted that simply reducing the size of a conventional joystick will also not require the user to do any dragging finger movements, as is the case with the virtual joystick used in Dead Trigger 2 (MADFINGER Games 2013).

4.2 The Choice of Game Genre

In the background chapter, five different examples of well-known mobile games were introduced: PUBG Mobile, Honor of Kings, Honkai Impact 3^rd, Street Fighter IV: Champion Edition and Dead Trigger 2. By creating a similar control schemes to one of these games when developing the prototype, we argue that the research will be made relevant to existing mobile games.

The decision of which game to use as a basis was evaluated based on the following factors:

1. The prototype is feasible to make within 2-3 weeks and

2. the comparison of joysticks within the game is believed to yield a difference.

The first factor was evaluated based on the authors of this study’s assessment of their previous experience with game development. The second question was answered based on the opinions of Japanese game enthusiasts and developers at Cygames, who are knowledgeable in the properties of Kinetics.

4.2.1 Game Evaluation

The Honkai Impact 3^rd-like prototype is difficult to assess for the authors as they have no experience with making similar games, which contributes to difficult development and uncertainty about the results of the prototype.

The Honor of Kings-like prototype was deemed feasible to make, but the difference between the two joysticks was deemed lesser compared to the other options.

The PUBG Mobile-like prototype was deemed feasible to make, and the joystick movement is crucial during the action-intensive moments of gameplay, so it was thought to be interesting to compare kinetics to conventional joysticks within this genre.

The Dead Trigger 2 prototype seemed easier to make than the PUBG Mobile prototype and was also thought to be interesting because it is a first-person shooter rather than a third- person shooter. This may be preferred because Jennet et al. (2008) mention that a player’s visual pattern covers a larger area of the screen in a third-person shooter, while they stay focused on the center of the screen in a first-person shooter.

Finally, the Street Fighter IV: Champion Edition-like prototype was an interesting candidate, as it is another game where joystick input is crucial during gameplay while the player is placing their focus elsewhere. Additionally, conventional fighting games use very precise joystick inputs as an important part of gameplay, which should generate interesting results.

(18)

4.2.2 Conclusion

Overall, shooters and fighting games were the most interesting candidates for the study. As was mentioned by Jennet et al. (2008), intense moments in shooter games draw the player’s attention to the place they are aiming, and in these moments, they expect to be able to move their character accurately without losing focus on aiming. This reasoning could support the hypothesis that there is a noticeable difference between the two joysticks when they are used under the same circumstances.

Fighting games are highly demanding on the speed and precision of inputs. Movements like

“dragon punch”, also known to be the input for the move “Shoryuken” from the Street Fighter series of games, were very difficult to do well when testing Street Fighter IV: Champion Edition for Android, as well as other well-known fighting games released for mobile. When fighting games and shooters were compared, fighting games were deemed slightly more time- consuming to make a prototype out of, but were also thought to be more yielding to differences between the two joysticks when asking Cygames employees.

A small experiment was done by playing the first-person shooter Dead Trigger 2 using its default joystick (which is of the same type as the conventional joystick) but using the finger on the touchscreen in the same way as Kinetics would have been used, had it been implemented into the game. The result indicated that even when played by tilting your finger as if playing with Kinetics, the joystick of Dead Trigger 2 functioned normally. Because of this result, the thoughts of Kinetics yielding a noticeable difference in first-person shooters were abandoned, and instead fighting games were chosen due to the complicated finger movements required to perform attacks.

4.3 Developing the First Version of the Game Prototype 4.3.1 Specification

The first version of the game prototype is illustrated in Figure 6.

(19)

Figure 6. The fighting game prototype.

The prototype started at the screen illustrated in Figure 6. The purpose of this screen was to give the player time to get used to the joystick, as was done by Oshita and Ishikawa (2012), before pressing “START”, when the game starts logging player activity. The player is then prompted with “Hadouken”, along with instructions of how to do the gesture at the top of the screen. Once the player performs “Hadouken” correctly five times, the “Hadouken” prompt is replaced with the instructions of how to perform “Shoryuken”. This is repeated five times again, having the player perform a total of 10 “Hadouken” and 10 “Shoryuken”. Then, the game returns to the practice screen, but with a different joystick. The player can again try out the new joystick on this screen, and then press “Start” to perform the same test again but with the other joystick.

Moving the character was done using the left and right directions on the joystick. It did not matter where the character was positioned: doing a move correctly (but your character punching the air) still resulted in progress.

4.3.2 Gestures

For each attack in the game that was to be experimented on, a gesture was developed. Two gestures that were the same as common gestures from existing fighting games were intentionally chosen. In this project these are called “Hadouken” and “Shoryuken” from their respective names in the Street Fighter series. The term “gesture” refers to the directional inputs that the player needs to perform before pressing the “Attack” button in order to do a certain attack. One example of such a gesture, “Hadouken”, is shown at the top of Figure 6.

Being some of the most common gestures in fighting games, “Hadouken” and “Shoryuken”

were thought simple enough for the subject to perform, but also difficult enough to yield an interesting result, striking a good balance for this study, and hopefully also contributing to the game’s flow (Csikszentmihalyi 1990).

(20)

In the case of “Hadouken”, the player is expected to do the directional inputs Down, Down- Right and Right in quick succession before pressing the “Attack” button. This will trigger a successful “Hadouken”. When testing the gestures with both the conventional joystick and Kinetics, it quickly became apparent that using the same gestures for Kinetics as the conventional joystick resulted in the player having to do two physically very different finger- movements.

In order to solve this problem, a new “Kinetics gesture” was defined for each attack. The

“Kinetics gesture” of an attack appears when the player is using the Kinetics joystick, and is designed to be similar to the physical finger-movement that is done when performing

“Hadouken” with the conventional joystick. Table 1 shows the conventional and Kinetics gestures for the attacks “Hadouken” and “Shoryuken”.

Attack Conventional gesture Kinetics gesture

Hadouken Shoryuken

Table 1. Conventional gestures and Kinetics gestures

The reasoning behind the design of the Kinetics gestures can be further explained through the illustration of the physical movement of the finger, relative to the origin of a conventional joystick. This is done in Figure 7.

Figure 7. Gestures for “Hadouken” and “Shoryuken” visualized.

In the first version of the prototype, only the Kinetics gesture of “Shoryuken” was implemented. For “Hadouken”, the conventional gesture was used both while the subject was playing with the conventional joystick and the Kinetics joystick, due to uncertainty of the necessity of defining a Kinetics gesture for “Hadouken” at all. In practice, this meant that the subject had to make a physical finger movement closer to Figure 8 while doing “Hadouken”

and using the Kinetics joystick in the first version of the game prototype.

(21)

Figure 8. Doing conventional gesture “Hadouken” using Kinetics.

4.4 Initial test

4.4.1 Experiment Procedure

An initial testing was performed containing 3 test subjects. After getting their informed consent, they were asked two questions relating to their previous experience with fighting games, mobile games and touchscreen joysticks.

After this, they were given the prototype with a small explanation on how to play. Once they had completed the challenge for one joystick, the prototype switched joysticks and they were interrupted with a questionnaire concerning the joystick. They then continued playing with the new joystick, and were given a final questionnaire at the end of the study.

The questions in the questionnaire asked questions extracted from the Immersion Questionnaire (Jennet et al. 2008) as well as specific questions about the joysticks and play session.

4.4.2 Results

Subject 1 began with using Kinetics. At first, they tried to use Kinetics by keeping their finger in the same place, but once they started dragging their finger across the screen instead, they seemed to prefer this method. Their general opinion was that using the conventional joystick seemed easier when doing Hadouken, but using Kinetics was easier when doing Shoryuken.

Subject 2 began with using the conventional joystick. While they completed the challenge using the conventional joystick, using Kinetics was much more difficult. It took them up to half an hour to complete the challenge, despite Subject 2 telling us that they were very familiar with fighting games. This led us to believe that the prototype was too difficult.

Subject 3 began with using the conventional joystick. It took them a while to be able to do

“Hadouken”, but even after that, “Shoryuken” was difficult. However, when switching to Kinetics they did a successful “Shoryuken” on their first try.

The general results of the questionnaire showed that subjects thought that the conventional joystick was easier to use and slightly more fun. Subjects reported feeling more immersed when using the conventional joystick.

(22)

For the questionnaire implementation, Google Forms was used. Each iteration of the questionnaire needed to be translated in order to be correctly presented to the Japanese testers at Cygames in Tokyo. This was time-consuming, especially for those involved in Cygames staff.

4.5 Developing the second version of the game prototype 4.5.1 Changes

Subjects who participated in the initial test had useful feedback for the prototype.

In the feedback retrieved from the subjects it was pointed out that only “Hadouken” was demonstrated during the tutorial phase. Once “Shoryuken” showed up in the game, there was a familiarity gap between how to do the two attacks. To avoid this, the practice phase was changed to display how to do all gestures.

Since the experiment had been very difficult for some subjects, including subjects who said they were used to fighting games, one possibility was that this was related to the choice of gestures. Since “Shoryuken” seemed to be one of the more complicated gestures, there were brief discussions about removing it completely. However, since this gesture is commonly used in existing fighting games, it seemed more likely that the difficulty in performing the gesture was due to some other factor. As such, “Shoryuken” was kept in the game prototype.

There was also the issue of challenging the subject with the same gestures after one another would give unreliable results. For example, in the first version of the prototype, “Hadouken”

was to be performed five times in sequence, and then “Shoryuken” was to be performed five times in sequence. There was a possibility that the subjects could mentally prepare for the gestures they would need to do next, which would generate biased data for attacks that appear after another attack. Therefore, two more simple gestures were added, “Left” and “DownUp”.

The prototype was made to present each attack in a seemingly random sequence, so that the subjects would not be prepared for which attack would come next.

There was a “Joystick” button that when pressed changes joystick. During the initial test, one of the subjects thought that this button was meant to be pressed. In accordance with the error prevention guideline mentioned by Kim & Lee (2015), this button was removed. Instead, a start screen with two options for which joystick to choose was implemented. The only way to view the screen again after an option was chosen was to restart the game completely.

The enemy and the ability to move the player character did not technically affect the result of the test in any way. However, it was still possible that the subject would be affected by the possibility to move their character next to the enemy and attack, even though it was not necessary to progress. These patterns were not observed in the initial test, but because this could still affect the performance of some subjects differently than others, the enemy and the ability to move your character were removed from the game prototype.

The game would start logging subject data when the subject pressed the “Start” button, but this neglects to log anything during the practice phase. Because the practice phase data is also valuable to analyse, another button named “INIT” was added which would start the data logging. This button can be seen in Figure 9. Once this button was pressed, it would be replaced by the familiar “Start” button that initializes the challenge, allowing the person

(23)

directing the experiment to demonstrate how to use the attacks to the subject, before pressing the “INIT” button and handing over the device to the subject for practice.

A log file was created the moment the “Start” button was pressed. This caused a lag spike on file creation, and the continuous I/O file streaming to the physical memory may have caused further performance issues as well. Due to this, it was decided to instead gather the log data in the allocated memory of the software, and output all of it into a file when a “Quit” button is pressed. This also enabled the option to not save a log for the session by quitting the application through other means, which was useful during development.

4.5.2 Specification

The second version of the game prototype is illustrated in Figure 9.

This version of the prototype starts with a selection screen for which joystick should be used first. When a joystick is picked the game opens in a neutral, non-test, phase. During this phase, the four different gestures, and how to perform them, are shown on screen. There are also three buttons. The “Attack” button, that is pressed in combination with a gesture in order to do one of the four moves, the “Options” button and the “INIT” button. The “Options” button is pressed when one wants to quit the game and end the test. A screen will appear with the option to “Quit” or “Continue”. If the “Quit” button is pressed, the game will quit and a log will be saved on disk containing all the gathered data of this session. The “INIT” button starts measuring data. Before this button is pressed, no data will be gathered and the button should therefore be pressed when the subject starts to play. When the “INIT” button is pressed, it will also change to the “Start” button. The “Start” button initializes the actual challenge and stops the practice period or tutorial phase.

When the challenge is started, the four tutorial gesture displays disappear and are replaced with one target move. Kinetics gestures were designed and used (when the Kinetics joystick was active) for all attacks in this version of the prototype. There are 20 moves or gestures in a pseudorandom order. It is important that it is the same each time and that it is not completely random in order to make it fair for each gesture. It is also important that it is not too predictable so the subject can assume what the next gesture will be. The game will only move on to the next target gesture once the subject has successfully done the last one.

(24)

Figure 9. The second version of the game prototype.

Once all 20 gestures are completed, the game will automatically change joystick and go back into the neutral, non-test, phase. The “INIT” button, as well as the four tutorial gesture displays, will reappear. The procedure is then repeated with the second joystick. When both the joystick challenges have been completed the game enters an end phase. The game no longer measures any data and there is no way to start another challenge. At this point, the game should be quit through the “Options” menu. If the test is to be discarded, the game can be shut down in an alternative manner. In that way, there will be no log file created.

4.5.3 Gestures

For the new version of the prototype, two more gestures were added to the game. The previous gestures, “Hadouken” and “Shoryuken”, were still unchanged. As new additions were “Left”

and “DownUp”. These were not directly translated moves from existing fighting games, like the two previous gestures, but were still based on some existing moves. The premise behind adding these moves was that they were much easier to perform than the two previous ones and would therefore add more diverse difficulty to the challenges. “Left” is just one direction on the joystick and is therefore the easiest one to perform. The “DownUp” gesture includes two directions and therefore works as a middle difficulty step between the singular direction of “Left” and the three directions of “Hadouken” and “Shoryuken”.

(25)

Attack Conventional gesture Kinetics gesture Left

DownUp Hadouken Shoryuken

Table 2. Updated Conventional gestures and Kinetics gestures

4.6 Second Test

4.6.1 Experiment Procedure

For this study, the questions about the subject’s habits with mobile games and fighting games were moved to the end of the experiment. Each experiment started with the subject giving their informed consent to participate in the test. After that, the subject would be shown how to do each of the 4 moves possible in the game prototype. Then, a time limit of 3 minutes on the practice period was introduced to give each tester an equal amount of time for practice.

Once the first joystick challenge was completed and the prototype switched joysticks, the subject was handed a questionnaire to fill out about their experience with the joystick. Then, they were once again shown how to perform each of the 4 moves with the new joystick, given 3 minutes to practice the moves, and at the end of the joystick challenge presented with two more questionnaires, one for their experience with this joystick and one for their previous experience with mobile game and fighting games, as well as general thoughts about the experiment.

4.6.2 Results

Subject 1 thought that using Kinetics was frustrating without a clear UI telling them what the joystick was interpreting.

Subject 2 thought that it took some serious practice before they were able to find a finger movement that would consistently let them successfully do the attacks. They said that Kinetics was easier to use, and made them feel less tired. However, they expressed that towards the end of the experiment, they felt tired.

Subject 3 gave up on the first joystick challenge. They said that they knew what to do, but could not input the “Shoryuken” attack fast enough. This indicates that performing “Shoryuken”

requires a high mental effort (Brown et al. 2010).

The results of the questionnaire reveal that subjects generally feel that they perform better when using the conventional joystick rather than the Kinetics joystick, and that they feel that using the conventional joystick was letting them feel like they were moving through the game according to their own will.

(26)

4.7 Developing the Third Version of the Game Prototype 4.7.1 Changes

Due to the experiments in the second pilot study taking a little too long, and some subjects getting stuck and frustrated, it was decided to cut down on the number of moves that the subject had to do. The number of moves for each joystick was changed from 20 to 12.

The logging structure was expanded upon to capture more data more efficiently, thanks to the advice of a helpful Cygames employee. While previously some data like the time in milliseconds before the last click of the “Attack” button had been calculated and logged with each click, now the system was changed to only log the necessary data, in this case the timestamp of the click. The time since the last click can then be concluded later by looking at all of the logged clicks. This sped up logging and freed the prototype from many bugs.

Thanks to the more flexible logging system being in place there was also time to implement the logging of more data. In the third version of the prototype, the screen coordinates of the finger position are logged every time the joystick is made to point in a new direction out of the eight possible directions. By logging this data, an analysis on where the subject places their finger on the screen, like the one demonstrated by Torok et al. (2017), will be possible.

4.7.2 Specification

Other than the changes mentioned above, the appearance and function of the third version did not change compared to the second version.

4.8 Logging

Data is captured automatically each time the subject taps the “Attack” button, since this is the button that has to be pressed in order to make a move. The data set collected for each click is as follows:

 Timestamp for the click.

 Last gesture (if any).

 Target gesture (if any).

 Gesture Buffer.

The timestamp is simply how long the game has been running, as well as a separate timestamp for how long the game has been recording data. The last gesture is the gesture that was done in combination with the “Attack” button click. This can be null in the case that no specific gesture was identified. The Target gesture is the gesture the game is asking the subject to perform. This will be null in the case that the subject is in the Tutorial phase and have not started a challenge yet. The Gesture buffer stores the ten latest directions the subject moved the joystick to in order, and uses those directions to evaluate a gesture. A size of ten was deemed reasonable to be able to see what the subject was doing while attempting to perform the gesture. From this small data structure, other data points can be derived:

 Clicks per successful gesture

 Average time for each successful gesture

(27)

4.8.1 Finger Position Logging

Data that is separate from the attack attempts is also logged every time the subject inputs a new direction. When this happens, the finger position and which direction the subject inputted is logged, along with a timestamp of when the change happened. In this way, there is a complete log of all directions that can later be matched with the ten directional inputs related to specific clicks of the attack button. This positional data of the finger can also be used to conclude how the subject was using the joystick during the experiment: for example, whether they were dragging their finger on the screen or keeping it in much the same place.

4.9 Coronavirus Aftermath

An immediate change of plans was necessary in order to do the experiments when the authors of this study had to evacuate Japan due to the COVID-19 outbreak complicating airborne travel. Because the authors of this study were not licensed to bring a build of Kinetics with them outside of Cygames offices, a remote experiment procedure was prepared to allow the experiments to be remotely controlled. These plans were also stopped in their tracks when the COVID-19 outbreak suddenly worsened in Tokyo, forcing Cygames employees to work from home.

It was then decided to change the aim of the study to concern a comparison between a different type of conventional joystick instead of Kinetics.

4.9.1 Deciding on a New Joystick

A joystick identical to the conventional joystick, except smaller in radius, was chosen for the new joystick. A joystick with dynamic position that appears wherever the subject places their finger was also considered, but due to the previously mentioned Dead Trigger 2 joystick showing similarities to Kinetics behaviour, a joystick that most closely resembles this Dead Trigger 2 joystick was deemed as the superior choice.

(28)

4.10 Developing the Fourth Version of the Prototype 4.10.1 Changes

Instead of Kinetics the smaller joystick was added. It was designed to be around the same size as the joystick in Dead Trigger 2, which was small enough for your thumb to completely cover it.

Figure 10. The prototype with the smaller joystick.

Also, a feature was added to aid testers who considered the test too difficult. By holding three or more fingers on the screen for five seconds, the prototype skips ahead to the next joystick challenge. This function should not be known to the subject in question until the experimenter deems it needed, at which point they should be told how to do it.

(29)

5 Evaluation

The goal with the evaluation is to find an answer to the hypothesis by comparing the two joysticks in usability (Brown et al. 2010), more specifically effectiveness and satisfaction. The effectiveness component can be evaluated using purely quantitative methods, which in this case was done using the data logs recorded in-game during the experiments. Conclusions regarding satisfaction must be entirely drawn from the qualitative questionnaires.

5.1 The Study

Ten subjects participated in the user study. As mentioned in Chapter 3, these were chosen from friends or relatives that were available for virtual or real-life meetings. Seven tests were done remotely on the subject’s own mobile device, while three tests were done physically on one of the authors’ mobile devices while being present in the same room as the subject. One of the subjects had a difficult time doing the Shoryuken attack, which led to the majority of the experiment getting skipped and only producing data for the first four attacks. This log was deemed too different from the rest of the experiments to be included in the analysis: thus, only the data from nine out of ten experiments was analysed. This means that the number of subjects who used the big joystick was five, compared to the four subjects who used the small joystick.

5.1.1 Experiment Procedure

Before beginning to play the mobile game prototype, the subject was shown a demo video instructing them how to play. Depending on the circumstances, a physical demonstration was used instead. After demonstrating, the subject was given 3 minutes to practice the attacks before starting the actual test. If the subject gave up during the challenge, or could not complete it in 10 minutes, the challenge was interrupted, and the subject would have to skip to the next challenge. After doing two challenges, one with each joystick, the subject would be handed the questionnaire. The log file produced by the test would then be sent from the mobile device to one of the authors’ E-mail addresses.

(30)

5.2 Analysis

5.2.1 Analysing Background Data

Figure 11. Mobile gaming habit. Figure 12. Fighting game habit.

As seen in Figure 11 and Figure 12, the subjects gaming habits vary greatly. Only one subject, plays mobile games weekly. Most of the subjects, however, seem to play mobile games every once in a while. When it comes to fighting games, though, most subjects very rarely or never play them.

Figure 13. Virtual joystick habit.

From Figure 13 it seems subjects where almost equally split on their virtual joystick habits, but no subject claimed to use the virtual joystick weekly or daily. Only one subject stated that they played fighting games on mobile at all, while the rest never do.

(31)

5.2.2 Analysing Effectiveness

In order to evaluate the results a program was created to compile the information available in the log files retrieved from the study. The following data was derived using this program:

 time spent on attempting to successfully do the attack (interval),

 the number of attempts it took to successfully do the attack (attempts).

The within-subject paired T-test was done by comparing the difference between the two joysticks’ mean interval and attempts for each subject. Both the test on interval (P = 0.42, mean difference = 0.43s) and the test on the number of attempts (P = 0.94, mean difference

= 0.04) indicated that a significant difference was improbable. Figure 14 and Figure 15 demonstrate the difference in mean interval and the difference in mean attempts for each subject respectively.

Various diagrams over the logged data can be viewed in Appendix C. Several figures contain visible outliers. These were all included in the analysis, as the overall number of samples were deemed too few. In order to clearly demonstrate the data, some diagrams provide a trimmed version which exclude the outliers.

The difference between the joystick the subject used first and the joystick that the subject used second was also calculated, using the mean interval and mean attempts as parameters. While the mean number of attempts showed no significant difference (P = 0.11, mean difference = 0.74), the mean interval was significantly lower for subjects when using the second joystick (P

= 0.04, mean difference = 0.97s). This indicates that interval and mean number of attempts decreased when subjects were using their second joystick, regardless of the type of joystick.

(32)

Figure 14. Box plot of each subject’s difference in mean interval between the two joysticks (negative values meaning the big joystick had longer intervals).

(33)

Figure 15. Distribution of each subject’s difference in mean attempts between the two joysticks (negative values meaning the big joystick had more attempts).

5.2.3 Analysing Effectiveness and Background Relation

In order to see if there is any connection between the subjects’ performance and their gaming habits a comparison between the quantitative and qualitative data was made. When comparing how much each subject played mobile games to how they performed, their time interval and their number of attempts, no correlation could be seen. Neither was there a correlation when comparing how used the subjects were to fighting games on any platform to their performance in-game (Appendix E). The comparison between how used the subjects claimed to be to the virtual joystick and how well they performed in regards to time and number attempts (Figure 16) only showed that those who answered that they never used the virtual joystick performed slightly worse.

(34)

Figure 16. The correlation between virtual joystick habit and performance for each subject.

1 = never, 5 = daily

The fact that there is no correlation to be seen between most of the data could be due to various reasons. It could mean that each subject has varying understanding of what playing mobile games “several times a week” entails, for example, and have therefore answered that they play more or less than they actually do. Also, even though a subject might be very used to mobile games or fighting games on another platform it does not necessarily say anything of how they would perform with the virtual joystick. The third comparison was a performance comparison to how used each subject was to the virtual joystick. This should definitely be a relevant skill in order to perform well in the prototype, however, no such correlation could be deducted from the data. In the end, there is little that can be said for any of the comparisons since there was such a small number of subjects that took part in the study.

5.2.4 Analysing Satisfaction

The joystick questionnaire can be used to put some of the logged data into perspective and to some extent evaluate the subjects’ satisfaction (Brown et al. 2010). The subjects were asked questions regarding what they thought of the joysticks on a scale from 1 to 5, where 1 is the small joystick and 5 is the big joystick. The first question: “Which joystick did you enjoy the most?” gave the median result of 5 for those who started with the small joystick and the a median of 1 for those who started with the big joystick. The second question: “Which joystick did you feel was easiest to use?” gave the median result of 5 for those who started with the small joystick and a median of 2 for those who started with the big joystick. From these results it would seem a majority of subjects preferred the joystick that they used during the second challenge, independently of which joystick they started with. The logs also show subjects completing the challenge with smaller failure ratios and intervals the second time around, regardless of which joystick they started with. Presumably this is due to the subjects having become more proficient by the time they get to the second challenge, which may also lead to a perceived greater usability for the second joystick.

(35)

Figure 17. Joystick intuitiveness.

However, no subjects seemed to think that the smaller joystick was easier to learn or most intuitive, as seen in Figure 17. The subjects that started with the small joystick thought that the bigger joystick was more intuitive, while many of those who started with the bigger joystick, a third, had no opinion on the matter. The few who did have an opinion, also leaned towards the bigger joystick. Some subjects also noted that the smaller joystick was harder to get a hold of and pin with the finger. Yet, one subject stated that it was “more responsive”.

5.3 Conclusions

Regarding effectiveness, our results show that a difference between the two joysticks on the measured parameters of interval and number of attempts is highly improbable. It is possible that subjects who were more proficient in general compared to other subjects all started with the same joystick, yielding unbalanced results. The analysis of the satisfaction component only showed that subjects would almost always prefer the second joystick they used. This could be due to subjects getting better at the game and at using the virtual joystick while progressing through the challenges and therefore enjoying it more. This is supported by the in-game data that clearly shows subjects performing better during their second challenge.

A total of nine subjects were tested, which is too thin of a sample size to draw any conclusions from. These nine subjects were also picked out of convenience and they therefore do not belong to any particular generalized group or population. Even if they did, nine subjects are way too few to represent said group or population. The hypothesis that there is some difference in usability can therefore not be accepted with these results.

HOW THE CHOICE OF VIRTUAL JOYSTICK AFFECTS USABILITY IN MOBILE FIGHTING GAMES