Making the Game Wiki Obsolete
Improving Approachability in Grand Strategy Games
Timmy Eklund
Department of Applied Physics and Electronics January 2020
Umeå University, Sweden tiek0006@student.umu.se
Abstract
The aim of this thesis is to explore and investigate the possibility of improve- ments in approachability and player retention, in Grand Strategy games. This is important for game developers to be able to develop games more suited to a wider audience. This study will investigate whether these improvements can be made by giving players access to information, traditionally found on game wikis, inside the game in a non-intrusive manner.
This was investigated through a pre-study of current trends in gamers usage of information resources outside the games themselves. It was determined that wikis are the most common resource, and that they are most commonly used in the middle of play. Because of this, three mock-ups of user interfaces, which could supply such information to the player in the midst of playing, were designed.
They were then evaluated by a panel of experts, consisting of game developers and experienced Grand Strategy players. The mock-up deemed by the experts to have the best potential, a windowed in-game encyclopedia, was implemented as a prototype in a small game. This game was developed to emulate the gameplay, and positive and negative aspects of Grand Strategy games. This prototype was then tested using A/B-testing.
Despite a low number of participants, the results showed that access to informa- tion found on wikis in the game does lead to better approachability. It can also be concluded that players are closer to reaching a Flow state, which will lead to potentially higher player retention.
Sammanfattning
Syftet med denna uppsats är att utforska och undersöka möjliga förbättringar inom användarvänlighet och bevarande av spelares intresse, inom spel i genren Grand Strategy. Detta är viktigt för spelutvecklare, för att lättare kunna utveckla spel som passar en bredare publik. Denna studie kommer att undersöka huruvida sådana förbättringar kan uppnås genom att ge spelare tillgång till information, som traditionellt sett finns på wiki-sidor, inuti spelet, på ett sätt som inte stör spelaren.
Detta undersöktes med hjälp av en förstudie av trender inom spelares använd- ning av information från resurer utanför spelen. Den vanligaste resursen visade sig vara wiki-sidor. De används framförallt under tiden som man spelar. Baserat på detta skapades tre förslag på användargränssnitt som kan tillhandahålla infor- mation till spelaren medan de spelar. De utvärderades därefter av en expertpanel, som bestod av spelutvecklare och erfarna spelare inom Grand Strategy genren.
En encyklopedi i ett fönster inuti spelet blev det förslag som ansågs ha bäst potential. En prototyp av denna encyklopedi implementerades i ett litet spel.
Detta spel hade utvecklats för att efterlikna spel inom Grand Strategy genren.
Både de positiva och de negativa aspekterna efterliknades. Prototypen testades därefter med A/B-testning.
Trots ett lågt antal deltagare visade resultaten att tillgång till information från wiki-sidor inuti spelet ledde till bättre användarvänlighet. Slutsatsen kan även dras att spelare var närmare att nå ett Flow-state medan de spelade, vilket potentiellt kan leda till att spelares intresse bevaras och vill fortsätta spela spelet.
Acknowledgement
First and foremost, I would like to express my deep gratitude to everyone at Paradox Arctic for the warm welcome and inclusion they’ve shown me. I would especially like to thank Mikhail Gofman, Jesper Blomqvist and everyone else on the team that has been my home for the duration of my thesis. They’ve supported me, inspired me and helped me grow, both professionally and as a person, and really made me feel like part of their family. I could not have asked for a better place to write my thesis!
Secondly, I would like to thank Simon Edman and Adam Nording for the great input and suggestions, and lively discussions during our peer reviews, which has significantly increased the quality of this thesis.
I would also like to thank Tomas Nordström, my supervisor at the University, for his academic help on what makes a good thesis and how to write better papers.
Furthermore, I would like to thank my family and friends, who have supported me during these months, either indirectly through words of encouragement, or more directly by participating in my surveys and tests.
And last, but not least, thank you, Caroline, for encouraging me, and helping me see reason, during my most difficult and frustrated moments.
1 Introduction . . . . 6
1.1 Background . . . . 6
1.2 Paradox Interactive & Paradox Arctic . . . . 7
2 Objective . . . . 9
2.1 Purpose . . . . 9
3 Theoretical Framework . . . . 10
3.1 Grand Strategy Games . . . . 10
3.2 Flow . . . . 10
3.3 Game Approachability Principles (GAP) . . . . 12
3.4 Wikis . . . . 13
3.5 Tooltips . . . . 14
4 Methodology . . . . 16
4.1 Pre-Study . . . . 16
4.2 Mock-up exploration . . . . 16
4.2.1 Mock-up 1: Expanding tooltips . . . . 17
4.2.2 Mock-up 2: Fullscreen in-game encyclopedia . . . . 17
4.2.3 Mock-up 3: Windowed in-game encyclopedia . . . . 18
4.2.4 Testing of mock-ups . . . . 18
4.3 Implementation and Evaluation . . . . 18
4.3.1 Pseudo-game . . . . 19
4.3.2 Pseudo-game development . . . . 20
4.3.3 Prototype . . . . 21
4.3.4 User testing . . . . 22
5 Results . . . . 24
5.1 Pre-Study . . . . 24
5.2 Mock-up exploration . . . . 25
5.3 Implementation and Evaluation . . . . 27
6 Discussion . . . . 31
6.1 Limitations . . . . 33
7 Conclusion . . . . 35
7.1 Future Work . . . . 36
8 Appendix . . . . 40
A Pre-study - Survey Questions . . . . 40
B Mock-ups . . . . 42
1 Introduction
The video-game industry is booming [1], and as the market grows, so does its diversity. Today, a myriad of different video-game genres exists, ranging from casual brainteasers, to intense war simulations, and engaging fantasy role playing games. This diversity is not limited to just genres, but also to difficulty. Some players desire difficult videogames that takes years to master, while others prefer games that can be picked up and enjoyed immediately [2].
But many of the game-developers of these tough games also want to invite new players to their games, without scaring them off with how challenging such games are to learn to play. Such an unfortunate scenario is, however, not uncommon, especially in a certain genre of games called Grand Strategy games [3]. These are games in which the player controls a nation and through various complex economic, military, and diplomatic means, seek to expand and help your nation prosper [4]. Such games can prove to be very intimidating to new players, due to their high complexity, which tends to result in poor approachability [3]. As a result, many players, both beginners and veterans, supposedly seek help and guidance from various internet resources when they play these games, but this results in a halt in the playing of the actual game, which has a negative impact on the player experience, according to game developers at a studio named Para- dox Arctic. These developers asked a question. Can the player experience be improved, by instead providing these types of resources inside the actual game experience?
This thesis will be an exploratory study on whether providing such information resources in-game could potentially improve approachability, and as a result, retention of players.
1.1 Background
Nearly a third of the world’s population plays video games, or have at least tried it out, as in 2016 it was estimated that approximately 2.5 billion people play games worldwide [1]. The video game industry is ever-expanding and is slowly but surely out-performing traditional media entertainment like the film industry. Just recently Avengers: Endgame by Marvel Studios set a new record for highest-grossing film of all time (not adjusted for inflation) by earning almost 2.8 billion USD [5]. Compare this to League of Legends, one of the world’s most popular video games, which in the same year earned 1.5 billion USD, a feat which they repeat yearly [6]. In 2019 the total global box office revenue was 42.5 billion USD [7]. The video game industry reportedly earned 120.1 billion USD in the same year [6]. Video games as a form of entertainment have become so prevalent and influential in society and pop culture that vintage video games hold the same prestige that only classic films used to [8].
Just like in the film industry, there are a great many types of different game genres, all of which attracts different types of players. Some prefer the skill ex- pression possible in First Person Shooters where careful precision with a mouse
or control stick is required to show off your mettle. Others prefer the brainteasers of puzzle games. And others still prefer immersing themselves in a compelling story of good and evil, knights and princesses, and dragons and caves in a story heavy Role Playing Game. But some players prefer the strategical mastery of military, economics and politics which are the core elements the genre of games called Grand Strategy. Simply put, a Grand Strategy game is a Real Time Strat- egy game where you as the player control the destiny of a nation or empire trying to take its place in the world [4]. For many the appeal of the games are the his- torical fantasies they have a chance to reenact. What if the Roman Empire never fell? What would have happened to the world if Germany proved unstoppable during World War 2? Or maybe you just want to see your favorite ancient coun- try rule the world. Whatever your motivation, the games have an almost infinite replayability as the games are based on systematic rules and mechanics which challenge the player, rather than a restrictive narrative which would force the player to follow a certain path. You can choose to do anything you please with your nation of choice [4]. This replayability means that the players who enjoy this genre tend to never really stop playing the games. They may play them more or less from time to time, but they always return. Grand Strategy games like Europa Universalis IV [9], Hearts of Iron IV [10], Civilization V [11] and Civilization VI [12] all hold a steady place in the top 50 games by player count on Steam [13]. And they are likely to stay there for a long time.
Grand Strategy games are one of few game genres that require an extensive amount of memorized information, micro-management skills and lots of experi- ence in the chosen game in order to fully experience and enjoy the games. This can be quite an obstacle for new players to overcome. Some might be scared off before they even have a chance to see if they enjoy the game [3].
Is this something that can be remedied? Is there a way to help new players learn, at their own pace, by giving them the tools to look up the information they lack, without forcing them to rely on external help such as online wikis or other third-party resources?
1.2 Paradox Interactive & Paradox Arctic
Paradox Interactive is a Swedish game publisher, with several internal game development studios. The primary studio located in Stockholm, Paradox De- velopment Studio, was founded in 1997 and is one of the world’s most popular strategy game developers. They are responsible for widely known strategy fran- chises, such as Europa Universalis [9], Crusader Kings [14], Hearts of Iron [10]
and Stellaris [15].
Paradox Arctic is a Swedish game development studio. The studio was founded in 2014 and is one of the internal studios owned by Paradox Interactive. Throughout the years the studio has done work on Magicka 2 [16], Pillars of Eternity [17], as well as minor work on all the major Paradox games such as Stellaris [15], Europa Universalis IV [9] and many more. The studio is currently focusing on creating
Deep, Appealing and Personal games in the Management genre, with a flair of the complex strategy of traditional Paradox Interactive games.
2 Objective
In order to investigate whether the initial player experience in Grand Strategy games could be improved, by giving them better access to information on game mechanics and game rules in-game, the following primary and secondary research questions needed to be answered:
Primary
a) Investigate if providing detailed information about game mechanics—in-game, at the players leisure—will improve game approachability.
Secondary
b) Investigate if this access to detailed information could improve player reten- tion
c) Determine the best way to provide this detailed information to the players in a non-intrusive way that does not frustrate or overwhelm the player.
2.1 Purpose
The purpose of the paper was to investigate the possible advantages of providing detailed information on game mechanics in-game, rather than outside the game.
The hypothesis was that it would improve approachability, that is, reducing the gate-keeping of new players by improving the learning curve, which in turn would lead to a higher player retention as frustrated players are more prone to quit the game.
3 Theoretical Framework
In order to investigate the objectives of potentially improving approachability and player retention, information first had to be gathered on relevant subjects;
Grand Strategy games and how they work, how to measure player enjoyment and approachability, and how the resources players currently use works.
3.1 Grand Strategy Games
A Grand Strategy Game, sometimes abbreviated as GSG, is a game, in the strategy genre of games, which focuses on the idea of grand strategy, which is a form of government planning and is an advanced way for a country or nation to plan for fulfilling a long-term goal [18]. In order to reach said goal it is necessary to identify a national objective and assess the resources available to the nation such as “military prowess, political leverage, diplomatic ability, and economic might”[18] and utilize those resources to their fullest potential. In other words, Grand Strategy Games lets the player take control of nation or country, real or fictional, and guide it as they see fit. The player can, among other things, declare war on neighboring nations, expand and develop cities, engage in diplomatic relations with other countries, research technology or build trade routes [4]. The end goal of the game is of the players choosing, as they are in full control of what they want their nation to do, but a common goal is world domination. Grand Strategy Games are often set during historic periods—such as the time of the roman empire, medieval times, or the Victorian era—in which the player is able to bend and shape history to their will [4]. Grand Strategy Games have a lot of overlap with other similar game genres such as the very closely related genre of 4X games—games in which the player takes control of a character or society which has to EXplore, EXpand, EXploit and EXterminate in order to survive in the world[19]—or the tangentially related genre management games—games which focuses on caring for and maintaining a system. For example, The Sims[20], where the player has to care for and help a household and its inhabitants flourish, or Prison Architect[21], where the player takes on the role of prison warden and has to build and maintain a successful maximum security prison. However, while overlap do exist, not all of these games belong to the Grand Strategy Genre, even though the lines which separate them may sometimes be blurry. Some examples of games that do belong to the Grand Strategy Game genre are:
– Europa Universalis IV[9]
– Hearts of Iron IV[10]
– Civilization V & VI[11][12]
3.2 Flow
In his book, Flow: The Psychology of Optimal Experience, from 1991, Mihaly Csikszentmihalyi talks about the psychological phenomenon he refers to as Flow
[22]. Csikszentmihalyi explains that what he refers to as Flow is a mental state in which an individual experiences a strong intrinsic motivation and satisfaction with their current activity. This results in the individual feeling that the activity is so rewarding in and of itself that they desire to do it, without external rewards, even if the activity is highly difficult or dangerous.
While in a state of Flow, the person experiencing the phenomenon loses their sense of reflective self-consciousness and experiences time subjectively different (in the sense that time may feel like it is passing by more quickly. “Time flies when you’re having fun.” [23]).
The phenomenon is generally a positive experience as Csikszentmihalyi explains that experiencing Flow in everyday activities is the key to living a happier life [22]. However, while the subject has not been extensively researched, the Flow state may potentially also inadvertently lead to addiction to the activity the Flow is experienced in, due to the high level of control and rewarding feelings the person experiences.
Video games could be called what Csikszentmihalyi refers to as an autotelic ex- perience; “from two Greek words, auto meaning self, and telos meaning goal.”[22].
The user does not perform the activity as to reach an external goal. The goal is the enjoyment of the activity itself; the main criteria for achieving Flow.
In order to reach Flow, eight criteria needs to be fulfilled according to Csikszent- mihalyi, as compiled by Penelope Sweetser and Peta Wyeth[24]:
– a task that can be completed;
– the ability to concentrate on the task;
– that concentration is possible because the task has clear goals;
– that concentration is possible because the task provides immediate feedback;
– the ability to exercise a sense of control over actions;
– a deep but effortless involvement that removes awareness of the frustrations of everyday life;
– concern for self disappears, but sense of self emerges stronger afterwards;
and
– the sense of the duration of time is altered.
In 2005, Sweetser and Wyeth published a paper on an evaluation model for the enjoyment of video games, utilizing these eight criteria for Flow as heuristics [24]. They call this model GameFlow and showed that it can be used to measure players enjoyment of games.
One of the heuristics is that “challenges in games must match the players’ skill levels”[24] even if the player’s skill increases. If there is a mismatch the player may experience frustration, if the difficulty is too high or boredom if the difficulty
is too low, compared to the player’s skill level. The balance between difficulty and skill is illustrated in Fig.1.
Fig. 1: The relationship between Flow, and user skill and activity difficulty.
Flow is also one of the main reasons players keep playing a chosen game, as con- cluded by Lee and Tsai in their 2010 paper “What Drives People to Continue to Play Online Games? An Extension of Technology Model and Theory of Planned Behavior” [25]. Their research was specifically on Online games, but could, in part, be attributed to other games as well. Aside from Flow, they also state that perceived enjoyment and perceived behavioral control (choosing to go through with an action or behavior, or not, based on whether it is believed it can be done successfully) are important factors in player retention.
3.3 Game Approachability Principles (GAP)
The Game Approachability Principles, abbreviated as GAP, are a set of guide- lines or heuristics proposed by Desurvire and Wiberg, in their study on game approachability [26]. These guidelines are meant to help game developers im- prove approachability in their games, by providing a set of heuristics by which approachability can be measured. Desurvire and Wiberg define Game Approach- ability as the concept of how easily an inexperienced player can learn the tools
and techniques necessary to play a game, by keeping them “sufficiently moti- vated” [26], and in doings so making sure that the player gains enough knowledge to continue playing the game, but keeping the player curious in order to motivate continued interest. They make a point that the game audience is shifting. Play- ers, traditionally, used to consist of solely hardcore gamers. However, in recent times, games have become so mainstream and popular that game developers can no longer expect an expert end-user when designing their products [26]. More often than not the end-user will be a casual or inexperienced player which needs to be taught how to play the game in such a manner that the player does not lose interest due to game being perceived as having a too high skill floor [26][27].
One of the principles proposed by Desurvire and Wiberg is “Information On Demand and In Time, System Thinking” [26], which is a measurement of if the
“player has access to answers regarding the game whenever needed when first coming across new material” [26]. This principle is based on the research by James Paul Gee. In his research article Learning by Design: Good Video Games as Learning Machines from 2005 Gee analyzes how and why those who play games are so intrinsically motivated to learn (in order to be able to play the game) and how it might be applied to traditional learning environments like schools [28]. One of the many principles he defines is the one called “Information
‘On Demand’ and ‘Just in Time’ ”[28]. This principle states that humans are quite inefficient at absorbing verbal information when received out of context and in large amounts. It is preferable to receive the specific information in the situation when it is needed. Gee draws a comparison to Game Manuals and how they “just like Science textbooks, makes little sense if one tries to read them before having played the game”. In other words, without the proper context and timing, the mind is unable to properly absorb verbal information and instruc- tions, as the mind is unable to insert the information in a proper mental model in which the information is supposed to be used. Gee exemplifies this with a comparison to a phenomenon where students who are straight A students in class and during tests are unable to apply what they have learned in practice [28].
3.4 Wikis
A wiki is an internet website which can “be modified or contributed to by users.”
[29] The concept of a wiki was first created by Ward Cunningham, in 1995, when he created a website called WikiWikiWeb, a website which could be edited by its users. The name comes from the Hawaiian word “quick” [29]. The most commonly used wiki is most likely Wikipedia; an online encyclopedia based on the wiki concept of user-generated and user-moderated content [30]. Wikis are often faced with a lot of criticism due to their nature. Many call in to question the fact that anyone can edit the contents of a wiki page and argue that there will always me malicious users who will try to undermine the efforts of the rest of the editors by “introducing false or misleading content” [29]. However, the supporters of wiki software argue that the community as a whole is a stronger
force than the malicious few and as such the community will always correct any false information. And although the system is not in any way foolproof, it stands as a testimony to the goodness of people [29].
In his article, from 2014, Matthew Barr explains that wikis might be one the most significant social media platforms for gamers, because of the scholarly, almost academical, work required to maintain and add on to the sites [31]. This implies that with a large enough dedicated user base of contributors, a wikis articles are likely to reach a very high standard of quality.
However, wikis suffer from a problem called participation inequality [32]; only a small number of users actually contribute to wikis. Sometimes also referred to as the “1% rule”, because of the distribution of users: 1% of users are active key contributors and 9% are intermittent contributors, but 90% of users are lurkers—users who only read and never contribute. This means that in order to have a wiki with up-to-date information, and a high-quality standard of articles, a very large user base is needed.
3.5 Tooltips
A tooltip is an element commonly found in most GUIs (graphical user interfaces) which displays a descriptive text of the element the mouse pointer is currently hovering over, in order to help the user understand a possibly ambiguous icon or button [33].
In order to solve the issue of their players having to rely on a third-party wiki website, Riot Games, developer of the video-game League of Legends, experi- mented with providing the more detailed type of information, that players were looking up on the wiki, through expandable tooltips [35]. The benefit of this was that the non-expanded tooltip could be made more simple in order to provide faster “In the moment” information; information you need to be able to read through and understand quickly [35], as can be seen in Fig.2a. At the same time, the expanded tooltip could provide players who needed more expert-level information the data they wanted, such as number data needed to make deci- sions for min-maxing [36], deciding which strategies to employ, planning on how to approach their next fight ahead of time, etc. [35], which can be seen in Fig.2b.
(a) Regular tooltip.
(b) Expanded tooltip.
Fig. 2: Tooltips for abilities (character skills) as seen in the video-game League of Legends (Riot Games, 2020). Screenshot by author. [34]
4 Methodology
As not much is known on whether players actually use outside resources when they play Grand Strategy games, a pre-study was performed to gauge if players use such resources and how often they do it.
Afterwards, three mock-ups were designed, based on current industry standards for how to supply players with information, such as tooltips and in-game ency- clopedias. These mock-ups were then evaluated by experts on Grand Strategy games. Based on the result of the expert evaluation, one of the mock-ups were selected for implementation and player testing.
The player testing of this implementation was performed in a small game, de- veloped at the same time as the pre-study and mock-up exploration stages. The selected mock-up was implemented as a prototype in this game and then tested on two groups of players using A/B-testing, where one group was a control group without the prototype.
The results of the test were evaluated based on whether the player could complete the game scenario or not, how quickly they did so, how often they utilized the prototype, as well as the testers own thoughts on how well they did and how useful they found the prototype.
4.1 Pre-Study
In order to properly evaluate the potential of improving in-game information sources available to players, the current state of players usage and need of such sources needed to be investigated. A survey was sent out measuring current trends and usage of outside resources—such as website, wikis, YouTube, strategy guides, etc.—during gameplay, as well as general statistics on enjoyment and approachability. The target audience of the survey was primarily gamers, but preferably with a focus on gamers who primarily play Grand Strategy games.
The pre-study survey was published on a private Facebook group of university students, currently enrolled and alumni, in interaction design at Umeå Univer- sity, “ParadoxPlaza”[37], the sub-page dedicated to games published by Paradox Interactive on the social internet platform Reddit [38], as well as “SampleSize”[39]
a sub-page dedicated to surveys on the same website. These platforms were cho- sen as they were most likely to produce participants in the target audience, but still on a broader spectrum of that audience, that is, not only experts of Grand Strategy games, but beginners as well.
4.2 Mock-up exploration
Before implementation and testing of a final prototype, the different ways of giving the player information needed to be evaluated. As it would be impossible to exhaustively test every single method of providing information to the player,
a few prevalent examples from other games were selected for prototyping and user testing.
The methods chosen for prototyping and testing were:
– Tooltip expansion to display additional information – Fullscreen in-game encyclopedia pausing play
(with and without clickable tooltip links) – Windowed in-game encyclopedia during play
(with and without clickable tooltip links)
These approaches were chosen as they are, at the time of writing, the most commonly found solution for similar problems in games of other genres.
The mock-ups were based on the design, layout and theme already present in Stellaris [15], the game which would serve as background for the mock-ups. Basic design was also based on industry standards for encyclopedias and tooltips. All mock-ups were designed in the prototyping tool Figma [40].
All of the mock-ups opted to not make use of the wiki way of having user generated content, as despite the many pros of a wiki website, the cons were simply too great in the context of being used in a paid product, such as a video- game. The fact that it is impossible for profanity and incorrect, and sometimes even malicious, information to not to sometimes slip through was simply to large to ignore.
4.2.1 Mock-up 1: Expanding tooltips
The first mock-up was based on the experiment done by Riot Games on providing players with additional information by allowing them to expand tooltips [35].
The first mock-up incorporates this ability to expand all tooltips to display more information that may be wanted by an expert player looking for detailed data or a beginner looking to learn more about the game mechanic they do not understand.
This mock-up can be seen in Fig.12 and Fig.13 in Appendix B.
4.2.2 Mock-up 2: Fullscreen in-game encyclopedia
The second mock-up incorporates the traditional, and most common, solution for providing large amounts of information in-game: an in-game encyclopedia, also known as a codex, database, or compendium. This encyclopedia is a fullscreen view, which pauses the game to allow for full attention from the user, without making them stressed that they are missing out on things happening in the game.
The second mock-up can be seen in Fig.14 in Appendix B.
4.2.3 Mock-up 3: Windowed in-game encyclopedia
The third mock-up was designed as a variant on mock-up 2. This mock-up also implemented an in-game encyclopedia, but windowed, rather than fullscreen, as to allow the player to keep playing the game while the encyclopedia was open, to allow having a reference page open in the encyclopedia, while following its instructions to achieve a result in the game. For example; In order to build a farm, the player needs to a) choose or create a builder unit, b) move the builder unit to the desired location, c) command the builder unit to build the specified building (the farm).
The reason this idea was chosen for prototyping was due to the results from the pre-study. The pre-study indicated that most players use third-party pages, like wikis, most often during gameplay, rather than before or afterwards. This indicated that players want the information at the same time as they play the game. They do not wish to look up the information afterward. They want to utilize the missing information, in real time, to be able to keep playing the game.
As such, it seemed appropriate to allow the player to access the information, without interrupting them from actually playing of the game.
The third mock-up can be seen in Fig.16 in Appendix B.
4.2.4 Testing of mock-ups
The mock-ups were all evaluated through user testing by experts. The test-group was a small group of experienced users, mostly consisting of game developers and players with a lot of time spent in Grand Strategy Games.
The participants were first shown a typical game scenario and the issue that comes with it. They were then asked how they would like the issue to be solved.
They were then shown the same scenario but featuring one of the three mock- ups. The participants were then asked what they liked about the mock-up, what they disliked about it, if the mock-up solved the problem, and how they generally felt about this solution, giving it a rating from 1-5 in efficiency, satisfaction, and approachability. This was then repeated for the last two mock-ups. Lastly the participants were asked to place the different mock-ups in their order of preference and give a final comment on the test and mock-ups in general.
The order in which the mock-ups were presented to the participant was different for each of the participants to not contaminate the results due to possible bias from the participants based on the order of the mock-ups.
4.3 Implementation and Evaluation
In order to evaluate how well the mock-up, chosen for implementation during the mock-up exploration stage, would improve approachability and enjoyment, the mock-up needed to be evaluated in natura. In other words, it needed to be
tested on a player playing a Grand Strategy game for real. It could not be tested using a mock-up in a lab setting.
This meant that the chosen mock-up needed to be implemented as an actual prototype in a Grand Strategy Game. However, learning to work with, and implement into, the game engine used by the Grand Strategy games developed by Paradox Interactive would be nigh impossible, given the timespan of the research project. As such, a small game that as closely as possible emulated a Grand Strategy game needed to be developed, to be used as a platform for testing the prototype. This was done over the course of the entire research project.
The prototype was then evaluated, quantitatively, through participants play- ing the game once—unsupervised, at a time of their choosing, in their choice of location—and then filling out a survey and attaching the save file of their playthrough to their response. The save file contained a data log on what the participant did, and at what times, measuring milestones and the usage of the prototype.
4.3.1 Pseudo-game
Over the course of the project a pseudo-game was developed in Unity, a game engine, in order to provide a platform for the final implementation and the final tests of the research, as implementing the prototype in a real Grand Strat- egy Game would be too time-consuming. These games often have very complex systems and require a lot of time learning the engine in order to successfully implement solutions in them.
This pseudo-game was instead developed in order to mimic such a game, but pro- vide an easier platform for implementation, as the systems and engine would not be as complex. The pseudo-game was based on the open-source project Mostly Civilized Hex Engine [41], created by Github and Youtube user Quill18 [42].
The intention was for the pseudo-game to simulate the experience of a Grand Strategy Game, in order to evaluate the difference in what a player may expe- rience when playing such a game with or without the systems proposed by this research. Therefore, the pseudo-game was developed to provide an experience as close to a Grand Strategy Game as possible in the given development time.
Due to various limitations and design choices the pseudo-game also shares a lot of similarities with a 4X game (EXplore, EXpand, EXploit and EXterminate).
However, this was not a problem as many Grand Strategy Games belong to the 4X games genre as well, due to the overlap in the type of gameplay typical for the two genres. The pseudo-game should therefore, despite skirting the genre of 4X games, be considered a very valid platform for the research. A screenshot from the game can be seen in Fig.3.
Fig. 3: Screenshot of the finished pseudo-game.
4.3.2 Pseudo-game development
The pseudo-game was, as previously mentioned, based on the open-source project Mostly Civilized Hex Engine. Initially the pseudo-game was developed very closely to the development tutorial for the project, but as the project got farther and farther into the development, the design choices wanted for the pseudo-game was diverging from the open-source project, and eventually the pseudo-games development was completely separate from the Mostly Civilized Hex Engine project.
The big components shared between the two projects were: a) a hex-grid based map, b) unit logic, c) unit pathfinding, and d) the mouse controls.
The key mechanics needed for the pseudo-game, which differs from, or expands upon, the Mostly Civilized Hex Engine, were: a) real-time gameplay (as opposed to turn-based), b) different nations visualized as different colors on the map, c) various systems and interfaces usually present in Grand Strategy games (such as diplomacy between nations, management of settlements, etc.)
The final version of the pseudo-game consisted of a gameplay experience, with a gameplay time of a maximum of 30 minutes, that simulated a small part of playing a Grand Strategy or 4X game. The player goal in the pseudo-game was to invade an enemy nation and claim one of their cities as their own. Sub-goals included, but were not limited to; building farms to produce food, recruiting armies (which cost food to recruit), make claims on foreign territory, declaring war on the enemy nation, and sending your armies into battle against the enemy.
The faster a player could succeed in the goal of the game, the more proficient at the game they were considered to have become.
4.3.3 Prototype
Based on the results of the mock-up exploration phase, Mock-up 3 was chosen as the best candidate for implementation as a prototype. However, with a few alterations. As suggested by testers of the mock-ups, the navigation of the win- dowed in-game encyclopedia was simplified, by adding a tab-system to the side of the window. This allowed for faster and more understandable navigation of the different categories in the encyclopedia. Icons were also added as image recog- nition allowed for easier digestion of the different categories and their content.
Finally, tooltips were kept, but in the more traditional simple form, rather than the expanding versions tested with Mock-up 1. However, these tooltips contained highlighted clickable words, similar to hyperlinks on web pages. Clicking on one of these highlighted words would immediately open the in-game encyclopedia on the corresponding page to the link clicked. This solved a concern testers had expressed regarding the Mock-up 3, as they had liked the possibility of reading while playing, but had found the traversal of the tree of categories confusing and time-consuming. This prototype was then implemented in the pseudo-game.
This implementation can be seen in Fig.4.
Fig. 4: The Windowed Encyclopedia prototype, here called Codex, implemented in the pseudo-game.
4.3.4 User testing
A final test of the prototype, as implemented in the pseudo-game, was done on a group of players of mixed skill and experience.
The test was performed using A/B-testing, comparing two groups of players;
one group which played through the pseudo-game scenario, with the prototype, the windowed in-game encyclopedia, available to help them reach their goal, and one control group where players had to learn how the game worked without help from the in-game encyclopedia.
The test was done without supervision, on the test participants own computers, in the comfort of their own home (or other personal choice of location), as online supervision likely would be extremely difficult, and physically meeting the participants would not be prudent as, at the time of research, a pandemic was spreading around the world.
At the start of the game, the participant was randomly selected for one of the test groups. The participant was then given a short explanation of the game, its basic rules and mechanics, and the goal of the game. They were also informed that the game had a time limit of 30 minutes, after which they would fail at completing the main objective of the game, and as such lose the game. This was done to help the players stay focused on the task of completing the goal as fast as possible.
This was desired, as the main measurement of success would be how fast a par- ticipant was able to complete the game in a satisfactory manner. The hypothesis was that if the testers, which belonged to the group which had the prototype to help them, on average performed better in the game, than the control group, then the addition of the prototype would indicate that the easy access to infor- mation allowed for players to more quickly learn the game. However, using how quickly a player won the game, as the only unit of measurement, was likely to not result in accurate data, as a huge number of participants would be required to fully eliminate the differences in experience with similar games the differ- ent participants may have. A more experienced participant would after all be more likely to quickly figure out how to play the game. As such, and additional requirement was added. Instead of only measuring how quickly a player could win the game, a second unit of measurement was added. The number of times the encyclopedia was opened on a relevant page, prior to the completion of one of the steps necessary to progress in the game. In other words, if a correlation between reading a relevant encyclopedia page and winning more quickly could also be found, then the hypothesis was likely to be correct.
Finally, in order to also measure how the participants themselves viewed their efforts and level of success in the game, the participants were asked to fill out a survey after their first playthrough of the game. In this survey the participants had to answer questions regarding their enjoyment of the game, their level of commitment to fulfill the goal of the game, if they felt the game frustrating or
difficult to learn, and whether the in-game encyclopedia had been of use to them.
The last question only being present if the participant belonged to the group which had the encyclopedia available.
This results of this survey could help indicate if a participant, despite poor re- sults, still enjoyed the game and was helped by the prototype in a satisfactory manner, or if in the case of an opposite experience, a participant quickly suc- ceeded but felt that the help had been lacking or had not enjoyed the game. This third level of measurement would help further steer the test results to indicate whether the participants had felt that the game was approachable or not.
5 Results
The results of the pre-study survey, which measured player tendencies to use outside resources while playing Grand Strategy games, indicated that a majority of players shift window focus to read wikis primarily while playing the game, as opposed to before or afterwards. Many of the participants also answered that they do not have access to all the in-game information they need when playing such games.
The expert evaluation of the three different mock-ups showed that both tooltips and windowed in-game encyclopedias are plausible alternatives for displaying detailed information, traditionally found on wikis, in a player-friendly and use- ful manner. The windowed encyclopedia, however, showed better potential for scaling, as tooltips were likely to grow too big, if the information they would contain was too great. The evaluation also showed that it is important to have accessible and easy to use navigation for such encyclopedias, as otherwise the players curiosity might be dampened which might have a negative impact on the players experience.
The user test of the implementation of the windowed encyclopedia suggested that having the encyclopedia as a resource in Grand Strategy games leads to lower levels of confusion and frustration for players, which in turn seems to lead to players performing better at the game, as they have a higher rate of victory and progress further in the game faster. Player perception of having access to the information they need also increased with access to the encyclopedia.
5.1 Pre-Study
The pre-study survey, on players current usage of outside resources while playing Grand Strategy games, were answered by 107 participants, from all ages between 14-51, the majority of which identified as male. Two thirds of all participants spend between 1-4 hours per day playing videogames and, on average, 51-60% of that time is spent on Grand Strategy games in particular. Worth to note is that none of the participants spent 0% of their time on Grand Strategy games, so no complete beginners answered the survey. The most commonly played games (played by at least 50% of the participants) were Sid Meier’s Civilization V &
VI, Crusader Kings II, Hearts of Iron IV, Europa Universalis, Victoria II and Stellaris.
A clear majority of players completely agreed that they currently enjoyed playing Grand Strategy games, and the data is almost identical for their prediction on if they would continue to enjoy playing such games in the future. However, when asked about their enjoyment during the first 6 hours of playing such a game, while many still enjoyed it immensely, there were a small portion of players who did not necessarily enjoy themselves during this initial time.
When asked whether the participant considered Grand Strategy games to be easy to play (in a difficulty sense), most answered that they thought it was neither
very easy, nor very difficult, however the number of participants who thought the games were at least slightly difficult were more significant than those who thought they were easy. A majority of participants also thought that Grand Strategy games are difficult to learn to play (in the approachability sense). As seen in Fig.5, a large majority of the participants felt that while playing they did not always have access to the information they desire or need.
Fig. 5: Pre-study - Player responses to the statement:
I always have all the in-game information I need when playing Grand Strategy.
(1: Completely Disagree, 5: Completely Agree)
On the topic of how often the participants shift window focus while playing—an action also known as alt-tabbing—a majority answered that they often shift focus in order to look up information they need. When asked on their usage of wikis, in relation to playing Grand Strategy games, no consensus could be reached on preference for reading wikis before playing games, but a larger portion of the participants did answer that they use wikis after playing the game. However, a strong majority of players reported that they use wikis in the middle of playing Grand Strategy games, as can be seen in Fig.6.
Finally, when asked which specific internet resources they use when playing Grand Strategy games, almost 90% of the participants answered that they use wikis. 71% used YouTube, and 40% used strategy guides (printed or online).
The survey questions in their entirety can be found in Appendix A.
5.2 Mock-up exploration
The expert evaluation of the three different mock-ups was performed together with six participants. All participants were male, with an average age of 31. Many of the participants were employed as video-game developers. The participants
Fig. 6: Pre-study - Player responses to the statement:
I use wikis related to the game I’m playing WHILE playing Grand Strategy.
(1: Completely Disagree, 5: Completely Agree)
all had a high level of experience with videogames in general and a mid-to-high level of experience with Grand Strategy Games in particular. The six participants were shown the mock-ups in different orders to eliminate any potential order bias.
The participants had to give the different mock-ups a rating of 1-5, where 1 is bad and 5 is good, in three different categories: Effectiveness, Enjoyment, and Accessibility. That is, how effectively this mock-up helped the player understand more about the game, if this type of design was something they personally liked, and whether the design was easy to understand and use. The experts were then asked to rate all the mock-ups in order of preference, after having seen them all.
Those ratings were used to calculate an aggregate rating. The results were, on average:
Category Mock-up 1
Expanding Tooltips
Mock-up 2 Fullscreen encyclopedia
Mock-up 3 Windowed encyclopedia
Effectiveness 4.1 3.8 4.1
Enjoyment 4.1 3.5 4.0
Accessibility 4.5 4.3 3.9
Aggregate rating 3.0 2.3 4.0
Total 15.7 13.9 16.0
(1: Very bad, 3: Neutral, 5: Very good)
Apart from these metrics, the expert evaluation consisted first and foremost of an interview.
In this interview, among other comments, some of the experts expressed concern regarding the Windowed encyclopedia mock-ups current form of navigating be- tween pages, and stated that if the navigation were better, they would have rated
said mock-up a lot higher. The problematic navigation can be seen in Fig.16 in Appendix B.
Other comments included, but were not limited to: Suggestions for icons to be included in navigation trees for the encyclopedias to better be able to quickly recognize categories; the desire to have all three mock-ups implemented, in order to facilitate different needs; concern that tooltips do not allow you to deep-dive and read more about other related subjects, if you are feeling curious; concern that the Fullscreen encyclopedia interrupted gameplay too much.
In conclusion, Expanding Tooltips and the Windowed encyclopedia were both very viable candidates, but provided the concerns in the comments could be addressed, the Windowed encyclopedia was deemed the best candidate for im- plementation.
5.3 Implementation and Evaluation
After implementation in the pseudo-game, the Windowed encyclopedia proto- type was tested. The quantitative test was performed on two groups. One group of 8, which had access to the Windowed encyclopedia, which we will call Group A, and one control group of 8, without it, which we will call Group B, for a total of 16 participants.
Group A: Had Windowed encyclopedia
Group B: Did not have Windowed encyclopedia
In Group A, 88% of the participants were male, and 12% were female. In Group B, there was an equal many male and female participants, in other words, 50%- 50%. Group A’s participants had an average age of 34, while Group B had an average age of 26. In Group A, the participants played games, on average, 2:10 hours per day. Group B played games 1:50 hours per day. Finally, on a scale from 1-5, where 1 is never and 5 is always, on how often the participants played Grand Strategy or 4X Games, Group A answered 2.4, while Group B answered 2.3. In other words, Group A was likely slightly more accustomed to playing games similar to the pseudo-game, than Group B.
After these initial general questions, the participants had to rate their experi- ences playing the scenario of the pseudo-game from 1-5, where 1 is Disagree Completely, 3 is Neither Agree nor Disagree, and 5 is Agree Completely. The results were:
Question Group A Group B
The game was fun to play. 3.8 3.6
The game was frustrating. 3.1 3.5
The game was difficult to learn. 3.3 3.0
I performed well in the game. 3.6 3.4
I put a lot of effort into winning the game.
3.6 2.3
As can be seen in Fig.7, players in Group A seemed to think the game was less frustrating, than players in the control group.
Fig. 7: Results - Levels of frustration for participants in both groups and corresponding predicted normal distributions
(Group A: With Codex, Group B: Without Codex)
Afterwards, the participants were asked to, in the same way, rate their experi- ences with the Tooltips, Windowed encyclopedia and access to information. (The participants in Group B did not receive the questions related to the Windowed encyclopedia.)
Question Group A Group B
I was often confused on what to do next.
2.6 3.6
I had access to the information I needed while playing.
3.4 2.4
The tooltips were useful. 3.4 3.1
I read the tooltips often. 3.1 3.3
The Codex (Windowed Encyclopedia) was useful.
4.1 I used the Codex (Windowed
Encyclopedia) often.
3.6
As can be seen in Fig.8, players in Group A also seemed to think the game was less confusing, than players in Group B, just like in the case of frustration.
Players in Group A were likewise more satisfied with the amount of information they had access to, than the control group players, as can be seen in Fig. 9.
Fig. 8: Results - Levels of confusion for participants in both groups and corresponding predicted normal distributions)
(Group A: With Codex, Group B: Without Codex)
Finally, the participants had to upload a save-file of their playthrough of the game. This save-file contained a log of events as they occurred, while the partic- ipant was playing the game. From this save-file, whether the player won or lost the game, and how long it took them to complete the game, was extracted.
The results were as follows:
Data Group A Group B
Win-ratio 88% 50%
Time to complete 00:20:19 00:25:48
The time to complete had a limit of 30 minutes, after which a player would lose the game. Both groups individual times were also tested with a statistical analysis called a “Two sample t-test”. A two sample t-test is a statistical test that can be used to determine whether two different groups of samples has a different mean-value. In other words, it would determine if Group A’s lower average time of completion, 20 minutes and 19 seconds, compared to Group B’s, 25 minutes 48 seconds, were due to statistically significant differences, or whether it was likely to be a result of random chance.
The result of the two sample t-test was that, with a significance level of 0.10 (90% confidence), it was determined that the two groups differences in time of completion were statistically significant. The result was, in other words, not due to random chance.
The differences in time of completion and their calculated trends can be seen in Fig.10.
Fig. 9: Results - Satisfaction of amount of information available for participants in both groups and corresponding predicted normal distributions
(Group A: With Codex, Group B: Without Codex)
Fig. 10: Results - Time of completion for participants in both groups and corresponding linear trends (predicted averages)
(Group A: With Codex, Group B: Without Codex)
6 Discussion
The results of the pre-study indicated that most players that use third-party resources, especially wikis, often did so during gameplay, rather than before or after. This likely meant that players felt the need for such information during gameplay and did not want to proceed with their play-session unless they could find the answer to the question they were seeking.
This was the basis for which the different types of mock-ups were chosen, as stated in the Mock-up exploration subsection of the Methodology section. The three different mock-ups chosen could represent three different levels of “depth of content” vs “ease of access”. The first mock-up, expanding tooltips, had the highest level of “ease of access” as they had very little interruption of gameplay when the player wanted to read them. However, due to the nature of the small size of a tooltip, it could not contain much information and therefore had a low level of
“depth of content”. The second mock-up, fullscreen in-game encyclopedia, instead had the highest level of “depth of content” as it quite closely could emulate the nature and functionality of a wiki page. It could contain large amounts of information, navigation, and further reading. However, it instead suffered from the same problem as a real wiki page. It completely interrupted the gameplay.
As such, it had a very low level of “ease of access”. Finally, the third mock-up, a windowed in-game encyclopedia, tried to take the best of both worlds, by allowing for greater amounts of information than a tooltip, but still was small enough of a window to allow the player to continue to play the game, while using the encyclopedia at the same time. This mock-up had a medium level of both
“ease of access” and “depth of content”.
The results of the expert evaluation of these three mock-ups at first indicated that players would prefer both expanding tooltips and the windowed encyclope- dia over the fullscreen encyclopedia. However, upon further reading and analysis of the comments of the participants, while many liked the expanding tooltips, just as many of the participants liked the windowed encyclopedia, despite it hav- ing many more shortcomings, many of which were quite easily fixed. As such, the decision was made to move forward with the windowed encyclopedia, rather than the expanding tooltips, provided that the shortcomings of the design could be remedied.
Many of the test participants expressed that they really liked the windowed encyclopedia, but did not like the provided way of navigating it. Many expressed concern that they would not use it, despite its usefulness, as navigating the tree of categories was cumbersome, and detracted from gameplay, the one thing that this mock-up was trying to minimize. However, the same participants said that they wished they could quickly access the encyclopedia by immediately jumping to the page they wanted to read, without having to rely on navigating the navigation tree. These comments gave birth to the idea of clickable links in tooltips, which could open the correlating page in the encyclopedia.
