Linköpings universitet SE–581 83 Linköping
Linköping University | Department of Computer and Information Science
Master thesis, 30 ECTS | Software Engineering
2020 | LIU-IDA/LITH-EX-A--20/002--SE
Engaging novice
programmers through
gamification
–
Puzzles to teach and motivate programming learning
Simon Delvert
Johan Nåtoft
Supervisor : Erik Berglund Examiner : Rita Kovordanyi
Upphovsrätt
Detta dokument hålls tillgängligt på Internet – eller dess framtida ersättare – under 25 år från publiceringsdatum under förutsättning att inga extraordinära omständigheter uppstår. Tillgång till dokumentet innebär tillstånd för var och en att läsa, ladda ner, skriva ut enstaka kopior för enskilt bruk och att använda det oförändrat för ickekommersiell forskning och för undervisning. Överföring av upphovsrätten vid en senare tidpunkt kan inte upphäva detta tillstånd. All annan användning av dokumentet kräver upphovsmannens medgivande. För att garantera äktheten, säkerheten och tillgängligheten finns lösningar av teknisk och admin-istrativ art. Upphovsmannens ideella rätt innefattar rätt att bli nämnd som upphovsman i den omfattning som god sed kräver vid användning av dokumentet på ovan beskrivna sätt samt skydd mot att dokumentet ändras eller presenteras i sådan form eller i sådant sam-manhang som är kränkande för upphovsmannens litterära eller konstnärliga anseende eller egenart. För ytterligare information om Linköping University Electronic Press se förlagets hemsida http://www.ep.liu.se/.
Copyright
The publishers will keep this document online on the Internet – or its possible replacement – for a period of 25 years starting from the date of publication barring exceptional circum-stances. The online availability of the document implies permanent permission for anyone to read, to download, or to print out single copies for his/hers own use and to use it unchanged for non-commercial research and educational purpose. Subsequent transfers of copyright cannot revoke this permission. All other uses of the document are conditional upon the con-sent of the copyright owner. The publisher has taken technical and administrative measures to assure authenticity, security and accessibility. According to intellectual property law the author has the right to be mentioned when his/her work is accessed as described above and to be protected against infringement. For additional information about the Linköping Uni-versity Electronic Press and its procedures for publication and for assurance of document integrity, please refer to its www home page: http://www.ep.liu.se/.
©Simon Delvert Johan Nåtoft
Abstract
With an increasing number of primary schools having programming as a mandatory sub-ject, students and teachers face new challenges by having to teach and learn a subject that is often difficult to engage students in. Many tools exist that introduce the problem solving aspect of programming, using a programming language not used in actual software. The programming languages used in these tools are often easier to understand and therefore engage the students more. In this study, an application teaching programming in Python is proposed while still engaging the students by including gamification elements. By hav-ing two groups of users, one group with gamification elements available in the software, and the other with them unavailable, the engagement could be compared between the two groups. With the data collected from Google Analytics, completion rate and surveys, it was shown that users with gamification elements available are slightly more engaged. Through the results gathered from this study, the use of gamification elements in this type of soft-ware can have an impact on user engagement and should therefore be considered when building similar types of systems.
Acknowledgments
We would like to thank our supervisor Erik Berglund for assisting us throughout the entire thesis. His encouraging words always helped us when we were unsure on how to proceed. We would also like to thank our examiner Rita Kovordanyi.
A special thanks to our families and friends that has helped and believed in us through-out our years at Linköping University. They never gave us up, they never let us down, they never ran around or deserted us. They never made us cry, they never said goodbye, they never told us lies or hurt us.
Contents
Abstract iii
Acknowledgments iv
Contents v
List of Figures vii
List of Tables viii
1 Introduction 1 1.1 CoderQuiz . . . 1 1.2 Motivation . . . 1 1.3 Research question . . . 2 1.4 Delimitations . . . 2 1.5 Expected results . . . 2
2 Theory & Related Work 4 2.1 Difficulties in learning programming . . . 4
2.2 User engagement . . . 5
2.3 Google Analytics . . . 9
2.4 Gamification . . . 9
2.5 Learning computer science through puzzles . . . 10
2.6 Feedback . . . 11
3 Method 12 3.1 Planned method for this study . . . 12
3.2 Implementing tracing quiz . . . 12
3.3 Implementing gamification . . . 13
3.4 Implementing feedback . . . 13
3.5 Code descriptions . . . 13
3.6 User engagement . . . 14
4 Results 18 4.1 The system - CoderQuiz . . . 18
4.2 User engagement . . . 26
5 Discussion 31 5.1 Results . . . 31
5.2 Method . . . 32
5.3 The work in a wider context . . . 34
6 Conclusion 35 6.1 Future work . . . 35
Appendices 37
A Appendix A 38
A.1 The User Engagement Scale . . . 38
Appendices 38
List of Figures
2.1 Model of engagement . . . 6
4.1 Feedback in the placeholder quiz . . . 19
4.2 Feedback in the selection quiz . . . 20
4.3 Feedback in the tracing quiz . . . 21
4.4 Code description dropdown-menu . . . 21
4.5 Code description modal . . . 22
4.6 Tracing quiz . . . 23
4.7 Progress bar showing percentage of assignments completed . . . 23
4.8 Levels in CoderQuiz . . . 24
4.9 Button unlocking the next level . . . 24
4.10 The storytelling robot, created by Erik Berglund . . . 25
4.11 Story modal from the robot . . . 25
List of Tables
3.1 Completed assignments categorization . . . 16
3.2 Survey scoring scale . . . 17
4.1 Variables used to calculate the indices . . . 27
4.2 Engagement indices . . . 27
4.3 Engagement score . . . 28
4.4 A-group survey result . . . 28
4.5 B-group survey result . . . 29
4.6 The average result of the survey for both groups . . . 29
4.7 Amount of users that completed assignments . . . 29
4.8 Percentage of users that completed assignments . . . 30
1
Introduction
Programming is being taught in more and more primary schools and the teachers and students therefore require tools that teach programming in an accessible way [1]. The web application CoderQuiz has been created with this purpose in mind. It consists of puzzles and quizzes designed to give students an introduction to programming without having to write or compile their own code. In the application the users are given a task that they have to solve by looking at a segment of code and to also understand the purpose or the specifics of the short program. This thesis will investigate how adding gamification elements in this type of application, can affect the users’ engagement.
1.1
CoderQuiz
CoderQuiz is a web application that contains puzzles and quizzes, with the purpose to teach the basic concepts of programming. The puzzles and quizzes are visual and interactive challenges that aim to teach the students in a way that is friendlier towards novice programmers. Without having to write and execute your own code, a user is introduced to several examples of code which the user have to understand in order to solve the challenge. The goal in one of the challenges, is to choose between different types of code snippets that should fill the white spaces in the code examples. In another type of challenge, the goal is to point out the correct element(s) in the code, depending on the question. The codes and quizzes are represented in the programming language Python 3. The application was created by Erik Berglund and Dennis Persson at Linköping University.
1.2
Motivation
Starting to learn programming is a frustrating experience for many, and some concepts have been proven to be particularly difficult for beginners to understand. For example, the notion of lists and for-loops which might seem trivial to the average programmer, are a few of those concepts that can be perplexing to many beginners [2]. It has also been shown that students with no prior programming experience, often lose motivation early on because of assignments that are too complex [3]. When having negative experiences with programming
1.3. Research question
in the past, it can lead to even worse performance and emotions later on [4]. To not lose motivation early on, it is therefore important to be able to teach the more difficult concepts in a way that builds confidence and motivation. Students have even expressed a desire for the use of example programs and interactive visualizations as a means of educational material when learning programming [5]. It would therefore be interesting to examine how motivation could be influenced by a web application like CoderQuiz, and how to keep users engaged throughout the use of the application.
1.3
Research question
By using the definition of the type of application CoderQuiz is, in Section 1.1, this thesis will seek to answer the following question:
How does the addition of gamification elements affect the engagement of users learning programming through CoderQuiz?
This question seeks to answer how CoderQuiz can be further developed so that users keep using the software, and do not quit or give up when the questions get more difficult, by using design techniques from games. Engagement refers to how involved the users are when using the application and the aspects that keeps them using the application. This is for example how much time users spend in the application during a session, how many clicks they perform and how often users return to the application. Engagement is also defined by how rewarding users find the experience and how attentive they are throughout the use of the application, and all of these aspects will be measured in order to calculate the user engagement.
The research question will be answered by developing two versions of a web application. One version will include added gamification elements that aims to increase engagement, while the other version will not include any such elements. The elements purpose is to give the users a sense of pride and accomplishment when completing challenges and giving them motivation to keep on using the software. Examples of such elements are high-score, leader-board and achievements. Problems with engaging students in other e-learning systems will also be observed and taken into account when developing new assignments for CoderQuiz, and the result will be observed by collecting data from the system and the users from both versions of the website.
1.4
Delimitations
The measurements taken are mainly from students taking an introductory course to programming in university. This means that the data collected is from students that are 18 years or older. Whether or not the conclusion will be applicable to students in primary school will not be examined within this thesis.
Because the result is based on our specific system, the web application CoderQuiz, some parts of the result will be difficult to replicate and might not be applicable to other systems.
1.5
Expected results
The difficulties of learning programming is often linked to a steep learning curve. Writing your own code without seeing any code examples before has been found to be a complex task for many beginners [5]. In CoderQuiz users get to see a lot of code examples and get to answer questions about the code’s elements without having to write or execute code by
1.5. Expected results
yourself. Therefore, users are expected to be engaged while using CoderQuiz, and especially engaged when using the version of the website that has gamification elements visible.
2
Theory & Related Work
In this chapter we present the theory that is relevant to understand the thesis.
2.1
Difficulties in learning programming
For students with little to no prior programming experience, some concepts have been found to be important to focus at. Lahtinen et al. organized an international survey for more than 500 students and teachers, which provided information about the difficulties that novice programmers experience [5]. This was done in order to provide a basis for recommendations for creating new learning material and approaches. Below are some of the concepts that the study found were difficult to understand for the students.
Loops
Novices often fail to understand what is happening during the loop, such as variables being updated.
Tracing through code
Wiedenbeck suggests that novices should focus on continuous practice with basic materials until they have familiarized themselves to the basic concepts [6]. This is because students often have trouble thinking about how to solve the problem when they do not fully understand the tools of programming. Thus being familiarized to code by "manually" executing it, might then free the mind to solve the actual problem. It has also been found by the survey organized by Lahtinen et al. that students find it very helpful to read example programs [5].
Finding bugs
Knowing what information to extract from an error message and how to process it, has been proven to be difficult for many novice programmers.
Recursion
Because recursion is an abstract concept, it is also cognitively complex to understand without a real world example to compare the recursion to.
2.2. User engagement
The main problem for novice programmers is not to understand these concepts, but to actually apply them. That is why Robins et al. suggests that the focus should be on a combination and use of these concepts [2].
2.2
User engagement
In order to define user engagement and understand what attributes that has an effect on it, this chapter provides a proposed model of user engagement in software and also a proposed way to measure it.
2.2.1
Model of user engagement
By performing an extensive literature study and exploratory study of users of many different services, O’Brien et al. attempts to deconstruct and define the term engagement as it applies to users’ experience with software [7]. They propose a model to present the flow of how the engagement changes throughout the use of an application, this model is illustrated in Figure 2.1. The model contains these three stages.
• The point of engagement is initiated by the presentation of the user interface, the users’ interest in the topic and the understanding that the users have enough time to use the application. The attributes that affect users to start getting engaged in an application are listed in Figure 2.1.
• The engagement of the users is sustained when their attention is kept throughout the usage of the application, and this is done by providing positive emotions to the users. Users desire an interface that is customizable, so that it meets their personal needs. They also want to be completely immersed in the application at times, and have social interaction with other users at times. To get direct and clear feedback from the application is also desired from the users. The attributes that affect how users are kept engaged in the application are listed in Figure 2.1.
• Users can disengage for several reasons, for example, that the challenge is either too difficult or easy, or because of distractions in their environment. The attributes that affect how users get disengaged are listed in Figure 2.1.
2.2. User engagement
Proposed Model of Engagement
Point of Engagement Engagement Disengagement
Re-engagement
Point of Engagement Attributes ● Aesthetics ● Novelty ● Interest ● Motivation ● Specific or Experiential Goal
Period of Engagement Attributes ● Aesthetics and Sensory Appeal ● Attention ● Awareness ● Control ● Interactivity ● Novelty ● Challenge ● Feedback ● Interest ● Positive Affect Disengagement Attributes ● Usability ● Challenge ● Positive Affect ● Negative Affect ● Perceived Time ● Interruptions
Figure 2.1: Model of engagement
2.2.2
Basic metrics of engagement
There are five different measures that are often used when talking about user engagement. According to Carrabis and Peterson, three of them, session duration, page views per session and visits per visitor, are reasonable proxies for engagement as long as their limitations are considered [8]. The other two measures used, conversion and satisfaction, are often said to be proxies for engagement but according to Carrabis and Peterson they are complementary metrics that do not have a strong relationship with actual engagement. All five of these metrics have weaknesses when evaluating them individually, and these weaknesses are also explained by Carrabis and Peterson.
Session duration
Session duration is the time that a user spends browsing the website on one occasion. It is calculated by taking the difference of the first and last time of interaction with the website during a session. What they are doing during this time is not taken into consideration, so this value can therefore be misleading. For example, if a user enters the website and immediately get distracted for a long while, the user have not been engaged with the website during this time. Session duration is however useful when used broadly, for example when looking at the percentage of sessions with duration under a certain amount of seconds.
Page views per session
Page views per session is calculated by dividing the number of page views by the number of sessions in a reporting period. Just like session duration, page views per session is a good general indicator which also have limitations. If a user goes to a website to give feedback or send an email it might take one or two page views to get there. This user has shown a lot of engagement compared to a user that, for example, have 30 page views but never returns.
2.2. User engagement
Visits per visitor
Visits per visitor is most commonly used to get insight to the loyalty, frequency and recency of a visitor over time. It is calculated by counting the times that a visitor has visited the website. Obviously, this metric does not take the length of the visit into account and can therefore be misleading when looking at engagement. The metric can however be improved upon by taking the frequency of visitation or recency of visit into account.
Conversion rate
Conversion rate means the rate of users that have completed a target action. For example, if a website has 100 visitors and receives 10 sales in one months time, the conversion rate would be 10 divided by 100, or 10%. While many consider conversion rate to be the ultimate measure of engagement, it also has its limitations. Depending on what target actions are defined, engaged users can appear like they are not. For example, imagine subscribing to the news feed is defined as a target action. Then, if a user subscribe to the news feed of a website, and return to the site 99 times for the next year, the conversion rate for the blog subscription will be no more than 1%.
Customer satisfaction
Customer satisfaction is measured by sending out satisfaction surveys to the users. These surveys are designed to collect information about the current relationship the user has with the website. However, a poorly satisfied user is not necessarily unengaged. If a user has a problem and spends a lot of time trying to find a contact form to send an angry message, the user is still highly engaged.
2.2.3
Measuring user engagement
Web Analytics Demystified has with support from Mr. Carrabis, created a framework that can be used to measure user engagement in software [8]. The framework uses Peterson’s definition of engagement, first proposed in December 2006:
"Engagement is an estimate of the degree and depth of visitor interaction on the site against a clearly defined set of goals."
By this definition, Web Analytics Demystified and Mr. Carrabis created a formula (2.1) to give an estimate of engagement.
ř
(Ci+Di+Ri+Li+Bi+Fi+Ii)
7 (2.1)
What this formula says is that user engagement is a function of the number of clicks (Ci), the visit duration (Di), the rate at which the users return to the site over time (Ri), their loyalty to the website (Li), their awareness of the brand (Bi), their inclination to contribute feedback (Fi) and the probability that they will engage in activities on the site designed to increase awareness and create a lasting impression (Ii) [8]. The result of this calculation is a percentage score of user engagement and is calculated individually for each user. The average for all users is then calculated for each index and the overall engagement score.
Click depth index (Ci)
Captures the amount of page views. In order to remove noise caused by visitors bouncing on and off after viewing only a few pages, a threshold of pages viewed is decided. The amount of sessions with that or higher amount of page views is then divided by all sessions. The threshold can be decided by analyzing how many page views it takes to do a core action within the website. For example if it takes 4 page views to make a buy, the threshold T is set to 4.
Ci= Sessions having at least “T page views”
2.2. User engagement
Duration index (Di)
Captures the time spent on a website. To calculate this index, a threshold T needs to be decided, for the time taken that is thought needed for a user to be engaged with the website. To calculate the duration index, the amount of sessions above the threshold is divided by all sessions.
Di= Sessions where “duration over T min”
All sessions (2.3)
Recency index (Ri)
Captures the rate at which the user returns to the website over time. Recency is, according to Jim Novo, “the number one most powerful predictor of future behavior” and he also argues that “the more recently a customer has done something, the more likely they are to do it again” [9]. There are two versions to define this index, one newer and one older method.
Newer method
The recency index is calculated by simply taking the inverse of the number of days elapsed since the most recent session.
Ri= 1
Number of days elapsed since the most recent session (2.4)
Older method
To calculate the recency index with the old method, the amount of sessions having more than T page views that happened during the past T weeks, is divided by all sessions [10].
Ri= Sessions having at least “T page views during the last T weeks”
All sessions (2.5)
Brand index (Bi)
Captures the user’s awareness of the brand or website. What needs to be done first to calculate this index, is to come up with a list of branded terms associated with your web page, for Apple this could be for example:
Apple, iPod, iPhone, iMac, Steve Jobs, Mac Pro, OS X
This list will then be used to find the sessions originating from a search containing one of these words. These sessions, and the sessions with no referring domain, are counted as branded sessions. To calculate the brand index, the branded sessions is divided by all sessions.
Bi= Branded sessions
All sessions (2.6)
Feedback index (Fi)
Captures qualitative information like the users’ inclination to supply feedback. There is an easy and a less easy way to measure this.
Easy way
With this method, the amount of sessions where a user is providing any type of direct feedback are counted. This can be through things like an email-link or a rating tool.
Fi= Sessions where visitors submit feedback
All sessions (2.7)
Less easy way
The less easy strategy is to send out online surveys and use that data as basis of the calculation of the feedback index. Then the amount of positive responses is divided by the total amount of responses.
Fi= Number of positive responses
2.3. Google Analytics
Interaction index (Ii)
Captures visitor interaction with content or functionality that is designed to increase the level of attention the user is paying to the brand or website. By deciding certain actions that indicates that a user is engaged, it is possible to calculate this index by dividing the amount of sessions with at least one of these actions by the total amount of sessions. The actions can be things like posting a comment, writing a review or downloading a PDF.
Ii= Sessions where visitor completes an action
All sessions (2.9)
Loyalty index (Li)
Captures the amount of returning users. A high loyalty index represents a user that is likely paying attention. The calculation for the loyalty index is as follows:
Li=1 ´ 1
Number of visitor sessions during the time frame (2.10)
2.2.4
User engagement survey
O´Brien and Toms presents in their study a survey tool that can be used to measure user engagement [11]. Several alternatives to measure engagement is discussed, and categorized into the following three classes: self-report, participant observation and biometrics. They determined that working with self-reporting and thereby a survey tool was the best alternative to measure engagement since there is not yet a good way to utilize the combination of participant observation and biometrics with self-reporting. This is for example partly because if a user spends a lot of time on completing a task it is impossible to tell if the user is engaged or just confused by the task.
The survey tool called "The User Engagement Scale", which can be found at Appendix A.1, includes 12 questions that were created for the specific purpose of measuring engagement for an e-commerce product. The data collected from the survey is then derived into six factors, Aesthetics, Endurability, Felt Involvement, Focused Attention, Novelty and Perceived Usability. Each factor is given a value based on the collected data and together they give an understanding of how the software is received from the users perspective.
2.3
Google Analytics
Google Analytics is a web analytics service developer by Google [12]. By implementing small snippets of JavaScript code into the application, users’ activity can be tracked while using the website. The information gathered is session duration, which pages they visit, bounce rate and much more. It is also possible to implement custom events into a website, which are sent to Google Analytics. By collecting and processing the users’ data, Google Analytics can generate this into different reports with statistical data and graphs that represent their activity.
2.4
Gamification
Gamification has been described by Karl M Kapp as “using game-based mechanics, aesthetics, and game-thinking to engage people, motivate action, promote learning, and solve problems.” [13]. This definition emphasizes that gamification is found not necessarily only in games, but in all kinds of context where user interaction is in focus and an increase in engagement is sought. It is most commonly used as a tool to increase motivation and productivity within areas not usually associated with games. What defines if a game is actually a game is however somewhat subjective. One component used to define a game, is the shared social
2.5. Learning computer science through puzzles
understanding of the constraints and rules, which is described by Deterding et al [14]. This has blurred the lines between games and "gamified" applications, but is concretely made separate through their level of applied gamification elements.
Gamification is applied by the use of gamification elements, where gamification elements not necessarily are visualizations, but also game mechanics, design and a way of thinking. When talking about gamification elements, Kapp presents four elements that can be used to increase engagement in learning environments: freedom to fail, interest curve, storytelling and feedback [15].
Freedom to fail
Freedom to fail is about giving users the possibility to learn through failure. This is popularly applied through the use of extra lives or lifelines. Kapp emphasizes that this is not just some plug and play aspect, but must be used with constructive feedback to help the user to understand what to do differently next time.
Interest curve
Kapp suggests that the interest level of the user should fluctuate during the learning period, in a specific pattern, in order to keep the user interested. It should start of with an exciting start, continue with a compelling middle that keeps the user engaged and finish with an interest climax before exiting the learning experience.
Storytelling
Storytelling is the notion of building user engagement through an interesting story. For example, in Super Mario where the player have to save the princess Peach that has been kidnapped by the villain Bowser.
Feedback
Feedback in video games is constant and in real-time. For example, how many lives the player have left or if the player is going in the wrong direction on a racing track is often displayed in games. This feedback instantly tells the user if progress has been made in the right direction, and this would play a vital role in an educational context.
Dicheva et al. studied other types of gamification elements and how they are used generally in an educational context. They found that the most common gamification elements are badges, leaderboards and points [16]. Badges are used as an alternative checkpoint to show an advancement of some particular skill. Points and leaderboards sort of go hand in hand, as leaderboards most commonly display the users that has the most points. One study by Olsson et al. found that badges were in general perceived as "nice to have", rather than something that drove user engagement or motivation [17]. It did however show that the use of progress bars gave users a good overview of their progress, which motivated them to complete more tasks.
2.5
Learning computer science through puzzles
In order to motivate students learning programming, Parsons et al. set out to make a puzzle game that teaches introductory programming in the language Pascal [18]. By sending out questionnaires to the students that used this puzzle tool, they got feedback of what students thought of this way of learning and what changes they wanted to the system. The puzzles consists of drag and drop exercises where the user should pick from different statements and put them in the correct order. Over 80% of the students using the application found it "useful" or "very useful" for learning programming.
2.6. Feedback
useful", the most common answer was that the application should provide detailed feedback about what errors that were made. The feedback to the students was given by pointing out the incorrectly placed statements but the program does not give any information as to why it was incorrect. The students also desired more variety in the type of puzzles in the application, rather than having only drag and drop. Something else that the users found important, was to have several examples of each problem type.
2.6
Feedback
If people make mistakes or get an answer wrong, they generally want to know how they can correct their mistake or know what the correct answer is. The notion of providing either of these is called feedback and is used across various types of sciences. Vasilyeva et al. proposed in a study that, in web-based learning systems, "Feedback is a response to the result of action primarily aimed to correct future iterations of the action, or related action [...], and is provided to a user to compare his or her performance with the expected one" [19]. They also claim that, since feedback can be provided with a difference in content, time of presentation and manner of representation, "[...] Properties of feedback are especially important in applications for users with a large variety of individual characteristics and goals". Since there are varying concepts in programming that prove more or less difficult for different students, this could therefore be one of those types of application areas. The study by Vasilyeva et al. set out to investigate adaptive feedback and that is why the different properties of feedback are mentioned. Adaptive feedback is about tailoring the feedback to each individual user to give them feedback that is relevant, that they can understand and that it comes in a time when the user can receive it properly.
There are two situations when feedback can be received, and that is either immediate or after some time has passed (also called delayed feedback). Immediate feedback is given immediately when the student produces and error or takes a step in the wrong direction of a successful completion of any given task. Delayed feedback is generally presented to the student either upon the students request or after a given task has been completed. There have been studies comparing both immediate and delayed feedback, and these found that immediate feedback made students more efficient at solving the tasks they were given than if given delayed feedback [20, 21, 22]. In other studies it could be shown that students that was presented with delayed feedback performed significantly better at similar tasks later on than those presented with immediate feedback [23, 24, 25]. The guidance hypothesis presented by Schmidt, Young, Swinnen & Shapiro argue that immediate feedback could prevent students from gaining important secondary skills such as, error detection, error correction and meta-cognitive skills [26]. This implies that for a student that is learning about programming, this could for example mean a loss in an ability to debug code and more generally damage a students perception on learning since they may become too reliant on feedback. Conclusions made by researchers even claim that immediate feedback could reinforce beliefs that problem solving is a single-step process as well as let students adopt an overly optimistic estimation of their level of understanding [27, 28].
3
Method
This chapter presents the method that this study has followed.
3.1
Planned method for this study
Based on the theory presented in the previous chapter, the plan for this study was to further develop the web application CoderQuiz by:
• Adding new types of quizzes with the focus on teaching concepts that are usually difficult to understand for students that are new to programming
• Developing current quizzes
• Giving users better feedback when they fail to improve upon the "freedom to fail"-scenario
• Adding gamification elements to motivate users to keep using the application and eventually finish all challenges
The data needed to answer the research question, was collected by:
• Sending users a survey based on the survey tool A.1 to measure engagement • Analyzing the Google Analytics data
• Measuring each user’s total progression
3.2
Implementing tracing quiz
Only two types of quizzes existed when the project started, and those were placeholder and selection. After the findings presented in Section 2.1 which talked about what types of difficulties that novice programmers usually encounter, a new type of quiz was introduced, namely the tracing quiz. The goal with the tracing quiz was to teach users the order in which a program executes. This meant that users had to understand which line of code that was being read and how the program navigated between each line as well as how many times a line of code was read. The idea became that users would be able to select different rows and
3.3. Implementing gamification
thereby order the rows in execution order through individual selection. The quiz would also have delayed feedback which will be talked about further in Section 3.4.
3.3
Implementing gamification
When deciding upon what gamification elements that were to be implemented, theory presented in Section 2.4 argued for the use of progress bars, feedback and some kind of storytelling. Because of this, the gamification elements that were chosen for implementation was:
• Progress bar • Levels
• Storytelling robot • Feedback
The progress bar was chosen to have colors that matched the overall application and to include a rolling animation that would hopefully appeal to the users. The levels were to include sets of quizzes and to only be visible if they had been unlocked by the user. The levels would also have their own progress bar to show progress on that particular level as well as a button to unlock the next level. The storytelling robot, that was designed by Erik Berglund, was chosen to be a robot since a robot consists of computers that in themselves could be programmed and CoderQuiz teaches programming. Feedback was designed individually for each quiz type and is further described in Section 3.4.
3.4
Implementing feedback
Starting off, the CoderQuiz project had no feedback mechanisms implemented besides letting the user know when a quiz was completed. The goal with the planned implementation of feedback mechanisms was to give the user an understanding of if they were on the right path or not. Each quiz-type’s feedback mechanism was designed for that particular quiz-type to provide feedback in a simple but effective way. The goal was also to design the feedback mechanisms in a way that gave the users the opportunity to reflect over their answer, before checking if it was the correct answer. This design is based on the theory about how delayed feedback could give users a deeper knowledge and a more advanced understanding of a subject which was discussed in Section 2.6.
3.5
Code descriptions
Since the application is teaching students about basic programming concepts, there was an obvious risk in introducing students to larger blocks of code and expecting extensive comprehension. Some parts of the code would include built-in functions and syntax usage that is far from intuitive for novice programmers and could possibly provoke users to seek knowledge outside of the application. In an attempt to keep users from navigating away from the application, something called "code description" was introduced. Code descriptions are informative modals that teaches the user about what a certain built-in function does and how to use it. These code descriptions were dynamically constructed based on which quiz the user were currently solving and placed in a drop-down menu named ’Code descriptions’ next to the quiz code.
3.6. User engagement
3.6
User engagement
The users’ data and activity was collected and analyzed in order to measure the user engagement.
3.6.1
Data collection through Google Analytics
Google Analytics was used to analyze and collect the activity of the users using CoderQuiz. By implementing JavaScript code into the application, the activity of users that are logged in to the website was sent to Google Analytics. When a user registers an account, the user is assigned to a group named either A or B. The group is decided by alternating between A and B every time a user registers. Depending on the group of the account, the data will be sent to a different Google Analytics account. The time span of the data was decided to be three weeks.
Group A
The users in group A sees the version of the website that hides the gamification elements.
Group B
The users in group B sees the version of the website that shows the gamification elements. The users that used CoderQuiz were mainly university students taking their first course in programming. The website was promoted to them by email, on seminars and on lectures by the examiner of the course.
Some custom events were also implemented to the website, which means that when a user performed a specific action, that data was also sent to the Google Analytics account. These events were used to analyze the users’ behavior in order to get feedback on the system and to measure the users’ engagement. These events were:
Complete
This event is activated whenever a user completes a quiz.
Error
This event is activated when the user have put in an incorrect answer or, in case of the tracing quiz, when the user presses the "Check answer"-button and has guessed the incorrect execution order.
Try
When a user has pressed the “Check Answer” button on the placeholder and tracing quizzes, this event will be activated.
Leave
This event is sent to Google Analytics if a user leaves a quiz without completing it.
3.6.2
Measuring user engagement through user activity
The basic measurements used to measure user engagement have several limitations which are explained in Section 2.2.2. In order to reduce the effect of these limitations, the framework developed by Carrabis and Peterson was used to measure the user engagement of both versions of the provided data from Google Analytics. Formula 2.1 was used to measure user engagement. The indices were measured by analyzing the data from Google Analytics and using the formulas explained in Section 2.2.3. The indices are meant to be measured for each user, and then to take the average of all users’ indices. Because there was no option to
3.6. User engagement
download all user reports in the free version of Google Analytics, it was decided to instead take the average values for all users and calculate the indices as if it was only one user, thus calculating the indices only once. Minor modifications were also made to some of the formulas. The reason for this was that some of the data that was needed for the original formulas was not available. These changes and other necessary clarifications are described in the following list of indices from the framework.
Click depth index (Ci)
It takes 4 clicks to enter a quiz from the home page, so the threshold for this formula was chosen to be 4.
Ci= Sessions having at least “4 page views” All sessions
Duration index (Di)
To be engaged in this application, 3 minutes was decided to be the threshold to use in this formula.
Di= Sessions where “duration over 3 min” All sessions
Recency index (Ri)
In the free version of Google Analytics it is not possible to export all users’ session data, and therefore formula 2.5 was chosen, which is the older version of the formula to measure recency. The time span of the data collected was 3 weeks, and therefore 3 page views during the last 3 weeks was chosen for this formula.
Ri= Sessions having at least “3 page views during the last 3 weeks” All sessions
Brand index (Bi)
Because of the time constraint of getting this application ready, the sessions originating from a search of a word related to the brand CoderQuiz, could not be extracted. Branded sessions is therefore defined as only the sessions with no referring domain.
Bi= Branded sessions All sessions
Feedback index (Fi)
The data of how many emails a user sent in one single session was not collected, so this index required a modified version. In the modified version of the formula the amount of emails received during the entire time span is counted. There is a contact page on the website, referring to an email address, so the users have the option to send feedback.
Original formula
Fi= Sessions where visitors submit feedback All sessions
Modified formula
Fi= Amount of mail received with feedback or problems All sessions
Interaction index (Ii)
The actions that has been defined for the calculation of this index are "Complete", "Error", "Try" and "Leave", as described in Section 3.6.1.
Ii= Sessions where visitor completes an action All sessions
3.6. User engagement
Loyalty index (Li)
The time frame of the data collected is 3 weeks.
Li=1 ´ 1
Number of visitor sessions during 3 weeks
The total total user engagement score is then calculated by taking the sum of the indices for all users, divided by the amount of indices in the formula.
ř
(Ci+Di+Ri+Li+Bi+Fi+Ii)
7 (3.1)
3.6.3
Completion
In order to see how far users have progressed, the amount of completed assignments for each user will be looked up in the database. With this information it is possible to see the possible discrepancy in how many assignments the users in both groups completed. Because this information is a good indicator of how engaged the students were using CoderQuiz, but not included in the user engagement formula 3.1, it was decided to include this information in a separate part. The completion will be categorized by the amount of quizzes that needs to be completed in order to unlock each level as well as the users having finished all assignments, as shown in Table 3.1. Only users having completed at least one assignment will be taken into account. This is because the users that signed up without trying the application, are not relevant to the comparison of assignment completion by users. The average amount of completed assignments per user in both groups will also be calculated in order to present the difference in progression.
Level 1 2 3 4 5 All levels completed
Assignments completed 1-12 13-22 23-33 34-45 46-58 59 Table 3.1: Completed assignments categorization
3.6.4
Measuring user engagement through user surveying
In order to validate the data collected through Google Analytics, a survey was conducted based on the survey tool discussed in Section 2.2.4. In this way, a subjective perception on user engagement could be calculated and used to validate previously collected data. The survey tool used was the shorter version of the two that the study by O’Brien et al. presented and was chosen due to encouragement from the authors [11]. Their argument was that a shorter version of this type of survey would increase the amount of users participating and thereby increase the validity of the data.
The survey was sent out to all users that used CoderQuiz and had completed at least five assignments. The threshold of five assignments was chosen, because users having done less than five assignments were presumed not to have gotten a clear impression of the application yet. Several users were also specifically asked to try the application, complete at least five assignments and to answer the survey. This was a different group of users to those that used the application during the time frame where data was collected by Google Analytics. Therefore, these users had no effect on the Google Analytics data or the completion data. These users had a programming experience ranging from beginner to intermediate. The data was collected anonymously and the users were also informed about that.
3.6. User engagement
The user engagement scale - Short form
This scale is based on the one found in Appendix A.1 and has been modified to fit the CoderQuiz application. The word "taxing" was changed to "tiring" on question PU-S.3 to make it easier to understand for non-native English speakers.
FA-S.1 I lost myself in this experience.
FA-S.2 The time i spent using CoderQuiz just slipped away. FA-S.3 I was absorbed in this experience.
PU-S.1 I felt frustrated while using CoderQuiz. PU-S.2 I found CoderQuiz confusing to use. PU-S.3 Using CoderQuiz was tiring.
AE-S.1 CoderQuiz was attractive.
AE-S.2 CoderQuiz was aesthetically appealing. AE-S.3 CoderQuiz appealed to my senses. RW-S.1 Using CoderQuiz was worthwhile. RW-S.2 My experience was rewarding. RW-S.3 I felt interested in this experience.
Scoring the survey
When calculating the overall user engagement score, there are four factors that make up this score together. These factors are:
• FA - Focused attention
Feeling absorbed in the interaction and losing track of time. • PU - Perceived usability
Negative affect experienced as a result of the interaction and the degree of control and effort expended.
• AE - Aesthetic appeal
The attractiveness and visual appeal of the interface. • RW - Reward
The perceived benefit of using the application.
There are three questions corresponding to each factor. These questions were scored by users answering the survey. The score of a factor was calculated by summing the scores of the corresponding questions and dividing the sum by the amount of questions. This way, a value for each individual factor could be calculated. To get the overall user engagement score, all questions from all factors were summed together and divided by the total amount of questions. The score depends on the answer, as shown in Table 3.2. To get the correct interpretation for all factors, the PU-factor (perceived usability) had to be reverse coded when performing calculations.
Strongly disagree Disagree Neither agree nor disagree Agree Strongly agree
1 2 3 4 5
4
Results
In this chapter the results of the thesis will be presented.
4.1
The system - CoderQuiz
The results presented in this section are the results of the technical implementation that was made in CoderQuiz together with this thesis.
4.1.1
Feedback
This section presents the implementation of a feedback system for each type of quiz in CoderQuiz.
Placeholder quiz
When a user initially performed this quiz, they could try out different combinations of snippets and when they had provided the correct combination, they would instantly receive a prompt telling them that they had completed the quiz. This meant that a user could swap snippets back and forth without any real thought behind their selection, and also without any indication if they were getting closer to a correct solution.
The changes that were implemented consisted of a button labeled ’Check answer’ and next to it, a text saying ’X/Y correct answers’. X is the amount of correctly placed snippets and Y is the amount of correct snippets needed to complete the quiz. The use of a button to prompt a check of the answer, allows the user some time to reflect upon the choice of snippets, before deciding to click and check if the correct answer had been provided.
Figure 4.1 displays how the feedback mechanism for the placeholder quiz was implemented. Primarily it consists of the ’Check answer’-button accompanied by the text on its right hand side, telling the user how many correct snippets that are currently in place. The figure shows the system after the user has pressed the button and when one out of three correct snippets are in place.
4.1. The system - CoderQuiz
Figure 4.1: Feedback in the placeholder quiz
Selection quiz
The selection quiz was also without feedback mechanisms, and a user was previously able to select as many elements in the editor as possible. There was an indication in the header of how many elements that were supposed to be selected, but no actual restriction on how many elements the user could have selected at any given time.
First off, a restriction was set so that a user can only select as many elements as is requested, and if a user still tries to select more elements than possible, a text will appear that tells the user to deselect an element before selecting another. Secondly, different colors were chosen to highlight elements and describe to the user if the element chosen was correct or not. When selecting an element, the element is highlighted with a white color. When a user has selected the same amount of elements as the quiz requested, the selected elements are highlighted in either green (if they are correct) or red (if they are incorrect). If a selection is incorrect and the user tries to select another element without deselecting the incorrect element first, the incorrect element will flash in the same red color to inform the user of the mistake.
The feedback mechanism for the selection quiz is only displayed when the user have selected the same amount of elements that the quiz is requesting. Figure 4.2 shows a situation where a user has selected one element out of two possible and where this selection is incorrect.
4.1. The system - CoderQuiz
Figure 4.2: Feedback in the selection quiz
Tracing quiz
The way feedback was implemented in tracing quizzes was that the order-numbers associated with a line of code, and placed there by the user, could be marked with one of three colors. If a user has selected a row in the wrong order, the order-numbers associated with that row will be marked with a red color.
If a user has selected a row in the correct order but that specific row is also used later in the execution, meaning that it should be clicked again later, the order-numbers for that row will be marked with a blue color. This indicates that the user is on the right way, but not quite there yet. If a row has been selected in the correct order and the appropriate amount of times, depending on the execution, the order-numbers for that row will be marked with a green color. This indicates that the user is 100% correct about that row’s participation in the program execution.
Just like the placeholder quiz, the tracing quiz feedback mechanism is triggered by a button named ’Check answer’. When a user presses the button, the order-numbers belonging to each line of code are colored based on how correct they are. In Figure 4.3 there are five lines of code that has been selected. Two of these lines are incorrect, one is almost correct and two are completely correct. In the header, the user is given a key that explains what each color means, and if that is not enough then there is more extensive explanations to be found through the ’Help’-button.
4.1. The system - CoderQuiz
Figure 4.3: Feedback in the tracing quiz
4.1.2
Code description
Code descriptions are modals with information about how built-in functions in Python 3 work. Since there are different sets of code for quizzes, each quiz is given a drop down menu containing only the built-in functions that appear in that specific quiz. This is done by parsing the code each time a quiz is loaded and identifying the built-in functions present in that particular set of code using regular expression.
4.1. The system - CoderQuiz
Figure 4.5: Code description modal
4.1.3
Implementation of the tracing quiz
The implementation of the tracing quiz was similar to the other quizzes in the way that it consisted of a code editor, but this time with distinct lines in-between each line of code. This visual distinction was implemented to make it easier for users to separate the lines of code when deciding which line to select next. When a user selects a line of code, a number corresponding to the programs guessed execution order appears next to the line on the right hand side. After selecting a line of code, the user could either keep selecting lines of code or check if the current order of selected lines is the correct one. The user checks this by pressing a button named ’Check answer’ located on top of the code editor.
When a user has pressed the ’Check answer’-button, the numbers on the right hand side of the code editor will be colored based on if the selected order was correct or not. This implementation was chosen based on discussions in the team, that agreed on that a coloring of numbers complemented by a color-key would suffice as an explanation on how the user was progressing with the quiz.
Figure 4.6 shows the implementation of the tracing quiz. The task is located at the header, though the task is the same for all quizzes of this type. The quiz buttons, that enables the user to interact with the quiz, are located above the editor for easy access by the user and consistency with the other quiz types. The user can click a row to mark it and remove its mark with either a right mouse-click or by pressing the ’Undo’-button.
4.1. The system - CoderQuiz
Figure 4.6: Tracing quiz
4.1.4
Gamification
This section presents the different gamification elements that were implemented into CoderQuiz and their usage.
Progress bar
The main progress bar is shown on the page where the user has an overview of all quizzes available and was implemented on the top of the page so that it is easy for users to locate it and even hard to miss. The progress depends on how many quizzes are completed, out of the total amount of quizzes. Figure 4.7 shows a progress bar for a user that has completed all quizzes.
Figure 4.7: Progress bar showing percentage of assignments completed
Levels
A level is a set of quiz-lists and contains no more than three quiz-lists at a time. A quiz-list is a set of quizzes of different type but usually with a common theme. The CoderQuiz game has five levels consisting of quiz-lists, sorted in a way of increasing difficulty for the users. Before users can progress to the next level they must complete a specific amount of quizzes. Each level has its respective progress bar that is displayed below the quiz-lists for the current level. When the user has completed the amount of quizzes that the level specific progress bar is showing, the button below this progress bar will go from reading ’Locked’ to ’Unlock’. By pressing this button, the user will unlock the next level. The locked levels are not visible to the user before pressing the unlock button.
4.1. The system - CoderQuiz
Figure 4.8: Levels in CoderQuiz
Figure 4.9: Button unlocking the next level
Storytelling robot
In an attempt to engage users and increase their usage of the application, a fantasy narrative was created with a robot at the center of it, which can be seen in Figure 4.10. The application was designed so that the robot would be a natural part of the users’ progression. At the quiz game page, users can interact with the robot by clicking on it, which then displays a modal containing the part of the story they are currently on. This modal can be seen in Figure 4.11. Users progress through the story in parallel as they progress through the levels of the game. Thus, users would be prompted by the robot whenever they advanced to a new level, that a new part of the story had been unlocked as well. The narrative is that the robot is being attacked by bugs and users can help the robot get rid of the bugs by completing quizzes. The robot appears at the home page, the quiz game page, the story modals and the win modals. At the home page there is no real interaction with the robot besides a visual reaction
4.1. The system - CoderQuiz
whenever a user clicks on the robot. At the quiz game page, the robot is positioned in the bottom right corner as seen in Figure 4.12. At the win-modals, the robot appears with a speech-bubble and some encouraging text, but is besides that as interactive as when appearing at the home page.
Figure 4.10: The storytelling robot, created by Erik Berglund
4.2. User engagement
Figure 4.12: Robot in the quiz game
4.2
User engagement
In the following sections the data collected from both the survey and Google Analytics is presented and also transformed into a representation of engagement. The data has been collected for two separate groups of users; group A with no gamification elements shown in the application and group B with the gamification elements shown.
4.2.1
User engagement calculated with Google Analytics data
With the data collected with Google Analytics and the emails sent to us, the indices and the total engagement score was calculated for both groups. In the three weeks of data that was analyzed, Group A had 33 users and Group B had 29 users log in to CoderQuiz. From those users’ activity, the variables needed to calculate the indices have been extracted, and the variables are shown in Table 4.1.
The indices needed to calculate the total engagement score are shown in Table 4.2 and calculated with the formulas listed below and by rounding the results to two decimal places. A more detailed description about what these indices mean can be found in Section 2.2.
4.2. User engagement
Variable Group A Group B
Sessions having at least "4 page views" 55 56 Sessions where "duration over 3 min" 49 49 Sessions having at least "3 page views during the last 3 weeks" 61 58
Branded sessions 35 44
Amount of mail received with feedback or problems 1 1 Sessions where visitor completes an action 45 54 Number of visitor sessions during 3 weeks 2.37 2.91
All sessions 97 96
Table 4.1: Variables used to calculate the indices
Click depth index (Ci)
Ci= Sessions having at least "4 page views" All sessions
Duration index (Di)
Di= Sessions where "duration over 3 min" All sessions
Recency index (Ri)
Ri= Sessions having at least "3 page views during the last 3 weeks" All sessions
Brand index (Bi)
Bi= Branded sessions All sessions
Feedback index (Fi)
Fi= Amount of mail received with feedback or problems All sessions
Interaction index (Ii)
Ii= Sessions where visitor completes an action All sessions
Loyalty index (Li)
Li=1 ´ 1
Number of visitor sessions during 3 weeks
Group Ci Di Ri Bi Fi Ii Li
A 0.57 0.51 0.63 0.36 0.01 0.46 0.58 B 0.58 0.51 0.60 0.46 0.01 0.56 0.66
Table 4.2: Engagement indices
The total engagement score for each group is calculated by dividing the sum of the indices in Table 4.2 by the amount of indices. The total engagement score is shown in Table 4.3.
4.2. User engagement
Group Total engagement score
A 0.45
B 0.48
Table 4.3: Engagement score
4.2.2
The survey and the user engagement scale
The user engagement survey, defined in Section 3.6.4, was sent out to the 53 users having completed five or more assignments. The survey was also sent to 15 users that were specifically asked to try the application, complete at least five assignments and to answer the survey. This group of users was needed in order to make sure that enough feedback was collected through the survey. Out of the total 68 users, 18 users responded with their feedback.
Table 4.4 and Table 4.5 shows the distribution of how users in group A and B, answered the questions from the user engagement survey. Table 4.6 shows how the different groups score the four factors: focused attention (FA), perceived usability (PU), aesthetic appeal (AE) and reward (RW). The different scores of the answers are shown in 3.2. The PU-questions have to be reverse coded because of how they are asked. The overall engagement score is calculated as an average of these factors.
Question Strongly disagree Disagree Neither agree nor disagree Agree Strongly agree
FA-S.1 1 1 3 3 0 FA-S.2 1 2 3 1 1 FA-S.3 0 3 1 4 0 PU-S.1 3 2 0 3 0 PU-S.2 1 2 1 4 0 PU-S.3 3 5 0 0 0 AE-S.1 0 1 3 4 0 AE-S.2 0 2 2 3 1 AE-S.3 0 1 3 4 0 RW-S.1 0 1 2 3 2 RW-S.2 0 1 2 3 2 RW-S.3 1 1 0 3 3
4.2. User engagement
Question Strongly disagree Disagree Neither agree nor disagree Agree Strongly agree
FA-S.1 0 4 2 4 0 FA-S.2 0 2 3 2 3 FA-S.3 0 1 2 7 0 PU-S.1 3 1 0 6 0 PU-S.2 3 4 2 1 0 PU-S.3 0 4 5 1 0 AE-S.1 0 0 3 5 2 AE-S.2 0 0 3 6 1 AE-S.3 0 0 3 6 1 RW-S.1 0 0 1 5 4 RW-S.2 0 0 0 7 3 RW-S.3 0 0 0 5 5
Table 4.5: B-group survey result
FA PU AE RW Overall engagement score
A-users 3.00 3.67 3.38 3.75 3.45
B-users 3.40 2.57 3.83 4.37 3.54
Table 4.6: The average result of the survey for both groups
4.2.3
Completion
By categorizing the completion to the amount of assignments required to unlock a new level as well as the amount of users that completed all assignments, the distribution is shown in Table 4.7 and Table 4.8. The average of completed assignments, for users that have completed at least one assignment, is shown in Table 4.9.
Assignments completed
1-12 13-22 23-33 34-45 46-58 59
A-users 22 2 9 3 1 1
B-users 19 4 1 3 3 2
4.2. User engagement
Assignments completed
1-12 13-22 23-33 34-45 46-58 59 A-users 57.9% 5.3% 23.7% 7.9% 2.6% 2.6% B-users 59.4% 12.5% 3.1% 9.4% 9.4% 6.3% Table 4.8: Percentage of users that completed assignments
Group Average completed assignments per user
A 16.45
B 18.63
5
Discussion
In this chapter the results and the method used to collect the results will be discussed.
5.1
Results
The discussion is divided into three parts, where we discuss the results extracted by the three methods we used to collect data.
5.1.1
User engagement calculated with Google Analytics data
The resulting indices for group A and group B were overall very close in value. The more visible differences can be seen in the brand index, interaction index and loyalty index. The differences of these indices are still quite small, and it is therefore difficult to make any kind of generalization based on these alone.
Group B had a higher brand index which means that the users with gamification more often opened CoderQuiz with no referring domain. Therefore, it seems like B-users had a greater brand awareness in general. The loyalty index was a bit higher for B-users as well, which means that B-users returned to the website more often than A-users did, and were therefore likely paying more attention to the application.
Group B also had a significant increase in interaction. This means that B-users went into the website and interacted with a quiz more often than A-users did. This can be seen as users that have gamification elements in their application, are reaching the point of engagement more often. According to the proposed model of engagement seen in Table 2.1, the attributes that influences users to reach this point are aesthetics, novelty, interest, motivation and specific or experiential goals. The biggest difference between the two versions of the website is that the gamification version have progress bars and a specific goal for the user to use, which may have influenced this result. Because the B-users have a visible goal to reach the next level and can see their total progress completed so far, B-users may be more encouraged to start doing quizzes. The gamification version also displays the robot mascot with an included story which should slightly influence the attributes "aesthetics" and "interest" of the application.
5.2. Method
The recency index was however slightly stronger with the A-users, and A-users were therefore more recent visitors for the examined time-span. This might be because an engaged user will be finished with more assignments faster, and might reach a point where progression will feel more difficult or when there are no more assignments left to do. It may also be because the A-users have more assignments available without having to unlock them, and will therefore feel like there are more things to do in the application than a B-user will. Overall the engagement score is higher with the B-users which is what was expected because of the researched positive benefits of gamification [15].
5.1.2
User engagement through surveying
As with the result from the data collected with Google Analytics, the numerical difference in user engagement that could be calculated from surveying was not significant, though still showed better results with users having gamification elements available. As expected, group B felt that the experience was more rewarding, that the website was more aesthetically appealing and felt more absorbed into the application than A-users did. However, B-users felt that the experience of using CoderQuiz was frustrating, much more than A-users did. This could mean that the use of gamification elements puts a type of pressure on users to finish what they started which also can be interpreted as the users being engaged. This was a notion that was presented earlier in the thesis and that talked about how a frustrated user can be an engaged user, which could be argued for through this result [8]. A-users could also easily choose between all assignments, which means they can easily give up on an assignment and try another one. B-users were much more limited on how many assignments they could give up on, before having to finish certain assignments in order to progress and unlock more content. Because of this it feels natural that B-users should feel more frustrated with the application in general.
5.1.3
Completion of assignments
In average, a B-user have completed around two more assignments than an A-user. This is mostly because some B-users progressed very far in the application. 12.1% of A-users and 25.1% of B-users have completed 34 or more assignments. The biggest difference is however visible in the groups of users that have completed 23-33 assignments where there are 21.7% of A-users and only 3.1% of B-users. This might be because, as mentioned before, the A-users have all assignments available from the start. They can easily try many different assignments and feel free to skip those that are too difficult. The A-users have however no gamification elements to encourage them to complete all assignments which might be why the B-users perform better in the completion rate overall.
5.2
Method
As discussed in the theory chapter, the use of gamification elements could affect user engagement. This does not mean that all gamification elements affect user engagement in the same way and different combinations of gamification elements could also differ in its effect on user engagement. The choice of gamification elements for CoderQuiz and their application in CoderQuiz specifically has most likely a close to unique affect on user engagement. Had the choice of gamification elements or their implementation been different, the possible effect on user engagement could either have been bigger or smaller depending on gamification elements and implementation. Generally, the replicability of this project is quite low since all measurements and results are made from usage of the CoderQuiz application which is somewhat of an original type of system.
