Competition or Cooperation?

IN

DEGREE PROJECT MEDIA TECHNOLOGY, SECOND CYCLE, 30 CREDITS

STOCKHOLM SWEDEN 2020 ,

Competition or Cooperation?

Using push notifications to increase user engagement in a gamified smartphone

application for reducing personal CO -emissions ₂ SEBASTIAN BLOMKVIST

KTH ROYAL INSTITUTE OF TECHNOLOGY

SCHOOL OF ELECTRICAL ENGINEERING AND COMPUTER SCIENCE

Tävling eller samarbete?

Användning av push-notiser för att öka engagemanget hos användare av en spelifierad smartphone-applikation för att minska CO

-utsläpp

SAMMANFATTNING

Ett effektivt verktyg för att forma, bryta eller bibehålla vanor och beteenden är en så kallad digital beteendeförändrings-intervention (eng. digital behavior change intervention eller DBCI). Dessa interventioner använder digital teknik för att hjälpa deras användare att antingen påbörja eller undvika särskilda beteenden. Ett vanligt problem med dessa är att användarna ofta inte är särskilt engagerade i interventionernas innehåll eller funktioner, vilket är viktigt för deras effektivitet. Emellertid har det visats att spelifierat innehåll och användandet av notiser—såsom push-notiser—kan ha en ökande effekt på engagemanget. Två vanligt förekommande spelkoncept är tävling och samarbete, båda med sina olika effekter på engagemang vilket också skiftar beroende på kontexten och användaren. Därför ska detta examensarbete undersöka hur push-notiser kan användas för att öka användar-engagemanget i en spelifierad och mobil beteendeförändrings-intervention genom att göra dess spelifierade element mer framträdande. Dessutom kommer det även undersökas om det är någon skillnad i effekt mellan notiser som antingen främjar tävling eller samarbete. Detta var utvärderat genom att använda två olika push-notis-strategier på Deedster—en spelifierad, mobil intervention som syftar till att minska dess användares CO2-utsläpp—och sedan följa användarnas beteende. Resultaten visade att användare som fick push-notiser var mer engagerade—startade mer sessioner och spenderade mer tid—inom applikationen jämfört med användare som inte fick några notiser. De utförde också signifikant fler önskade beteenden. Det var ingen skillnad i effekt på antalet önskade beteende utförda mellan att främja tävling eller samarbete och endast några få skillnader i engagemang. Dock visade det sig att användarens kön var en betydande faktor i effekten av notiserna. Notiserna som främjade tävling var mer effektiva för manliga användare jämfört med de som främjade samarbete. Denna effekt syntes inte bland kvinnliga användare.

1

Competition or Cooperation?

Using push notifications to increase user engagement in a gamified smartphone application for reducing personal CO

-emissions

Sebastian Blomkvist

KTH Royal Institute of Technology Stockholm, Sweden

sblomk@kth.se

ABSTRACT

A helpful tool in forming, breaking, and maintaining habits and behaviors is a digital behavior change intervention (DBCI). These are interventions that leverage digital technologies to help their users to either take on or avoid certain behaviors. A common problem is a lack of user engagement with the interventions’

content, which is key for its effectiveness. It has however been shown that gamified content and using prompts—such as push notifications—may have the effect of increasing user engagement, for both DBCIs and other applications. Furthermore, two commonly occurring game concepts are competition and cooperation, each with different influences on engagement which in turn may vary depending on the context and the user. Therefore, this thesis set out to examine how push notifications can be used to increase user engagement with a gamified DBCI by making its gamified elements more salient. Additionally, it will investigate if there is any difference in influence on engagement of notifications that either promote competition or cooperation. This was evaluated by deploying two different push notification strategies on Deedster—a gamified mobile DBCI with the aim to get its users to reduce their personal CO2-emissions—and tracking user behavior.

The results of the evaluation showed that users who received push notifications were more engaged—started more sessions and spent more time—with the application than users who did not receive any. They also performed a significantly higher amount of target behaviors. There was no difference in the influence on performed target behaviors between the notifications promoting competition or cooperation, and only one significant difference—usage of intervention features—regarding user engagement. The gender of the user was also found to be a considerable factor in the influence of the push notifications. Competition increased engagement more than cooperation for male users, but not for female users.

Keywords

Sustainable HCI; digital behavior change interventions;

gamification; user engagement; competition; cooperation; push notifications.

1. INTRODUCTION

Digital behavior change interventions (DBCI) are today a common method to support users in breaking or gaining new habits and behaviors [33] in various domains, for instance health [24, 35] and sustainability [3, 12, 14]. A DBCI can be described as an intervention which utilizes digital technology, e.g. a smartphone application or a website, to either promote or discourage specific behaviors and habits [33, 44], often referred to as target behaviors.

To achieve this, DBCIs can employ a variety of different behavior change techniques (BCTs), which can be defined as observable and replicable features of the intervention that aims to influence behavior [22]. 93 different techniques are defined and classified in what is called the Behavior Change Technique Taxonomy [22]—

some examples of BCTs are feedback on behavior, prompts and cues, social comparison, goal setting, and instructions on how to perform a behavior. For these kinds of interventions to have any influence on behavior it is essential that the user engages with the interventions’ content and features [1, 30, 38, 42, 43]. However, DBCIs generally have a hard time keeping users engaged [3, 33, 37], and therefore often suffer from low user retention rates [7, 37].

A way of coping with this lack of user engagement is to implement additional features to increase it: such as game design elements—

i.e. gamification [6], prompts in the form of e.g. emails, text messages, or push notifications [1, 31], and personalizing or tailoring content to the user [4, 15]. Introducing elements such as rewards, leaderboards, and badges has shown to have a positive impact on users’ motivation, and thus engagement, in a variety of contexts [9, 14, 20]. Other frequently used game concepts are cooperation and competition, between or within groups of players, or between individual players. Each having different influences on various aspects on player engagement, performance, and enjoyment, which may also vary depending on the context and user characteristics [16, 17, 25, 29]. For instance, competitive gameplay have been observed to in some cases contribute to an increase in engagement [16, 25]—although sometimes only in the short term [21]—and has in some cases only shown to increase the engagement and performance of male users and not at all for female users [5, 13, 41].

2 Reminders and prompts, suggesting the user to return to the intervention or to complete tasks, have proven to be effective methods to retain users and to increase engagement [1, 8, 15, 23].

These can specifically be an effective method for mobile interventions, due to the fact that users almost always have a smartphone available and are easily reached by push notifications [8, 23]. The success of notifications however is dependent on multiple factors, such as timing, content, and how it is framed.

1.1 Deedster

Deedster—the company which this thesis will be in collaboration with—has developed a DBCI, a gamified smartphone application, with the aim to get its users to reduce their personal CO2-emissions.

The application challenges the user to complete sustainable deeds—which are micro-missions e.g. “eat a vegetarian lunch”.

The user gets offered these deeds by completing quiz-questions, see

Figure 1. Left – example of quiz-question. Right – view when a deed is completed.

Figure 2. Left – a journey. Right – leaderboard within a team.

figure 1. If the deed seems interesting and feasible, the user can accept it, otherwise it is possible to reject the deed. Some deeds can be shared with other users and others can be made repeatable after completion to let the user form habits, i.e. habit-deeds. When first using the application, each user chooses a journey, which is a challenge consisting of a set of levels called chapters. Each user completes the chapters by answering correctly on a set of quiz- questions, after which the user gets rewarded by having the option to accept a number of deeds and to progress to the next level. When choosing a journey, the user also—in most journeys—join or create a team to compete in. The teams compete in completing the most amount of deeds and, additionally, the users compete in doing the most deeds within their team, see figure 2. The user is at any time able to switch to another journey and progress through several journeys simultaneously. This also means that users are members of multiple teams at the same time.

1.2 Research Question

As it has been demonstrated that push notifications and gamified intervention content has the ability to positively influence engagement in many different contexts, this thesis will investigate how push notifications can be used to increase engagement by making gamified elements—particularly those including either cooperation or competition—of the DBCI more salient. This will be done by deploying two different push notification strategies on the Deedster application. More specifically the thesis will assess differences in the influence on engagement of push notifications that either promotes within-group cooperation or within-group competition. Additionally, the notifications will contain information that previously has been shown effective, in other or similar contexts. It will also be examined if the gender of a user will make any difference in how effective either of the strategies are.

Hence, the research questions of this thesis are:

(1) Are push notifications able to positively influence user engagement in a gamified application that aims to reduce personal CO2-emissions?

(2) Does the extent of the influence differ between push notifications that either promote (A) within-group competition or (B) within-group cooperation?

(3) Does a possible increase in user engagement lead to increased performance of the target behavior?

(4) Is the gender of a user a factor in whether (A) or (B) will positively influence engagement?

1.3 Limitations

In this thesis, engagement will only be measured by user behavior—that can be detected and logged by the application—and not by users’ subjective experience, i.e. conclusions about users’

preferences and engagement will solely be drawn from observed behavior, not from any interaction with the user (e.g. surveys or interviews). Furthermore, the only specific aspect of the content of the push notifications to be evaluated is the difference in influence on engagement of notifications that either promote within-group competition or within-group cooperation, meaning that any behavior change technique or other strategy present in the content will not specifically be evaluated.

3

2. THEORY & RELATED WORK 2.1 Engagement and DBCIs

2.1.1 Engagement – a definition

User engagement is an important factor for the effectiveness of a DBCI [1, 30, 38, 42, 43]. However, to accurately assess the amount of engagement with a DBCI, and what influences it may have, a working definition of what engagement really entails is first needed. The concept of engagement can be split into two subdomains: engagement as a subjective experience and engagement as behavior [30]. As a subjective experience engagement is characterized by attention, interest and affect, which can be exemplified by the focused mental state of flow or being fully immersed in a computer game. This is engagement in a qualitative sense. Engagement as behavior—on the other hand—

looks at engagement in quantitative behavioral terms, i.e. the extent of usage of an intervention. The extent can for instance be the frequency of intervention contact or the variety of intervention content consumed, which can be quantitatively measured by tracking how users behave in the intervention, e.g. the number of sessions logged, the duration of these sessions and what content they view in the DBCI. It may also be of use to differentiate between engagement in general and effective engagement, which can be defined as engagement that contributes to the target outcome [43], i.e. a behavior or establishing a habit. More engagement may not always ensure a more effective intervention.

2.1.2 What influences engagement

Engagement can be influenced in a variety of ways. Perski et al.

[30] divides factors which may affect the engagement with a DBCI into two synthetic constructs: context and DBCI. The context is factors outside of the actual DBCI and are in turn divided into differences in the population (e.g. demographic and psychological characteristics, personal relevance, expectations, and self-efficacy) and the setting (i.e. the social and physical environment). Within the DBCI, engagement can be influenced by the content (e.g. BCTs, prompts and reminders, rewards, and social support features) and the delivery (e.g. personalization, interactivity, medium, message tone, narrative, and aesthetics). [30]

2.2 Notifications 2.2.1 Push notifications

To inform users of new events in an application on a smartphone outside of the user’s current focus, push notifications are often utilized. This gives application developers an opportunity to communicate with their users without them having to open the application. These notifications can be perceived as disruptive and frustrating by users, but also useful and valuable depending on its content and timing [36]. The most valuable notifications, and the ones most clicked by users, are those containing a message from another user or important updates about people and events [31, 32, 36]. Of course, what is recognized as “important” may vary between contexts and users, however in general content is considered more important when conveying information about the user’s social environment [36]. Depending on the push notification’s source—e.g. direct/group messaging apps, email

clients, news sites, calendar apps, or games—the notification achieves drastically different click rates [31, 32] and its perceived importance is rated differently [36]. However, non-social applications—e.g. news apps [39] and learning apps [31]—have shown to be able to increase their user engagement by utilizing push notifications.

2.2.2 Prompts in DBCIs

The BCT taxonomy introduces prompts and cues as a technique that “introduce or define environmental or social stimulus with the purpose of prompting or cueing a behavior” [22]. While designing DBCIs, it has on multiple occasions become apparent that potential users want the intervention to remind and prompt them to perform certain behaviors and to notify them of their progress [10, 26, 34].

But do these wished-for features make people more avid users of the intervention?

How notifications, e.g. prompts and reminders, can be used to increase the engagement with DBCIs has been explored in a variety of studies. When examining how and if prompts can promote engagement with digital health interventions, it was found that they have a small-to-moderate positive influence on engagement [1].

The content of the prompts in this case included suggesting intervention content, reminding users to complete sections, providing feedback, and BCTs such as social support, feedback on behavior, social reward, prompts and cues, feedback on outcome of behavior, and instruction on how to perform the behavior. The prompts in this case were mostly delivered in the form of emails which were sent to the users once a week at most. Studies which used prompts early in the users contact with the intervention was found to be more likely to engage its users [1]. However, it has also been found that while the use of an intervention progresses, users are less likely to respond to prompts [2, 8].

2.2.3 Push notifications in DBCIs

Freyne et al. [8] explored how push notifications can be used as prompts to complete tasks in a weight management application.

They found that this was an effective method to increase the number of tasks done by users and engagement with the application, at least in the short term. Another insight was that they were able to deliver three daily prompts—in the form of push notifications—without annoying the users too much. However, the effectiveness of the prompts declined over time in line with the overall decline of interaction with the application. It has also been shown that the frequency of using an application is correlated with the likeliness to respond to a prompt, i.e. low use frequency is often a sign of low prompt interaction [2]. Furthermore, a variety in both content and timing of push notifications are needed in order to keep users engaged [2, 8]. When Bidargaddi et al. [2] investigated the impact of push notifications that either contained tailored suggestions (e.g. perform activity X or Y today) or tailored insights (e.g. you have an opportunity to achieve something), they found that—in general—suggestive content lead to higher application interaction. However, this changes with the frequency of using the application. Insights lead to increased engagement for frequent users, while suggestions lead to the opposite [2].

4

2.3 Gamification – Competition and Cooperation

An additional method to achieve higher user engagement is to apply gamification [9, 11, 28, 40]. For example, Pechenkina et al. [28]

demonstrated that the introduction of a gamified education application can be used to raise academic results of students by increasing engagement and retention with learning activities.

Gamification can be defined as “the use of game design elements in non-game contexts” [6]. Some game elements—particularly extrinsic rewards, i.e. external rewards, such as points or badges—

may on the other hand undermine the user’s pre-existing intrinsic motivation, leading to the user becoming less likely to engage in a target behavior [27].

2.3.1 Competition and cooperation between users

Cooperation and competition have long been two commonly occurring elements in game design with separate strengths and weaknesses [29]. Nicholson [27] describes in A Recipe for Meaningful Gamification that engagement has two definitions in this context, whereof one is social engagement, which raises the question how players are supposed to interact with each other.

Should they compete or cooperate—or perhaps both—and is it to be done in groups or individually? Perski et al. [30] also mentions, when discussing social support features to enhance engagement, that features in a DBCI which support competition can motivate increased user engagement. However, promoting competition may encourage engagement for some types of players, but also run the risk of discouraging other types [30].

In addition to be more suitable, and increase engagement, for certain kinds of people, incorporating elements of competition and cooperation into a gamification system can influence the player in other aspects. It may, for instance, influence user enjoyment, user participation, and increasing the likelihood that the user recommends the system [25]. Competitive game elements may also have different effects between female and male users. It has been found that, in general, males increase their performance in competitive environments, while there is no such effect on females [5, 41]. For instance, competitive learning strategies in mathematics have demonstrated to be more effective for boys than for girls—however cooperative learning strategies were better for both genders [19]. A study has shown that competitive play led to higher positive emotional responses for males, but not for females [18], and the same pattern is seen when looking at the effectiveness of competitively-framed performance feedback in a crowdsourcing application [13].

When Morschheuser et al. [25] compared cooperative, competitive, and inter-group competitive gamification in a crowdsourcing platform, they found that adding a cooperative element made users more likely to recommend crowdsourcing platforms in general.

However, they were less engaged with the application—spending less time in the app and participating less in the crowdsourcing—

than those who competed. Competition however, had a stronger positive influence on participation in crowdsourcing, engagement with gamification features, and user enjoyment, when compared to cooperation. Further, in the realm of education and learning

technology, studies have shown that gamified elements positively influence students' performance in learning mathematics, specifically when combining competition, narrative and adaptivity to capability [16]. Collaborative elements were not as successful regarding performance. However, Ke & Grabowski [17] found in their study that although there were no significant difference in performance between students who have been gameplaying either competitive or cooperative in math education, students who gameplayed cooperative developed a more favourable attitude towards the subject at hand.

2.3.2 Gamification, competition, and cooperation in DBCIs

Similarly to crowdsourcing and education, differences in competitive and cooperative gameplay in digital interventions which aim to change the user’s behavior have been investigated. In a study aimed to explore how the effectiveness of exergames—i.e.

games which combine physical exercise with game elements to promote behavior change—is influenced by competitive versus cooperative gameplay, Marker & Staiano [21] found that cooperative exergamers had higher motivation which in turn promoted longer sustained play, self-efficacy, and adherence when compared to those who played competitively. When designing a gamified system for sustainable consumption, two advocated requirements for an effective intervention and increased engagement are enabling social interaction and collective action [14]. A particularly effective way of achieving this is to introduce within-group cooperation and/or between-group competition. This adds a novel opportunity for users to socialize. When working collectively towards a goal together in a group the effects of the effort becomes more salient and a single action is no longer

“perceived as a drop in the ocean” [14], thus adding to motivation and higher engagement.

3. METHODS

3.1 Push Notification Design

The content of the push notifications was to a great extent based on results from previous studies [2, 8, 31, 36], apart from additionally encouraging cooperation or competition. For instance, they were tailored, in timing and content, to current active deeds, e.g. “Lunch soon? Time to do [vegetarian-deed-name]” sent at 11.00 right before lunch. Some notifications were suggestions—like the aforementioned one—and others were insights such as “100 deeds done in your team! You guys have saved 188 kg CO₂e”. However—

based on the results of Bidargaddi et al. [2]—to make the notifications as suitable as possible to a wide range of users, they were mostly suggestive, and a few were insights. They also contained a number of BCTs, specifically feedback on behavior, feedback on outcome of behavior, and habit formation. Feedback on behavior is defined as presenting “informative or evaluative feedback” [22] regarding a behavior, e.g. “Wow, you’re on a streak.

You have done [habit-name] 4 days in a row!”. Additionally, feedback on the outcome is feedback which highlights the outcome of a behavior [22], e.g. “You have done 5 deeds, which means you have saved 38 kg CO2”. Habit formation is when the intervention

5 continuously prompts the user to repeat a behavior [22] and is for instance seen in the top notification in figure 3.

Furthermore, gamified elements and features—such as completing deeds, saving CO2, taking quizzes, creating habits, leaderboards, and sharing accepted deeds with other users—was prominent in every notification. What differed in the content was how actions were encouraged and how the message was formulated. To promote cooperation a notification could for example end with “Do one for the team”, and “Don’t fall behind” could be used to promote competition. In some cases, it was hard to promote competition.

This is for instance seen in the bottom notification in figure 3. To share a deed would not seem that beneficial in a competition. So, to encourage that, the notification highlighted the opportunity to maybe get another deed back and through that get assistance in moving forward in the competition. Except for these differences in competition and cooperation as much as possible was kept the same in the content. Thus, the only changing factor was the cooperative and competitive theme of the push notifications.

Figure 3. Some examples of push notifications used in the evaluation. The one at the top promoting cooperation and the

other two competition.

3.2 Evaluation

To evaluate any differences in influence on engagement from the push notifications a part of Deedsters user base was divided into 3 test-groups—2 of them receiving notifications that either promoted within-group competition (group A) or within-group cooperation (group B) and the last group was a control group (C) which received no push notifications. Since the push notifications encourage actions within a team of players, it was important that as many users as possible in a team received the same kind of notifications, i.e.

were in the same test-group. Therefore, instead of randomly assigning users to test-groups, entire teams, for the most part, were randomly assigned to the groups. Since users were able to join

several teams, the first team the user joined determined its test- group. On average, 86% of users in each team (with > 1 active user) were in the same test-group. The average team size on the application was 8,4 users (N = 2353; SD = 8,9; max = 1150; min = 1). However, only 2,6 users on average per team—from 171 different teams—was active, i.e. had opened the application, during the evaluation period (SD = 2,1; max = 19, min = 1). Users were required to have started at least one session during the evaluation period in order to be a participant of the evaluation. Otherwise, users who no longer use the application, or do not even have the application installed on their smartphone anymore, would have been part of the evaluation results. Group A had 265 users (82 female, 45 male, and 138 unspecified), group B 206 users (59 female, 36 male, and 111 unspecified), and group C 280 users (108 female, 50 male, and 122 unspecified). Because users themselves had to enter their gender into the application, and not every user did, there was a loss of gender information—resulting in a high number of “unspecified” genders in the groups. Each user received a maximum of 3 push notifications per day—depending on e.g.

recent activity by the user, what deeds has been accepted and not finished, and team activity—over the span of 19 days. This is the same daily amount as used by Freyne et al. [8]. Most of the notifications were sent out at 3 different daily time slots, one at 09:00, 11:00 and 18:00. There was a set of different possible notifications at each time slot and it was randomized which to send, thus offering some variety in the notifications sent. Notifications that notified the user of a reached goal or that the team is close to a goal was sent out directly when the right conditions were met, e.g.

when the 100th deed is completed in a team all users in that team will be notified immediately.

3.3 Data – Tracking and Analysis

During the evaluation user behavior was tracked in the application to assess the level of engagement. To determine the extent of the engagement, as defined by Perski et al. [30], the specific behaviors tracked were session count, session duration, total time in application, levels done, and amount of deeds done, shared, and accepted. This was also a similar approach as taken by Pham et al.

[31] and Morrison et al. [23] when exploring push notifications’—

and by Alkhaldi et al. [1] when looking at email prompts—impact on intervention engagement. A session was defined as a minimum of 10 seconds of interaction with the application and a session has ended at the time of the last event when no other events have been detected in the application for 30 minutes. This means that if a user leaves the application for 20 minutes and then returns, it is only registered as one session. Interaction with the push notifications, e.g. click rate and remove rate—which would be useful to determine their effectiveness—could not be tracked due to technical limitations of the application. Hence, the effectiveness of the push notifications is decided based on the aforementioned tracked behaviors, e.g. a difference in session count for users having received notifications compared to users who have not.

Moreover, in this case, a deed performed by a user is seen as a target behavior of the intervention. All other behaviors are seen as the extent of engagement with the intervention. Any observed differences in engagement between the test-groups was deemed

6 significant by performing variance analysis (ANOVA) and t-tests.

Due to the differences in sample sizes between the groups, the effect size was calculated using Hedges’ g.

3.4 Ethics and Privacy

Privacy concerns were taken into account during this evaluation.

Since users’ behavior within the application was tracked quite closely, and the behavioral data may be regarded as sensitive, all data was anonymized. Hence, no information could be tied to a specific individual user. Furthermore, all gathered behavioral data was only used to answer the thesis’ research questions, not utilized in any other way, and was not shared with any third party.

4. RESULTS 4.1 Engagement

As displayed in table 1, some discernible differences between the notification-receiving groups and the control group are apparent.

Users in both group A (M = 4,50; SD = 10,23) and B (M = 3,49;

SD = 9,04) had a significantly higher session count (p < 0,001 and p = 0,007 respectively) when compared to users in group C (M = 1,7; SD = 3,22). The effect size (Hedges’ g) between group A and C was 0,37, and 0,28 between group B and C. Although group A

Table 1. Comparison of user engagement metrics between test-groups.

Metric Group Mean SD

Session count

A 4,50 10,23

B 3,49 9,04

C 1,7 3,22

Session duration (seconds)

A 364 697

B 415 639

C 423 786

Total time in app (seconds)

A 1917 5250

B 1789 4758

C 1031 1634

Chapters completed

A 1,58 4,35

B 1,28 3,62

C 2,07 4,71

Deeds accepted

A 3,67 10,13

B 2,88 8,74

C 4,18 10,61

Deeds shared A 0,049 0,23

B 0,0097 0,09

C 0,036 0,27

Habit-deeds enabled

A 0,53 1,39

B 0,19 0,66

C 0,23 1,02

had a higher session count than B, any statistical significance could not be determined (p = 0,32). The largest difference in average session duration is seen between group C (M = 423, SD = 786) and A (M = 364, SD = 697), but is however only almost statistically significant (p = 0,06).

Just as in session count, the notification-receiving groups—A (M = 1917, SD = 5250) and B (M = 1789, SD = 4758)—spent significantly more time in the application (p = 0,01; g = 0,23 and p

= 0,04; g = 0,23 respectively) than the control group—C (M = 1031, SD = 1634). The difference in total time spent in application between group A and B is not statistically significant. Users in group C completed more chapters on average than both A and B, but only the discrepancy between B (M = 1,28; SD = 3,62) and C (M = 2,07; SD = 4,71) was determined to be significant (p = 0,04;

g = 0,18). Group A (M = 0,049; SD = 0,23) shared the most amount of deeds on average, closely followed by C (M = 0,036; SD = 0,27), and B (M = 0,0097; SD = 0,09) shared the least amount—a significant difference was found between group A and B (p = 0,01;

g = 0,22). Group A (M = 0,53; SD= 1,39) also enabled significantly more habit deeds than both group B (M = 0,19; SD = 0,66; p <

0,001; g = 0,30), and C (M = 0,23; SD = 1,02; p = 0,004; g = 0,24).

No significant differences were found between the groups regarding the number of deeds accepted.

Figure 4. Average number of started intervention sessions per user each day in all test-groups, aggregated into 4 intervals, of

the evaluation.

Figure 4 indicates that users in group A started marginally fewer sessions as the evaluation progressed, peaking at on average 0,27 started sessions per user between day 1-5, and dropping down to 0,22 between day 16-19. However, the decrease was not statistically significant. The same trend is visible in group B, although the session count is quite stable during the first 15 days of the evaluation and only dropping the last 4 days. Nor this change was significant (p = 0,1). Group C on the other hand saw a significant (p < 0,05) increase between day 1-5 and 6-10 followed by no other significant differences.

4.2 Target Behaviors

The target behavior of the intervention is the performance of sustainable deeds, whereof there are two kinds: one-time deeds

0 0,05 0,1 0,15 0,2 0,25 0,3

1 – 5 6 – 1 0 1 1 – 1 5 1 6 – 1 9

AVERAGE

DAYS

A B C

7 (which are just called “deeds”) and habit-deeds (which are deeds that the user has completed and then made a repeatable deed in the application, hence “habit-deeds”). Although the differences are not statistically significant, users in group A completed on average the most deeds (M = 1,91; SD = 4,42), followed by users in group C (M = 1,5; SD = 3,82) and B (M = 1,49; SD = 4,55) who performed approximately the same amount of deeds, see table 2. A considerable and significant discrepancy is visible between the notification-receiving groups—A (M = 4,51; SD = 13,52) and B (M

= 2,95; SD = 15,09)—and the control group (M = 0,42; SD = 2,76) in the number of habit-deeds completed, p < 0,001 in both cases. In terms of effect size, g = 0,42 between A and C, and g = 0,25 between B and C. However, the difference between group A and B, although noticeable, was not significant (p = 0,25).

Table 2. Comparison of performed target behaviors between test-groups.

Metric Group Mean SD

Deeds completed

A 1,91 4,42

B 1,49 4,55

C 1,50 3,82

Habit-deeds completed

A 4,51 13,52

B 2,95 15,09

C 0,42 2,76

As the evaluation progressed users in group A significantly decreased their average number of completed deeds per day from 0,16 day 1-5 to 0,058 day 6-10 (p = 0,015), see figure 5. A large increase is present between day 11-15 and 16-19 for group A but is only almost statistically significant (p = 0,063). Neither Group B nor C saw any significant changes in completed deeds.

Furthermore, no significant changes are apparent in the average amount of habit-deeds done by the test-groups during the evaluation, as seen in figure 6. The graph does however indicate an increase in completed habit-deeds for users in group A (p = 0,14).

Figure 5. Average amount of deeds completed per user in all test-groups each day of the evaluation.

Figure 6. Average number of habit-deeds completed per user in all test-groups each day of the evaluation.

4.3 Gender

As seen in table 3, some significant differences are present between male and female users regarding intervention engagement.

Especially within group A, but also between the test-groups. When looking at session count, male users (M = 12,09; SD = 17,86) in group A started significantly more sessions (p = 0,008; g = 0,65) than female users in the same group (M = 4,41; SD = 6,63). The male population in group A also started more than double the number of session than males in group B (M = 5,89; SD = 10,05), however the difference is marginally not significant at p < 0,05 (p

= 0,053; g = 0,42). Regarding total time spent in the application, even though male users in group A (M = 4184, SD = 8953) approximately spent twice the amount of time in the application compared to female users in the same group (M = 2054, SD = 3554) and males in group B (M = 1886, SD = 3076), no statistical significance was found (p = 0,13 and p = 0,11 respectively). Female users spent close to the same amount of the time in the application between both notification-receiving groups. The only significant difference in the number of completed chapters was between male and female users in group A (p = 0,03; g = 0,33), where the female user group (M = 3,63; SD = 6,76) completed more than their male counterpart (M = 1,71; SD = 3,2).

Multiple discrepancies are visible in the number of habit-deeds enabled. Male users (M = 1,53; SD = 2,11) in group A enabled more than female users (M = 0,7; SD = 1,5) in their group (p = 0,02; g = 0,48) and male users (M = 0,31; SD = 0,82) in group B (p < 0,001;

g = 0,73). Although female users in group A enabled more habit- deeds than both females and males in group B, the difference was determined not significant (p = 0,12 and p = 0,07 respectively). The closest statistically significant difference between the genders in group C was in the number of times males and females viewed the leaderboards. Male users (M = 1,9; SD = 5,73) viewed a leaderboard almost four times more than female users did (M = 0,49; SD = 1,42; p = 0,09; g = 0,41). This difference is however much larger—and significant (p = 0,027; g = 0,57)—in group A where males (M = 7,67; SD = 20,04) viewed a leaderboard almost ten time more than their female counterpart (M = 0,83; SD = 1,91).

No difference is seen between male and female users in group B.

And even though male users in group A viewed the leaderboards more times than males in group B and C, it was not a significant 0

0,05 0,1 0,15 0,2

1 – 5 6 – 1 0 1 1 – 1 5 1 6 – 1 9

AVERAGE

DAYS

A B C

0 0,05 0,1 0,15 0,2 0,25 0,3

1 – 5 6 – 1 0 1 1 – 1 5 1 6 – 1 9

AVERAGE

DAYS

A B C

8 difference (p = 0,17 and p = 0,068 respectively). Nor is the higher number present in the female population of group B when compared to group A and C statistically significant (p = 0,26 and 0,16 respectively).

Table 3. Engagement comparison between males and females, within and between test-groups.

Metric Group Gender Mean SD

Session count

A M 12,09 17,86

F 4,41 6,63

B M 5,89 10,05

F 5,44 10,80

C M 3,02 5,20

F 2,25 3,32

Total time in app (seconds)

A M 4184 8953

F 2054 3554

B M 1886 3076

F 2172 4358

C M 1423 2396

F 1285 1789

Chapters completed

A M 1,71 3,20

F 3,63 6,76

B M 2,92 4,87

F 2,46 5,06

C M 3,16 3,66

F 3,44 6,49

Habit-deeds enabled

A M 1,53 2,11

F 0,70 1,50

B M 0,31 0,82

F 0,39 0,79

C M 0,38 0,99

F 0,30 1,13

Leaderboard views

A M 7,67 20,04

F 0,83 1,91

B M 2,83 10,48

F 2,20 9,21

C M 1,90 5,73

F 0,49 1,42

When looking at target behaviors (table 4) there are no significant discrepancies in the number of done deeds within the test groups, however, one is found between the groups. Female users in group A (M = 4,15; SD = 6,78) did more deeds than female users in group C (M = 2,09; SD = 3,86; p = 0,015; g = 0,36). Some more

differences can be seen in the number of completed habit-deeds between genders. The male users in group A (M = 13,64; SD = 20,3) completed more than the female users (M = 3,96; SD = 13,18) in their group (p = 0,005; g = 0,60) and more than the male users in group B (M = 4,17; SD = 10,67; p = 0,009; g = 0,57).

Table 4. Comparison of performance of target behaviors between males and females, within and between test-groups.

Metric Group Gender Mean SD

Deeds completed

A M 2,64 3,22

F 4,15 6,78

B M 3,11 6,86

F 3,17 6,01

C M 2,74 4,14

F 2,09 3,86

Habit- deeds completed

A M 13,64 20,3

F 3,96 13,18

B M 4,17 10,67

F 6,20 25,58

C M 1,34 5,87

F 0,45 1,86

5. DISCUSSION

This thesis set out to examine how push notifications are able to positively influence the user engagement of a mobile digital behavior change intervention—a gamified smartphone application that aims to reduce its users CO2-emissions—and how said influence may impact the effectiveness of the intervention, i.e. will a change in engagement lead to a change in the likeliness of the performance of a target behavior. The following sections will examine how the results of the evaluation manage to answer the aforementioned research questions.

5.1 Influence on Engagement 5.1.1 Notification vs. no notification

A significant influence from the push notifications has been observed in many aspects of the user engagement, particularly in the amount of started intervention sessions and total time spent in the intervention. On the other hand, in a few aspects—session duration, accepted deeds—no influence is apparent between the test-groups. These results are in line with previous studies, for instance Pham et al. [31] also found significant effects of push notifications on session count and total time spent in application but detected no difference in session duration. The effect sizes between the groups in session count are however only small to medium, being 0,37 between group A and C and 0,28 between B and C.

Although not considered large, these effects are much more substantial compared to the effects of Pham et al. [31] which saw an effect size of 0,08 between the experimental group and control group in session count, and 0,05 in total time spent in the

9 application. Both notification-receiving groups in this thesis saw an effect size of 0,23 in total time spent in the application when compared to the control group. Even though non-social push notifications tend to be considered less important by users and therefore less interacted with, the results have confirmed their effectiveness in increasing the likelihood of users engaging with the application that sent the notifications. This may indicate that the timing and frequency of the notifications were well suited to the promoted behaviors and not too interruptive and, additionally, that the content—the BCTs, suggestions, reminders, and competitive and cooperative encouragement—was successful in increasing engagement and performance of target behaviors.

It is hard with the current evaluation results to draw any conclusions about if the increased level of engagement may remain as more time passes. Previous studies have found prompts, specifically push notifications [2, 8, 31], to be effective short term, but then lose their effectiveness over time as seen in a decline in notification interaction and intervention engagement. Additionally, extrinsic rewards—which the push notifications to some degree promote, e.g. completing deeds to climb the leaderboards—are also regarded to not be effective for long-term behavior change [27]. The results of this thesis do indicate that the session counts of the notification- receiving groups are slowly declining, but not significantly. On the other hand, the number of performed target behaviors did not show any signs of diminishing during the evaluation, especially not for group A. It is also worth mentioning that the success of the push notifications is largely dependent on the DBCIs content and features, since that is what the notifications convey information about. If the DBCI is not updated with new content for its users, it is not inconceivable that users will lose interest in the intervention, regardless of having push notifications reminding them about habits or promoting features. And, additionally, if the content and features does not build up—or impair—users’ intrinsic motivation, any long-term effects are improbable. Hence, it could be argued that it is not the notifications’ task to keep users engaged over longer periods of time. It is rather the responsibility of, continuously developed, intervention content and features. The push notifications’ task is then to increase user engagement with said content and features. If notifications which promotes competition, and hence extrinsic rewards, are the best way to achieve this is however still questionable.

5.1.2 Competition vs. cooperation

It was on the other hand harder to distinguish any significant differences in influence between the two notification-receiving groups—group A, which received push notifications promoting within-group competition, and group B, which received push notifications promoting within-group cooperation. No significant discrepancy was detected in session count, session duration, or total time in application. The only differences were found in the number of shared deeds and habit-deeds enabled, of which group A did the most. Interestingly enough, making the competitive aspects of the DBCI more salient, instead of the cooperative, increased the number of shared deeds by the users—a seemingly cooperative act.

This indicates that users in group A engaged more with intervention features than users in group B, which is in line with the results of

Morschheuser et al. [25]. Morschheuser et al. [25] however also found a difference in total time spent in the application, that is not apparent in the results of this evaluation. Furthermore, the composition of the teams may favor group A, since it is not implausible that it is more effortless to compete with strangers rather than cooperating with them. And that it is just as easy to compete with friends as it is cooperating with them. Therefore, competition may be more effective regardless of there being any existing relationships between the users in a team or not.

More differences between group A and B were however found when dividing and analyzing the groups based on gender. Firstly, the session count of male users in group A were more than double that of male users in group B. Although not significant at p < 0,05 (p = 0,053), it is a substantial difference with a medium effect size (g = 0,42). They also spent more than twice as much time in the application, however only at p = 0,11. The most significant difference, and largest effect size (g = 0,73), is seen in the amount of enabled habit-deeds by males in group A when compared to those in group B—which indicates a stronger commitment to perform more target behaviors. These results point towards competition being a better carrot than cooperation for increasing engagement with male users, which is very much in line with previous research [5, 13, 18, 41]. Furthermore, the results also indicate that promoting competition is more effective for males than for females in encouraging engagement. Male users in group A had a significantly higher session count, enabled more habit- deeds, and viewed leaderboards more than female users in the same group, further confirming results of previous studies, especially that of Huang et al. [13]. Interesting to note is that they completed less chapters than female users in their group, this could indicate that the notifications’ influence also affected how the intervention is used. Instead of progressing further in the journey, which is not required to actually advance in the competition, it seems like male users shifted their focus to turning deeds into habit-deeds and continuously completing them. Thus, further advancing in the competition. This does however mean that they are not exposed to as much of the interventions’ content. In group B, on the contrary, the engagement levels between female and male users are noticeably even in all metrics.

5.2 Influence on Target Behaviors 5.2.1 Notification vs. no notification

While no noticeable discrepancy in the amount of finished deeds was present between the test-groups, the push notifications made a vast difference in the amount of completed habit-deeds. Users in group A and B completed 8 and 7 times more habit-deeds respectively than users in group C. Worth noting is that group B and C on average actually enabled roughly the same amount of habit-deeds, but B proceeded to outperform C in completing them.

This clearly indicates that push notifications are an effective tool in prompting habitual behavior. This supports the findings of Freyne et al. [8] that push notifications are effective in reminding users to complete tasks. However, due to the shorter duration of this thesis’

evaluation it can neither support nor deny the claim that push notifications are a short-term solution to prompting behavior [8]. It

10 could also be argued that the increase in user engagement followed by the push notifications is effective engagement, as defined by Yardley et al. [43]. A rise in engagement was observed together with more target behaviors being performed. A causal relationship between the events is nonetheless hard to establish, or for that matter which event is the cause and which is the effect. However, it cannot be determined if the engagement increase will result in lasting habits for the users, which can be viewed as a wished-for outcome of the DBCI.

5.2.2 Competition vs. cooperation

No significant differences were detected between group A and B in the amount of deeds and habit-deeds completed. Meaning that it cannot be determined if there is a difference in influence on the performance of target behaviors between the push notification strategies. Users in group A did however complete a noticeable amount more deeds (28% more) and habit-deeds (53% more) than users in B, which gives an indication that group A may have performed better. When looking at specific genders, a more significant disparity is apparent. Male users in group A completed more than four times more habit-deeds than males in group B. This confirms that not only does promoting competition significantly increase in-app engagement for male users, when compared to cooperation, it also increases performance of target behaviors.

These results, and the fact that group A males also completed more than four times more habit-deeds than group A females, reveals that the large—however not statistically significant—difference between group A and B regarding completed habit-deeds is mostly due to hyper-engaged male users. In a strict effectiveness- increasing sense, it could further be more effective, engagement- wise and regarding performed target behaviors, to frame more content in the intervention competitively only for male users.

5.3 Method Discussion

The results of the evaluation did suffer from the short time frame of the evaluation. This is seen in e.g. habit-deeds done where a 50%

disparity could not be deemed significant at p < 0,05. The shorter evaluation period also made it impossible to draw any conclusions about the persistence of the engagement increases. Another shortcoming is the loss of gender data of roughly 50% of the users.

This is especially worrisome since the influence of the push notifications significantly differed depending on the users’ gender.

However, if the user has not shared their gender when creating an account in the application, there is currently no way of adding that information at a later stage. The data loss did in turn lead to smaller sample sizes and hence the risk of fewer significant results. The evaluation also lacked any data regarding interaction with the push notifications, such as click rates or dismiss rates, which would be useful in the analysis. This was however due to a technical limitation and, much like the gender data issue, not much could have been done about it.

The decision to only look at engagement as user behavior, in contrast to a subjective experience, is arguably beneficial for the results and the context of the thesis. First off, behavior can be collected passively through the application while experiences would have to be actively shared, articulated, and quantified by the

user—either through e.g. surveys or interviews—which ensures that all data is collected from all the users. This, in turn, leads to the largest sample size possible and more statistically significant results. Observed behavior is also a completely objective measure and would not yield any dishonest or erroneous information, which could be possible when instead measuring subjective experience.

This does however not rule out that subjective experience would be an appropriate measure in a different evaluation setting. Although, by not measuring the users’ subjective experience, and the fact that no interaction with the push notifications was measured, this evaluation cannot determine the users’ attitude toward the notifications.

5.4 Future Research

The research done in this thesis could be extended by additionally examining push notifications that encourage increased engagement by promoting between-group competition and cooperation, instead of within-group. It could also be interesting to explore the effectiveness of the utilization of different BCTs in push notifications, since some are used in this evaluation, but the influence of specific ones is not assessed. Considering competitively framed notifications were so successful in increasing engagement with male users, it could be useful to look at how other aspects of DBCIs could be framed competitively to increase engagement. Conversely, it could also be interesting to look at how cooperative and competitive encouragement could be used as a mix in push notifications, if that could have an even more extensive influence on users, and if gender would be less of a factor in the influence. Furthermore, the influence on engagement as a subjective experience by the push notifications could be studied as a complement to the current results. Alternatively, explore how push notifications could be used to build intrinsic motivation to foster long-term behavior change, and if competitive framed content has any place there. Lastly, the same, or a similar, evaluation could also be conducted in another context, e.g.

behavioral change interventions within healthcare. This would further determine similarities and discrepancies in the effectiveness of the current push notification strategies between multiple contexts.

6. Conclusion

This thesis set out to explore push notifications’ ability to positively influence, i.e. increase, user engagement in a gamified mobile digital behavior change intervention that aims to reduce its users’

CO2-emissions. It was also examined whether there was a difference in influence by notifications that either encouraged engagement by promoting within-group competition (received by group A) or within-group cooperation (received by group B).

Additionally, it was examined if the gender of a user played any part in the influence of the two different notification strategies.

Firstly, it was determined that utilizing push notifications resulted in a significant increase in some aspects of engagement, particularly in session count and total time spent in the intervention.

They did not, for instance, increase the duration of the sessions.

Together with the engagement increase, a rise in the amount of performed target behaviors was observed by the users who received

11 push notifications, indicating that the increase in user engagement made the intervention more effective. Because of the shorter evaluation period, whether the increased levels of engagement would be maintained during a longer period of time could not be determined. Furthermore, there was no significant difference in the amount of performed target behaviors between group A and B, and only smaller—yet significant—differences in engagement (amount of shared deeds and enabled habit-deeds). However, more significant discrepancies were detected when looking at gender.

Male users in group A had a significantly higher session count and enabled, and completed, significantly more habit-deeds than male users in group B and female users in group A. Any differences between female users in the two groups was not detected.

Consequently, it is concluded that competitively framed notifications have a stronger influence on engagement and in the performance of target behaviors in male users. These results do furthermore confirm that gender is a considerable factor in the extent of the push notifications’ influence on engagement and target behaviors.

Financing

This thesis was carried out as part of the project ‘Designing digital technologies for supporting energy-related behavior change in the kitchen‘, funded by the Swedish Energy Agency, project number 48099-1

7. REFERENCES

[1] Alkhaldi, G., Hamilton, F. L., Lau, R., Webster, R., Michie, S., and Murray, E. 2016. The effectiveness of prompts to promote engagement with digital interventions: a systematic review. J. Med. Internet Res. 18, 1, Article 6 (Jan. 2016), 14 pages. DOI: https://doi.org/10.2196/jmir.4790

[2] Bidargaddi, N., Pituch, T., Maaieh, H., Short, C., and Strecher, V. 2018. Predicting which type of push notification content motivates users to engage in a self-monitoring app.

Preventive medicine reports 11 (Sep. 2018), 267-273. DOI:

https://doi.org/10.1016/j.pmedr.2018.07.004

[3] Comber, R., Thieme, A., Rafiev, A., Taylor, N., Krämer, N., and Olivier, P. 2013. BinCam: Designing for engagement with Facebook for behavior change. In IFIP Conference on Human-Computer Interaction (IFIP '13), September 2-6, 2013, Cape Town, South Africa, Springer, Berlin,

Heidelberg, 99-115. DOI: https://doi.org/10.1007/978-3-642- 40480-1_7

[4] Couper, M. P., Alexander, G. L., Maddy, N., Zhang, N., Nowak, M. A., McClure, J. B., and Johnson, C. C. 2010.

Engagement and retention: measuring breadth and depth of participant use of an online intervention. J. Med. Internet Res. 12, 4, Article 52 (Dec. 2010), 11 pages. DOI:

https://dx.doi.org/10.2196%2Fjmir.1430

[5] Croson, R., and Gneezy, U. (2009). Gender differences in preferences. J. Econ. Lit. 47, 2 (Jun. 2009), 448-74. DOI:

https://doi.org/10.1257/jel.47.2.448

[6] Deterding, S., Khaled, R., Nacke, L., and Dixon, D. 2011.

Gamification: Toward a Definition. In 2011 Annual Conference on Human Factors in Computing Systems (CHI '11), May 7-12, 2011, Vancouver BC, Canada, 4 pages.

[7] Eysenbach, G. 2005. The law of attrition. J. Med. Internet Res. 7, 1, Article 11 (Mar. 2005), 9 pages. DOI:

https://doi.org/10.2196/jmir.7.1.e11

[8] Freyne, J., Yin, J., Brindal, E., Hendrie, G. A., Berkovsky, S., and Noakes, M. 2017. Push notifications in diet apps:

influencing engagement times and tasks. Int. J. Hum–

Comput. Int. 33, 10 (Mar. 2017), 833-845. DOI:

https://doi.org/10.1080/10447318.2017.1289725 [9] Hamari, J., Koivisto, J., and Sarsa, H. 2014. Does

gamification work? --A literature review of empirical studies on gamification. In 2014 47th Hawaii international

conference on system sciences, January 6-9, 2014, Waikoloa, HI, USA, IEEE, 3025-3034. DOI:

https://doi.org/10.1109/HICSS.2014.377

[10] Harder, H., Holroyd, P., Burkinshaw, L., Watten, P., Zammit, C., Harris, P. R., and Jenkins, V. 2017. A user- centered approach to developing bWell, a mobile app for arm and shoulder exercises after breast cancer treatment. J.

Cancer Surviv. 11, 6 (Dec. 2017), 732-742. DOI:

https://doi.org/10.1007/s11764-017-0630-3

[11] Hedin, B., Lundström, A., Westlund, M., and Markström, E.

2017. The energy piggy bank—A serious game for energy conservation. In 2017 Sustainable Internet and ICT for Sustainability (SustainIT '17), December 6-7, 2017, Funchal, Portugal, IEEE, 1-6. DOI:

https://doi.org/10.23919/SustainIT.2017.8379794

[12] Hedin, B., Katzeff, C., Eriksson, E., and Pargman, D. 2019.

A Systematic Review of Digital Behaviour Change Interventions for More Sustainable Food Consumption.

Sustainability 11, 9 (May 2019), 23 pages.

[13] Huang, N., Gu, B., Burtch, G., Hong, Y., Chen, L., Wang, K., and Lan, W. 2017. Designing Effective performance feedback notification systems to stimulate content contribution: evidence from a crowdsourcing recipe platform. In Proceedings of the 50th Hawaii International Conference on System Sciences (HICSS '17), January 4-7, 2017, Waikoloa, Hawaii, United States of America, 10 pages. DOI: https://doi.org/10.24251/hicss.2017.175 [14] Huber, M. Z., and Hilty, L. M. 2015. Gamification and

sustainable consumption: overcoming the limitations of persuasive technologies. In Hilty L., Aebischer B. ed. ICT innovations for sustainability, Springer, Cham, Switzerland, 367-385. DOI: https://doi.org/10.1007/978-3-319-09228- 7_22

[15] Irvine, A. B., Russell, H., Manocchia, M., Mino, D. E., Glassen, T. C., Morgan, R., and Ary, D. V. 2015. Mobile- Web app to self-manage low back pain: randomized controlled trial. J. Med. Internet Res. 17, 1, Article 1 (Jan.

2015), 21 pages. DOI: https://doi.org/10.2196/jmir.3130

12 [16] Jagušt, T., Botički, I., and So, H. J. 2018. Examining

competitive, collaborative and adaptive gamification in young learners' math learning. Comput. Educ. 125 (Oct.

2018), 444-457. DOI:

https://doi.org/10.1016/j.compedu.2018.06.022

[17] Ke, F., and Grabowski, B. 2007. Gameplaying for maths learning: cooperative or not?. Brit. J. Educ. Technol. 38, 2 (Mar. 2017), 249-259. DOI: https://doi.org/10.1111/j.1467- 8535.2006.00593.x

[18] Kivikangas, J. M., Kätsyri, J., Järvelä, S., and Ravaja, N.

2014. Gender differences in emotional responses to cooperative and competitive game play. PloS one 9, 7 (Jul.

2014), 16 pages. DOI:

https://doi.org/10.1371/journal.pone.0100318

[19] Kolawole, E. B. 2008. Effects of competitive and cooperative learning strategies on academic performance of Nigerian students in mathematics. Educational Research and Reviews 3, 1 (Jan. 2008), 33-37.

[20] Looyestyn, J., Kernot, J., Boshoff, K., Ryan, J., Edney, S., and Maher, C. 2017. Does gamification increase engagement with online programs? A systematic review. PloS one 12, 3 (Mar. 2017), 19 pages. DOI:

https://doi.org/10.1371/journal.pone.0173403

[21] Marker, A. M., and Staiano, A. E. 2015. Better together:

outcomes of cooperation versus competition in social exergaming. Games Health J. 4, 1 (Feb. 2014), 25-30. DOI:

https://doi.org/10.1089/g4h.2014.0066

[22] Michie, S., Richardson, M., Johnston, M., Abraham, C., Francis, J., Hardeman, W., and Wood, C. E. 2013. The behavior change technique taxonomy (v1) of 93

hierarchically clustered techniques: building an international consensus for the reporting of behavior change interventions.

Ann. Behav. Med. 46, 1 (Mar. 2013), 81-95. DOI:

https://doi.org/10.1007/s12160-013-9486-6

[23] Morrison, L. G., Hargood, C., Pejovic, V., Geraghty, A. W., Lloyd, S., Goodman, N., and Yardley, L. 2017. The effect of timing and frequency of push notifications on usage of a smartphone-based stress management intervention: an exploratory trial. PloS one 12, 1 (Jan. 2017), 15 pages. DOI:

https://doi.org/10.1371/journal.pone.0169162

[24] Morrissey, E. C., Corbett, T. K., Walsh, J. C., and Molloy, G.

J. 2016. Behavior change techniques in apps for medication adherence: a content analysis. Am. J. Prev. Med. 50, 5 (May 2016), 143-146. DOI:

https://doi.org/10.1016/j.amepre.2015.09.034 [25] Morschheuser, B., Hamari, J., and Maedche, A. 2019.

Cooperation or competition–When do people contribute more? A field experiment on gamification of crowdsourcing.

Int. J. Hum-Comput. St. 127 (Jul. 2019), 7-24. DOI:

https://doi.org/10.1016/j.ijhcs.2018.10.001

[26] Mummah, S. A., King, A. C., Gardner, C. D., and Sutton, S.

2016. Iterative development of Vegethon: a theory-based

mobile app intervention to increase vegetable consumption.

Int. J. Behav. Nutr. Phy. 13, 1, Article 90 (Aug. 2016), 12 pages. DOI: https://doi.org/10.1186/s12966-016-0400-z [27] Nicholson, S. 2015. A recipe for meaningful gamification. In

Reiners T., Wood L. ed. Gamification in education and business, Springer, Cham, Switzerland, 1-20. DOI:

https://doi.org/10.1007/978-3-319-10208-5_1

[28] Pechenkina, E., Laurence, D., Oates, G., Eldridge, D., and Hunter, D. 2017. Using a gamified mobile app to increase student engagement, retention and academic achievement.

International Journal of Educational Technology in Higher Education 14, 1, Article 31 (Aug. 2017), 12 pages. DOI:

https://doi.org/10.1186/s41239-017-0069-7

[29] Peng, W., and Hsieh, G. 2012. The influence of competition, cooperation, and player relationship in a motor performance centered computer game. Comput. Hum. Behav. 28, 6 (Nov.

2012), 2100-2106. DOI:

https://doi.org/10.1016/j.chb.2012.06.014

[30] Perski, O., Blandford, A., West, R., and Michie, S. 2017.

Conceptualising engagement with digital behaviour change interventions: a systematic review using principles from critical interpretive synthesis. Transl. Behav. Med. 7, 2 (Jun.

2017), 254-267. DOI: https://doi.org/10.1007/s13142-016- 0453-1

[31] Pham, X. L., Nguyen, T. H., Hwang, W. Y., and Chen, G. D.

2016. Effects of push notifications on learner engagement in a mobile learning app. In 2016 IEEE 16th International Conference on Advanced Learning Technologies (ICALT '16), July 25-28, 2016, Austin, TX, USA, IEEE, 90-94. DOI:

https://doi.org/10.1109/ICALT.2016.50

[32] Pielot, M., Vradi, A., and Park, P., 2018. Dismissed! a detailed exploration of how mobile phone users handle push notifications. In Proceedings of the 20th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI ’18). Association for Computing Machinery, New York, NY, USA, Article 3, 1–

11. DOI: https://doi.org/10.1145/3229434.3229445

[33] Pinder, C., Vermeulen, J., Cowan, B. R., and Beale, R. 2018.

Digital Behaviour Change Interventions to Break and Form Habits. ACM Trans. Comput. -Hum. Interact. 25, 3, Article 15 (June 2018), 66 pages. DOI:

https://doi.org/10.1145/3196830

[34] Revenäs, Å., Opava, C. H., Martin, C., Demmelmaier, I., Keller, C., and Åsenlöf, P. 2015. Development of a web- based and mobile app to support physical activity in individuals with rheumatoid arthritis: results from the second step of a co-design process. JMIR Res. Protoc. 4, 1, Article 22 (Feb. 2015), 14 pages. DOI:

https://doi.org/10.2196/resprot.3795

[35] Roberts, A. L., Fisher, A., Smith, L., Heinrich, M., and Potts, H. W. 2017. Digital health behaviour change interventions targeting physical activity and diet in cancer survivors: a systematic review and meta-analysis. J. Cancer Surviv. 11, 6