IN
DEGREE PROJECT
MEDIA TECHNOLOGY,
SECOND CYCLE, 30 CREDITS
,
STOCKHOLM SWEDEN 2020
User Engagement Metrics in
Story Focused News Articles
BEATA VON GROTHUSEN
KTH ROYAL INSTITUTE OF TECHNOLOGY
User Engagement Metrics in Story Focused News Articles
Beata von Grothusen
KTH Royal Institute of Technology
Stockholm, Sweden
beatavg@kth.se
SAMMANFATTNING
Nyhetsartiklar som fokuserar på berättande är en ny typ
av nyhetsartiklar som innehåller fler visuella och interaktiva element, utvecklade för att engagera en
yngre publik för digitala nyhetssidor.
Användarengagemang har tidigare definierats som det “emotionella, kognitiva och beteendemässiga kontakten mellan användaren och resursen”, och olika mätetal
används för att mäta användarengagemanget hos läsarna
av nyhetsartiklar som fokuserar på berättande. Däremot
finns det ingen tidigare forskning på vilka av dessa mätetal som beskriver användarengagemang på bäst
sätt. Den här studien har därför som mål att ta reda på
vilka mätetal som borde användas vid mätning av användarengagemang för nyhetsartiklar som fokuserar på berättande, genom att intervjua läsare av tre olika artiklar och jämföra deras engagemangsnivå med uppmätta mätetal.
Resultaten visar att 2 av de 3 artiklarna kan anses
engagerande enligt definitionen, och mätetalen som de
båda har gemensamt är ett högt genomsnittligt scrolldjup, låg nivå av studsar och höga siffror för sidvisningar. Studien drar därför slutsatsen att en
kombination av dessa tre mätetal beskriver
användarengagemang på bästa möjliga sätt. Dessutom
har båda de engagerande artiklarna ett stort antal bilder,
gallerier och videor jämfört med den icke engagerande artikeln, vilket indikerar att visuella element av olika slag är ett vinnande koncept för historieberättande artiklar.
ABSTRACT
Story-focused news articles are a different type of news
articles, containing more visual and interactive
elements, developed in order to engage a younger
audience for online newspapers. User engagement has
been defined as the “emotional, cognitive and
behavioral connection between a user and a resource”,
and different metrics are used to track the user
engagement of the readers on these articles. However,
there is no prior research on which of these metrics
describe user engagement in the most accurate way.
This study therefore aims to find out what metrics to use
when measuring user engagement on story-focused
articles through interviewing readers of three different
story-focused articles and compare their engagement levels with actual metric values tracked.
The results show that two out of the three articles can be
considered engaging according to the definition, and the
metrics they both have in common is high values of
scroll depth, low values of bounce rate and high values
of page views. The study therefore concludes that a
combination of these three metrics describes user engagement in the most accurate way possible.
Furthermore, both the engaging articles have a large
number of images, galleries and videos compared to the
non-engaging article, which indicates that visual
elements in different forms are a winning concept for
story-focused articles.
Author Keywords
User Engagement; Generation Z; User Research; Digital News; Story Focused News Articles
INTRODUCTION
Generation Z (people born between 1995 and 2003), a
part of the population that is growing older, is used to a
different type of news reading than traditional news
reading. The majority of Generation Z internet users
(58.7%) say they get their news from social media [19],
which is something traditional newsrooms therefore
have to adapt to. The Norwegian newspaper Verdens
Gang (VG) has implemented a new type of online news
article which they call story-focused article, which
includes more visual elements, shorter text snippets and
interactive elements in order to engage a younger
audience. However, the metrics currently used to track
the user engagement in these articles are the same as in
traditional news articles, which leads to user researchers
not knowing what the metrics actually mean for user engagement in this type of news article.
Research on how to measure user engagement in more
traditional news articles has been made in the past [9,
16]. It mainly focused on how to make readers stay on
an article page. Similar research on story-focused
Generation Z becomes a bigger part of the online news readership and ultimately the driving force behind how online news are represented.
Time and resources have been put into developing and
implementing these story-focused articles at VG, but
also at other news brands within Schibsted, the media
house VG is part of. This has been done without fully
knowing what the outcome would be, and it is therefore
of interest to find out how to measure the user
engagement in story-focused articles and what metrics
are relevant. Some metrics are tracked on the story-focused articles today, but they are not always
precise and can be hard to interpret. This degree project,
in collaboration with Schibsted, aims to know how and
if to change this article format in order to achieve a
higher user engagement. The desired outcome will be a
model for user researchers and designers to use when
measuring user engagement within story-focused
articles. It will state what metrics to use to obtain the
most accurate and valid results. This will ultimately lead
to a way of measuring the impact of a story-focused
article, and how far it reaches, which will be useful for
everyone conducting user studies on online news since it
affects validity of the research. The model will also
become more important over time as story-focused and
similar articles become more common as news formats. This project´s research questions are “What metrics best describe user engagement in story-focused news
articles?”, which is followed by these sub-questions:
“What metrics exist?” and “How do certain elements in
the article affect user engagement?”
BACKGROUND
In order to understand the field of user engagement in
online services, and particularly in online news articles,
it is of importance to understand the research state-of-the-art within the area, the metrics commonly used in engagement research and the impact on engagement of certain elements in news articles.
Additionally, to understand the readers and the target
groups of story-focused articles, it is also important to
obtain information on Generation Z and their news
reading habits.
Generation Z
A common way to consider groups of people in research
is to divide them into age groups based on their
generation. Generation in this sense describes people
born in the same cultural and historical context and who
are therefore exposed to the same experiences and
historical events [15]. This leads to a similar
consumption behaviour for these individuals, which
allows marketers to define them as a target group [18].
Generally, Generation Z is defined by people born
between 1995 and 2003, however the exact years when it starts and ends are vague and other definitions can also be found in research.
What differs Generation Z from other generations is that
they have grown up with internet and smartphones,
which means they are so called digital natives [4, 11].
They spend a lot of time online and retrieve basically all
information from their smartphones [6], which has
ultimately led to a very short attention span with an
average of 8 seconds [7]. They therefore have developed
the ability to quickly sort through a lot of information
and understand complex visual imagery [7, 5], which
has led to visual content playing an important role in
catching the short attention span of Generation Z [20].
Other studies have also shown that they prefer visual imagery over text due to this [16].
User Engagement
When looking at previous studies on user engagement,
the definition can look a little bit different in different
cases depending on the angle of the study. One
user-centered study refers to user engagement as “the
quality of the user experience that emphasizes the
positive aspect of the interaction with an online service
and, in particular, the phenomena associated with
wanting to use that service longer and frequently” [3].
Stakeholders with a more business angle however,
consider user engagement as a way to create revenue
and therefore constantly seek new ways to keep the
users engaged by serving interesting content in an
attractive manner [2]. This is with the goal that the users
will engage with the advertisements since they range between 60% and 70% of the total revenue of an
average newspaper [12]. In this study however, the more
user-centered and design approach to user engagement
will be taken, and the definition made by Lalmas and Lagun in [9], being the “emotional, cognitive and
behavioral connection between a user and a resource”.
This is due to the nature of this particular project, where
the readers will be in focus and no measures on revenue
or advertisement clicks will be taken.
In a different study [11], Lehmann et al. find that the
main driving force for user engagement in online news
is the reader’s interest in a specific topic and the
availability, on the article page that is being read, of
links related to other articles on similar topics. However,
the definition for story-focused news reading that
Lehmann et al. are using is not the same as the reading
of the story-focused articles used in this study, since it
same topic from different news sources instead of the
reading of an article type. Because of this, they are
looking at articles from many different news sites,
whereas this study is focusing on VG and this particular
type of news article that the Schibsted employees call
story-focused article, a terminology widely used within
the Schibsted media group. Additionally, they are not
examining user engagement metrics to establish their
efficacy in measuring user engagement, but
investigating how engagement changes with a certain
reading behaviour. Nevertheless, the results might be of
interest for this study, since the presence of links to
related articles in the VG story-focused articles might
affect user engagement on these.
When it comes to metrics used in previous studies, a lot
of focus has been put on the user engagement metric dwell time, which is defined as the total time a user has
spent on a page [9]. It has proven to be a meaningful
and robust metric for user engagement in the context of
web search [1] and recommendation tasks [21],and has
some correlation with the article length and the presence
of rich content such as videos and photos [21]. One
study however, has shown that the dwell time metric has
great limitations, since it does not tell us, for example,
anything about the user’s attention on the page, and it is
impossible to see any attention patterns when comparing
it to pages with similar dwell time values [9]. Instead,
the study suggests viewport time as a key metric,
defined as the position of a web page visible at a given
time [9]. Other suggested metrics include gaze, which
has been proven to be a reliable indicator of interestingness of an article and has correlation with self-reporting engagement metrics [2]. Measuring gaze
requires expensive eye-tracking technology, and a
cheaper alternative proposed by [14] is mouse cursor
movement. It has been shown that mouse position is
often aligned with gaze position, leading to mouse cursor movement outperforming dwell time as an
engagement metric. Other studies mention highbounce
rate, the percentage of visitors who enter the site and
then leave without further action, as an indication on
low user engagement [8], and other metrics mentioned
are number of page views , number of unique users,
click-through rate and return rate [10].
The previously mentioned metrics are all considered objective measures, which include web analytics and
mapping of user actions. Measuring user engagement
also includes looking at subjective measures, which
involve self-reporting and interviews, and rely on the
user’s perception [2]. One study mentions affect as an
important subjective measure, and refers to the emotional mechanisms that influence our everyday interactions and can possibly act as the primary
motivation to sustain user engagement [13]. Another
paper suggests focused attention, a feeling of energized
focus and total involvement, often accompanied with
loss of awareness of the outside world and distortions in
the subjective perception of time [17]. The goal in this
degree project is to examine some of these objective and
subjective measures in order to establish which ones to
use when measuring user engagement in story-focused
articles.
METHOD
In order to have a comprehensive way of investigating
the research question, a combination of different
methods were used, producing results that are based on
both objective (data from tracked metrics) and subjective (interview answers) measures (see Figure 1).
Figure 1. Methods used in this degree project. Employee Interviews
The first part of the method consisted of interviewing
employees currently working at Schibsted within relevant fields. In total, two employees were
interviewed in person, both of them working as user
researchers in Sweden and Norway. The interviews had
understand user engagement, and what additional
metrics they think could be useful to measure it. This
then laid the ground for what metrics to evaluate and
compare with the results from the other parts of the
method.
Article Tracking Data
This part of the method consisted of observing some
selected story-focused articles and retrieving the tracking data Schibsted has been tracking for these articles. The articles were chosen depending on their
traffic and publication date, and the goal was to have
articles with high numbers of traffic to be able to
investigate that metric properly. In total, three articles
were chosen for the test, all published in December
2019 or January 2020. This was due to the importance
of the articles having had approximately the same
amount of time to generate metric values in order for
them to be comparable. Two of the articles had similar
word count, 3031 and 3663 words, whereas one was longer, 6488 words. The content of the articles was
varied. The first article tells the story of a boy who
disappeared very suddenly, the second is related to the
history of the Norwegian island Svalbard, and the third
one is about a family destiny after the Tsunami in
southeast Asia. What all of them have in common is the
the seriousness of the topics. The hypothesis was that
these articles would put the readers in the same sort of
mood, and that this would make them comparable.
All user events at Schibsted are tracked with the system
Pulse, which stores the tracking data in different
databases. Pulse data on VG user events was provided to
the author by Schibsted. It was then retrieved through
the database fetching system Snowflake and SQL queries. Metrics used were: page views, unique page views, amount of comments, dwell time, amount of share button clicks, bounce rate, and scroll depth. From
this data, the metric scroll depth over time could also be
computed which was also used as a metric in the study. The data was then imported and visualized using the program Amplitude. Here, the average numbers on scroll depth and dwell time were computed and
visualized and the other metrics simply visualized in
diagrams and graphs in order to better grasp the raw
numbers. This made it possible to form hypotheses on
what articles were engaging or not engaging, something
that would later be tested against the results from the reader interviews.
Reader Interviews
Reader interviews were conducted to be able to estimate
engagement levels of readers. They consisted of 12 semi-structured interviews with people aged between 19-53, all familiar with VG in beforehand. All interviews were held over the video conference system Whereby, since this study was conducted during the covid-19 pandemic. The participants were asked to
choose one of the articles and read it before coming to
the interview, and were all encouraged to read as much
of the article as they found interesting or entertaining.
The first part focused on understanding user
engagement level through asking questions on focused
attention and interest. This was partly inspired by the
method in [2], where these measures were proven to be
important identifiers of engaged readers. In addition,
questions about general interest and knowledge in the
relevant topic were present in the interviews, since this
can affect the engagement level. The second part of the
interview consisted of open-ended, qualitative
questions, focusing on what the participant remembered
from the article, impressions of the article and their likeliness to talk about it with others.
All reader interviews were recorded with the consent of
participants, which made it possible to transcribe them
manually afterwards. After transcription, the interviews were analysed through colour coding the answers into different categories, being positive, negative and neutral
answers. This way it was possible to count how many
readers were in each category for each question. The
articles could then be divided into being engaging or
less engaging, which could then lead to suggestions
about what metrics are the most useful to measure user
engagement in story focused news articles.
Information that came up in the interviews that was not
related to the research conducted in this study is not
presented in this report, but communicated to VG and
the rest of Schibsted since it might be of importance to
them.
RESULTS
In this section, the results of employee interviews,
article tracking data and reader interviews are presented.
Employee Interviews
The first employee interviewed was a user researcher at
the newspaper Aftonbladet who has done studies on user
engagement in the past, however never on story-focused
measured in Pulse, but that might be valuable that were
mentioned in this interview were scroll depth over time,
which gives you an idea of how fast someone is reading,
data points,which means measuring how many readers
reach a certain point on the page, and also recirculation,
which is the number of people leaving a page but
coming back to it later.
The second interview was held with a Norwegian user
researcher that has done user research on story-focused
articles for VG in the past, however never on engagement metrics. The results from this interview ended up being more focused on the different visual elements in the articles and how they might affect
engagement and could be used as a subjective metric,
for example amount of videos, length of text and
amount of images.
Both of the interviews laid the ground for what metrics
to investigate for each of the articles. Scroll depth over
time was chosen because it could be easily computed
from the tracking data. Additionally,amount of images
and amount of videos were counted manually.
Recirculation and data points were not used since they required more advanced technical resources, something
outside the scope of this master thesis. All the metrics,
apart from the manually counted onesamount of images
and amount of videos are summarised and defined in
Table 1.
Page views Number of times the article page has
been opened
Comments Number of comments on the article
Shares Number of times readers have shared
the article on social media Scroll depth Percentage of how deep into the
article page the average reader scrolls
Dwell time How much time the average reader
spends on the page
Bounce rate Percentage of visitors who enter the
site and then leave immediately Unique
page views
Number of individuals that have visited the article page
Scroll depth over time
Scroll depth divided by dwell time. How quick the scrolling was
Table 1. Metric definitions.
Article Tracking Data
The metrics that are automatically tracked for each
article in Pulse are: page views, shares, amount of
comments, average scroll depth, average dwell time, bounce rate, unique page views andscroll depth over time.The reason behind these metrics being used is that
they were the only ones accessed at Schibsted or
possible to compute manually. The specific numbers
that Pulse has tracked for each metric for each article
can be seen in Table 2, where all the numbers are from
the same day (10th February 2020).
The metric shares refers to the amount of shares on
social media an article has generated through the
“share” button on the article page.Dwell timerefers to
the average length of time a reader has spent on a page.
Scroll depth is also an average number and refers to how deep into the article an average reader scrolls before
leaving the page. Bounce rate refers to how many
people, out of the total amount of page views, have left
the page without further action such as scrolling. The
last metric was computed through dividing the scroll
depth with the dwell time which we call scroll depth
over time. For article one, this metric was 1.95% per minute, for article two 1.72% per minute and for article three 2.39% per minute.
Article One Article Two Article Three Page views 142 273 91 487 93 823 Comments 9 7 9 Shares 44 50 28 Scroll depth 43.4% 34.4% 45.5%
Dwell time 22.3 min 20.0 min 19.0 min
Bounce rate 22.7% 26.3% 23.2% Unique page views 75 181 62 440 66 971 Scroll depth over time 1.95% per minute 1.72% per minute 2.39% per minute
Table 2. Metrics for article 1, 2 and 3.
article, Figure 3 shows the average scroll depth for each article and Figure 4 the bounce rate for each article.
Figure 2. Amount of page views for each article
Figure 3. The average scroll depth for each article.
Figure 4. The bounce rate for each article.
In addition to these metrics, article one consisted of text
together with 17 moving images, which is a short
looping video without sound, 1 video being 1 minute
and 15 seconds long, 26 non-full screen images, and was 6488 words long. Article two consisted of text together with 16 non-full screen images, one image gallery (see Figure 5), one full screen image, and was
3663 words long. Article three had text and 2 vertical
image galleries (see Figure 6), 3 moving images, 2
autoplayed videos being 43 and 59 seconds long, 14
non-full screen images, one full screen image, and was 3031 words long.
Figure 5. Image gallery in article 2.
Figure 6. Vertical image gallery in article 3, where the user changes image and text through scrolling. Reader Interviews
Out of the 12 participants in the study, 5 chose to read
article one, 5 chose to read article two and 2 participants
chose article three. To be able to compare the results
from the different articles, the results for each article will be presented individually in this section.
Article One
None of the 5 participants that chose to read article one
answered that they were familiar with the content of the
article before reading it, but 4 out of the 5 participants
had the overall feeling that the article was interesting,
whereas the last one said that they “got bored in the end,
but was interested in the beginning” (subject 2). Overall
participants felt that it was interesting to see the whole
story and how it progressed, and that the article showed
different sides of the story. One participant mentioned
that “it is written in a way that keeps me alert and my
interest up” (subject 5). However, none of the
about the topic afterwards, since they all felt like they
got enough information in the article. Since article one
is following a case of a missing person, 2 participants
mentioned that they would have liked to know more if
additional information about the case became available
in the future. When asked what the first impression of
the article was, 3 participants said the headline was very
attractive, and that the moving background behind it
(see Figure 7) helped to enhance that feeling even more.
One of them added that the fact that the moving
background was there was one of the reasons they chose
to read that article, since it raised questions and curiosity
about why the boy was running. Three out of the 5
participants mentioned that the interactive elements were the thing that they remember the most about the article, such as the short videos, large images and interactive maps. One participant said “the interactive
elements really enhance the experience and it would not
have been as engaging without them. It gives a stronger
impression” (subject 4). Other participants mentioned that they mostly remembered the feelings the article
brought up while reading it, and the fact that the boy
“disappeared so quickly” (subject 4), as one participant
described it.
Figure 7. Moving background behind headline.
On the questions that were more related to the focused
attention, 4 out of 5 participants answered that they were
focused and unaware of their surroundings while they
were reading, although one of them found it “harder to
focus towards the end of the article” (subject 2). The last
participant said they found it hard to focus and had to re
read some parts of it after losing attention, but had good
focus compared to normal news articles. All of them
also answered that it felt like time went fast while they
were reading (the time they were reading varied
between 30-45 minutes). When asked what the length of
the article did to their overall interest in reading it, 3 out
of 5 participants thought it was good that it was long
and needed to be that long to get this reading experience. One participant even said “long articles seem more trustworthy” (subject 3), while another one
said “most of the space is taken up by videos and photos
so it doesn’t feel as long to read as you first think”
(subject 4). The other two participants both thought the article was too long, some details repeated too many times, which made them lose interest.
Article Two
Three out of 5 participants stated that they had some
interest or knowledge about the topic before. However,
all participants answered that they found the topic
somewhat interesting even if they did not know much
about it before, and especially not the historical aspect
about Svalbard. Four participants mentioned that they
wanted to find out more about the topic after finishing
it: one said “it would be nice with a follow-up article”
(subject 7), and another person mentioned they wanted to go and visit Svalbard after reading this article. Still
only 1 out of 5 participants said that they found the
article interesting the whole way through, and one
commented that “the amount of text was tiring” (subject
6). Due to the introduction of the article which consists
of a moving background (see Figure 8), some
participants said they expected more visual elements in
the rest of the article as well. Three participants also
mentioned that the article gave the impression of being
long and “well made” and one of them said “the article
looks expensive” (subject 8). In general, on the question
about what they remember the most from the article, the
answers were all related to the content of the article, and
none of the participants mentioned any specific visual
elements. Instead, participants remembered things like “history about Svalbard after world war one” (subject
7), “Norway’s relationship with Russia” (subject 9), and
“the importance of Svalbard strategically (subject 8)”.
Figure 8. Moving image background behind headline on article 2.
On the focused attention related questions, 4 out of 5
participants answered that it was hard to focus while
reading the article. One participant said “It was hard to
to be involved in the reading then.” (subject 6), while
another person said “It got pretty boring after a while,
which made it hard to focus. But I kept on reading
anyway” (subject 7). However, only one participant said that time went slowly while reading, while the rest thought time passed away quickly. All participants agreed that the article was very good, but on the
question on what the length did to their level of interest,
one participant mentioned that “it changed the way I
read it, I considered it more like a chapter of a book,
which made me keep my interest up” (subject 7), while
another one said “the topic made it interesting despite
how long it was, but if this article had a different topic I
wouldn’t have finished it.” (subject 8). Two participants
mentioned that they were more interested at the
beginning, but got tired of reading after a couple of
paragraphs. Article Three
Both participants that chose to read article three said
they had some knowledge about the topic from
beforehand, but had never heard of the specific family
the article describes. They both still found it interesting
to read and one of them said “I really enjoyed reading it,
it was easy to understand and very emotional” (subject
11). Despite this, both participants said they did not feel like finding out more about the topic afterwards, and
that it was covered well enough in the article. One of the
participants mentioned the headline being “very
dramatic” (subject 12), which added to the overall first
impression and caught their attention. The other participant said about the introduction of the article
which was a looped visual image (see Figure 9): “I love
how visual it is and how it makes it so easy to visualize
as if you were there” (subject 11). For article three, the
two participants remember the visual elements to a great
extent, where the short videos “puts you in their situation” (subject 11).
Figure 9. Moving image background behind headline on article 3.
Both participants generally had a high level of focused attention: one of them said “I forgot about my
surroundings a little bit while I was reading” (subject 12), and the other mentioned they felt absorbed and ignored things that were happening around them at the time. However, none of the two participants felt they lost track of time while reading, and they were both aware of the time they spent on reading the article.
DISCUSSION
The purpose of this study is to evaluate the validity of
different metrics used to track user engagement on
story-focused articles. This is done through interviewing
readers of three selected articles to estimate their level
of engagement in order to draw conclusions on what
metrics correspond to high and low levels of user
engagement. This method would of course give more
valid results if the study contained more participants in
the reader interviews and if it could be done on a wider
variety of articles, something that could not be done in
this study. The results could possibly also have been affected by the covid-19 pandemic, where people tend
to read a lot of news but mainly related to the pandemic.
This may have changed the way people read news
related to other topics, such as the articles used in this
study.
The interviews conducted were all semi structured in
order to let the participants express all their thoughts
and feelings about the articles. Having all the interviews
being more structured, for example answer questions with numbers on scales and check boxes, would
possibly have given results that were easier to analyse.
These kinds of results would then have been easier to
draw concrete conclusions from, but also missing the important aspects that could take part in the semi
structured interviews such as individual thoughts and
feelings of the participants. Therefore the results might
not have been as rich and complete with structured interviews.
Engagement of the articles
User engagement in this study is defined as the “emotional, cognitive and behavioral connection
between a user and a resource” [9], therefore an
engaging article should fulfill all these three
requirements to some extent. Article one fulfills the
emotional aspect very well since all participants
mentioned being emotionally affected in some way,
additionally the cognitive aspect can also be considered
fulfilled since all participants generally mentioned that
Behaviorally, all the participants chose to read the
whole article despite it being quite long, which indicates
a behavioral connection to the article. A majority of the
participants also considered themselves focused while reading and were not disturbed by their surroundings,
which also indicates a behavioral connection to the
article due to a high level of focused attention.
Generally, article one can therefore be considered an engaging article over all.
Just like in article one, all the participants that chose to
read article two read the whole article, which indicates engagement in the behavioral aspect. However, the results from the interviews show that these participants
were less focused on the article and more aware of their
surroundings while reading. This therefore indicates a
lower level of engagement on the behavioral aspect. Also, none of the participants mentioned having any emotional reactions to the article, which means the article was lacking in the emotional aspect of user
engagement as well. On the cognitive aspect however,
all the participants thought that the article was interesting, both participants that had knowledge about
the topic before and the ones that were new to it. This
means the readers were engaged in the cognitive aspect
of user engagement, but still this could not be
considered enough to be able to identify the whole
article as engaging in general.
The fact that only two people chose to read article three
makes the results less generalisable compared to the
other two articles. It also indicates a lower level of
engagement in the behavioral aspect of the definition.
However, both of them still read the whole article after
choosing it, which indicates behavioral engagement for
those people to some extent. They also expressed being
focused and absorbed while reading, something that
points to a higher level of behavioral engagement as
well. The emotional connection between the two readers and the article can be considered high since the two participants mentioned feeling emotional, but still enjoying reading the article. They also both mentioned
the article being interesting to read, something that can
both indicate a higher cognitive involvement, but not
enough to be wanting to find out more about the topic
afterwards. Based on the two interviews conducted on
article three, it can be considered an engaging article in
all three aspects of the user engagement definition.
However, as mentioned earlier, these two participants
might not be enough to draw confident conclusions on
the general engagement of this article.
Engagement connected to metrics
Considering that both article one and three are engaging
articles, it is of interest to look at what metrics have high
values for these articles. Since they both have higher numbers of page views than article two, this would
suggest that a large number of page views indicates a
high level of user engagement. For article three
however, the amount of page views is only marginally
higher than for article two, which would then mean that
no concrete conclusions can be drawn on this. One way
to go around this could be to compare the page views to
unique page views, which would mean that article three has the highest value since the unique page views make
up the biggest part of the total amount of page views. It
also means that article one has the lowest part of unique
page views to total page views and therefore that the
two engaging articles have opposite metric values in this
aspect. Drawing conclusions on whether page views and
unique page views as metrics can describe user engagement should therefore be done with caution. Another metric where we can see similar values for
article one and three, and has differences to article two,
is scroll depth. This would then suggest that a large
percentage value on scroll depth describes high user
engagement on that article. The same thing goes for the
metric bounce rate, where both article one and three
have low values compared to article two. This means
less readers have left the page without scrolling on these
articles, which indicates an interest for the article after
seeing the first view.
When it comes to the word count on the articles, article
one had nearly twice as many words as the other two.
The readers were still very engaged in this article which
would suggest that long articles generate user engagement, but since the other engaging article, article
three, had had a much smaller amount of words it is
hard to draw a conclusion on what the amount of words
does for user engagement. Also, dwell time does not
seem to drastically change with the amount of words,
since article all articles had similar average dwell times,
which indicates that readers might not read every word of the article but rather watch videos, images and other visual elements. The fact that all articles have similar values for dwell time indicates that this metric can not
be used to measure engagement which in turn leads to
scroll depth over time not being an accurate number to
track for engagement measures. The point with this
metric was to track how fast or slow a reader is reading
the article, something that might be misleading since it
contains. An article with a lot of text can for example
have quicker reading time than an article with a lot of
videos where the reader stays still on one point in the
article for a long time. This does however not
necessarily mean that one of them is more engaging
than the other, and it could also be a possible reason for
that dwell time seems to have had no effect on engagement with the readers.
The fact that the least engaging article (article two) has
the highest amount of shares on social media suggests
that this metric does not describe user engagement for story-focused articles. This might have to do with
certain groups of readers being more prone to share
news on social media in general, especially if it is about
a topic the reader has a special interest in which, as
mentioned in the results section, some of the
participants did for article two. When it comes to the
amount of comments each article had, article one and
three had higher numbers than article two, but the
differences are very small which makes it hard to draw
any concrete conclusions from that information. As mentioned in the introduction, the metrics
automatically tracked are not always precise and can be
misleading, which is why this project has been
questioning the relevance of them. For example, a deep
scroll depth does not necessarily say anything about
where the attention of the reader lies on the page, and
dwell time does not necessarily mean the reader has
been looking at the screen the entire time. Looking at
the results in general, they indicate that some of these
metrics are more valid than others, which also shows that not all qualitative data can be trusted.
Engagement connected to visual elements
Generally, the results point towards higher user
engagement with more visual elements in the article.
Article two, which according to the results is the least
engaging article out of the three, only consisted of one
image gallery and one full screen image, while both article one and three had either moving images, vertical galleries, videos and full screen images. Therefore there
seems to be a correlation between user engagement and
the amount of visual elements an article contains. Usage
of classic, non-full screen images, however, seems to
only have limited effect on user engagement, since
article two still contained 16 of these images, but still
can not be considered an engaging article in this definition. On the other hand, the other two articles also
consisted of the same type of images to a great extent,
only that other visual elements were added as well. It is
therefore safe to say that classic, non-fullscreen images
do not bring user engagement down, but rather keep it
on a consistent level. Future research
In the future, it would be interesting to explore different
aspects of what affects user engagement in online news
and how story-focused news articles change user
engagement. User engagement could be analysed on the
content of the articles instead of the visual elements, and
research on how positive or negative emotions affect
engagement is still to be explored. Research even more
related to this study that could be developed further is
analysing what makes readers leave articles at around
40% scroll depth. Is this where the content starts
repeating or are the visual elements fewer there? Many
different questions are still yet to be answered on this
topic.
CONCLUSION
The aim of this master thesis was to find out what
metrics best describe user engagement in story-focused
news articles. The results show that high values for the
metric scroll depth is directly connected to user
engagement, as well as low values for the metric bounce
rate. There are also indications that high values forpage
views indicate user engagement, and together with scroll depth and bounce rate, one can draw the conclusion that
an article is engaging for the readers. Therefore, the
combination of the previously mentioned metrics would
generally speaking be the best description of user
engagement in story-focused news articles. This can be
used to a great extent when analysing story-focused
articles, in order to establish how to develop new visual
components. The study also shows that visual elements that are not classic text or non-full screen images
contribute to user engagement for the article they are in,
which encourages further research and development of
visual elements to include in story-focused articles.
ACKNOWLEDGEMENTS
First of all, I want to thank my KTH supervisor Sandra
Pauletto for being a great support during the whole
process. Furthermore, a big thank you to Schibsted
legends Gaute Tjemsland, Hilde Skjølberg and Pernilla Danielsson for sharing their invaluable experience and
being of great help along the way. Thanks also to my
life long partner in crime Dan, always cheering on me
from the other side of the world. And last but not least,
thank you to my father Henrik, who suddenly passed
teaching me what is important in life. I know you would have been proud of me.
REFERENCES
1. Agichtein, E., Brill, E., & Dumais, S. (2019). Improving Web Search Ranking by Incorporating
User Behavior Information. ACM SIGIR Forum,
52(1), 11-18.
2. Arapakis, I., Lalmas, M., Cambazoglu, B., Marcos, M., & Jose, J. (2014). User engagement in online News: Under the scope of sentiment, interest, affect, and gaze. Journal of the Association for Information Science and Technology, 65(10), 1988-2005. 3. Attfield, S., Kazai, G., Lalmas, M., & Piwowarski,
B. (2011). Towards a science of user engagement (position paper). In WSDM Workshop on UMWA. Available at
http://ir.dcs.gla.ac.uk/~mounia/Papers/engagement.p df
4. Dolot, A. (2018). The characteristics of Generation
Z. e-mentor, 74(2).
5. Finch, J. (2015). ‘What is Generation Z, and what
does it want?’. FastCompany, May 4th. Accessed
January 31st 2020 from
https://www.fastcompany.com/3045317/what-is-gen eration-z-and-what-does-it-want
6. Google/Ipsos. (2016). The Mobile-First Mindset of Gen Z. Available at:
https://storage.googleapis.com/think/docs/GenZ_Insi ghts _All_teens.pdf#page=8
7. Hallowell, E. & Ratey, J. (2011). Driven to
distraction: Recognizing and coping with attention deficit disorder. New York, Anchor Books. 8. Kunz, L. (2011). Documentation in a Collaborative
World: What We've Learned. Proceedings of the
58th international conference on Technical Communication. Sacramento, California, USA. Available at:
https://summit.stc.org/wp-content/uploads/2011/05/p roceedings_2011-FINAL.pdf#page=14
9. Lagun, D., & Lalmas, M. (2016). Understanding and Measuring User Attention and Engagement in
Online News Reading. Proceedings of the Ninth
ACM International Conference on Web Search and Data Mining, San Francisco, California, USA.
Available at:
http://www.dcs.gla.ac.uk/~mounia/Papers/wsdm201 6.pdf
10. Lalmas, M., & Hong, L. (2018). Tutorial on Metrics of User Engagement: Applications to News, Search
and E-Commerce. Proceedings of the Eleventh ACM
International Conference on Web Search and Data Mining. New York, NY, USA. Available at: https://doi.org/10.1145/3159652.3162010 11. Lehmann, J., Castillo, C., Lalmas, M., &
Baeza-Yates, R. (2017). Story-focused reading in
online news and its potential for user engagement. Journal of the Association for Information Science and Technology,68(4), 869-883.
12. Manduchi, A., & Picard, R. (2009). Circulations, Revenues, and Profits in a Newspaper Market with Fixed Advertising Costs. Journal of Media Economics, 22(4), 211-238.
13. McCay-Peet, L., Lalmas, M., & Navalpakkam, V. (2012). On saliency, affect and focused attention. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’12).
Association for Computing Machinery, New York, New York, USA. Available at:
https://doi.org/10.1145/2207676.2207751
14. Navalpakkam, V., Jentzsch, L., Sayres, L., Ravi, S., Ahmed, A., & Smola, A. (2013). Measurement and modeling of eye-mouse behavior in the presence of
nonlinear page layouts. Proceedings of the 22nd
international conference on World Wide Web. New York, NY, USA. Available at:
https://doi.org/10.1145/2488388.2488471 15. Newman, N. (2019). Journalism, Media and
Technology Trends and Predictions 2019. Reuters
Institute for the study of Journalism. Available at: https://reutersinstitute.politics.ox.ac.uk/our-research/ journalism-media-and-technology-trends-and-predict ions-2019
16. Nielsen Norman Group. (2016). Designing for Young Adults (Ages 18-25). Available at:
https://www.nngroup.com/reports/designing-for-you ng-adults/
Information Science and Technology, 61(1), 50-69. Available at
https://doi-org.focus.lib.kth.se/10.1002/asi.21229 18. Pilcher, J. (1994). Mannheim's sociology of
generations: An undervalued legacy. (British
sociology). The British Journal of Sociology,45(3),
481-495.
19. Taylor, K. (2019). ‘The State Of Gen Z’. Business
Insider, July 1st. Accessed December 4th 2019 from https://www.businessinsider.com/gen-z-changespolit ical-divides-2019-7?r=US&IR=T
20. Twenge, J. (2017). iGen. Why Today’s
Super-Connected Kids Are Growing Up Less Rebellious, More Tolerant, Less Happy – and Completely Unprepared for Adulthood. New York, Atria Books.
21. Yi, X., Hong, L., Zhong, E., Nan Liu, N., & Rajan, S. (2014). Beyond clicks: dwell time for
personalization. Proceedings of the Eighth ACM
Conference on Recommender Systems. New York, NY, USA. Available at:
