Dashboard design and its relation to KPIs: A qualitative case study on a software company

(1)

Faculty of Technology Department of Informatics

Dashboard design and its

relation to KPIs

-A qualitative case study on a software company

Author: Christopher Berglund Author: Amar Tenic

(2)

Abstract

Monitoring key performance indicators (KPI) give practitioners immersive experience that is priceless when it comes to decision making and performance-enhancing in software companies. Used together with different tools that enable visualization of KPIs, users obtain big advantages that enable quick troubleshooting and detections of errors that could emerge in a product or a system. Many studies have been followed out in the field of data visualization using KPIs and digital dashboards. Still, the subject of KPIs providing valuable information to companies that are developing HR and payroll systems is relatively unexplored.

The purpose of this thesis has been to investigate how essential KPIs should be visualized on a digital dashboard using a case company that focuses on developing HR and payroll systems. To investigate the phenomenon, five different interviews were conducted, and a digital dashboard was developed. The interviewees that participated in the empiric data collection were employees stationed in different teams with various authorities and experiences in the field of dashboards and KPIs. Previous works in the field of data visualization indicates that KPIs can be used and presented in various ways. When presenting KPIs on a dashboard, there are different factors that are of big influence of how successful the visualization gets. There are no complete templates on how KPIs should be visualized, however there are guidelines on how a dashboard could be shaped. Something that previous works and different interviewees in the present study agreed on was that a dashboard should consist of 4 to 8 KPIs. Too many KPIs can decrease the chance of obtaining the advantages that monitoring and visualizing KPIs on a digital dashboard can bring. Something that emerged from the study is the importance of first identify what to visualize and then implement how.

Among the answers during the data collection, many interviewees found the number of users that were logged in on their products as something that would be useful to monitor. The reason for this was partly that the interviewees considered that by monitoring these numbers, it could increase the pride among the colleagues. The interviewees thought that it might boost morale among the employees if they visualized the many users of the products they are developing.

(3)

Abstrakt

Genom att övervaka nyckeltal (KPIer) kan användare erhålla fördelar som är ovärderliga när det gäller beslutsfattande och prestandaförbättring hos mjukvaruföretag. Tillsammans med olika verktyg som möjliggör visualisering av KPIer får användare stora fördelar som gör det möjligt att snabbt upptäcka fel som kan uppstå i en produkt eller i ett system. Många studier berör ämnet datavisualisering och digitala instrumentpaneler. Trots det här är ämnet KPIer, specifikt de som ger värdefull information till företag som utvecklar HR- och lönesystem outforskat. Syftet med den här studien har varit att undersöka hur viktiga KPIer bör visualiseras på en digital instrumentpanel hos ett företag som fokuserar på att utveckla HR- och lönesystem. För att undersöka fenomenet genomfördes fem olika intervjuer. En digital instrumentpanel utvecklades även som ett komplement till att svara på forskningsfrågan utifrån tidigare forskning och de empiriska resultat som framkom under studiens gång. Informanterna som deltog i den empiriska datainsamlingen var anställda på ett fallföretag och stationerade i olika utvecklingsteam. Informanterna hade sedan tidigare olika erfarenheter av att använda instrumentpaneler och KPIer. Tidigare studier inom datavisualisering indikerar att KPIer kan användas och visualiseras på olika sätt. När KPIer presenteras på en instrumentpanel finns det olika faktorer som har stort inflytande på hur framgångsrik visualiseringen blir. Det finns inga kompletta mallar för hur KPIer ska visualiseras men det finns riktlinjer för hur en instrumentpanel kan utformas.

Något som tidigare studier samt informanterna i den här studien enades om var att en instrumentpanel bör bestå av ett antal mellan fyra och åtta KPIer. Genom att använda för många KPIer så minskar chansen att få de fördelar som övervakning och visualisering av KPIer på en digital instrumentpanel kan ge. Något som den här studien belyser är vikten av att först identifiera vad som ska visualiseras för att sedan implementera hur. En av flera KPIer som ansågs värdefulla för informanterna var antalet användare som var inloggade i deras produkt. Anledningen till det här var delvis att informanterna ansåg att övervakning av användare skulle skapa ökad stolthet bland kollegorna på kontoret.

(4)

Key words

Digital dashboard, data visualization, key performance indicator (KPI), metrics, HR and Payroll systems.

Acknowledgments

To begin with, we would like to thank Visma Enterprise for allowing us to carry out this case study within their organization. We would also like to thank all five interviewees from the company, who took time from their workday to participate in our case study and contributed us with valuable information. Finally, we would like to thank Fisnik Dalipi for the guidance and input we received during the writing of the thesis.

Växjö, June 2020

(5)

1 Introduction

Information that is target-oriented and provides real-time monitoring over a product or service is essential for success in companies to further efficient production (Bracht, U et al. 2011). Few (2006) addresses that data visualization tools like dashboards are used as a single screen display that shows important information in real-time and can be used to read and monitor KPIs. By visualizing the right information through different technologies, it becomes easier to understand a whole factory or a specific production line. Järvenpää et al. (2015) argue that technologies like dashboards are likely to become the most common way of visualizing target-oriented and real-time information as digitalization replaces Excel-based production observation. Järvenpää et al. (2015) also argue that most companies use sophisticated execution systems where the KPIs often are monitored through dashboards.

The importance of monitoring KPIs through various technologies are continually increasing due to society being more information-dependent. Monitoring essential information can be used for various purposes. Reinking et al. (2020) describe that profit-oriented enterprises uses displays like dashboards to market their product, and according to Pestana et al. (2020), hospitals monitor important KPI values like heart rate and other essential and measurable information that ensures a patient's well-being. Measuring the current status using key performance indicators during situations of crisis is also crucial so that governments have the basis for prioritizing where resources need to be allocated. WHO (World Health Organization) is working for the United Nations on the mission to monitor data that describe disease outbreaks such as Cholera, Ebola, and COVID-19. The purpose of this is primarily to measure the size of the issue using key performance indicators together with a digital dashboard (World Health Organization 2020). This data is vital in the sense that it allows the organization and governments to see which areas are more infected. As a result of this, the monitored KPIs could serve as a basis for decision making that, in this case, could lead to restrictions being performed.

A case company that develops HR (Human Resource) and payroll systems was used in this study. By using a company as a business case, an analysis was performed to study which KPIs (Key Performance Indicator) that could give value. This study does also investigate how these KPIs could be visualized. To help answer this question, a digital dashboard was developed.

Background

Management of modern technology is rapidly changing. New tools and technologies are more accessible than ever, and businesses are continually striving to increase their efficiency. Failures that occur in applications could result in delays that increase the frustration for both employees and customers (Nagy et al. 2008). To solve these failures, companies need to address the root of the problem to determine what has gone wrong, which is rather time-consuming.

Systems gather big volumes of data every day and this data, combined with different tools and technologies, could be used to identify failures in an early stage. If used correctly, data visualization could be used to detect changes over time. Ljungberg and

(7)

Larsson (2012) argue that information needs to be available and should be visible. Gusnadi and Hermawan (2020) describe that displaying and monitoring useful core information has become more common amongst software companies that develops HR systems due to big changes in sales and in size of the applications that nowadays are developed.

The path of finding relevant literature studies has led to the conclusion that there are no specific guidelines for how a company that is developing HR and payroll systems could display essential information. The goal with this study has therefore been, to raise the issue and give a demonstration of how KPIs should be presented on a digital dashboard.

Problem formulation

Analysing data using different values can be a way to become productive and to measure dataflows (Gusnadi & Hermawan 2020). A direct result of this phenomenon is a large amount of research in the world of computer science and, more specifically in papers that cover different types of digital presentation tools and software. Despite the big interest in the subject and a large number of research articles and studies, none of them capture the perspective of a company based in the software industry with the focus on developing HR and payroll systems. There are also no studies that proclaim or mention how different KPIs that are useful for a company in the field of work previously described should be visualized.

Purpose & Research Questions

The purpose of this study is to gain an understanding of how KPIs should be visualized on a digital dashboard and to investigate which KPIs are valuable to an organization that develops HR and payroll systems. The research questions that the present work aims to investigate is the following:

How should KPIs be visualized on a digital dashboard?

o Which KPIs give value for a software company that develops HR and payroll

systems?

Scope/Limitation

In this study, the focus is partly to investigate how KPIs should be visualized through a digital dashboard. Different services were used to develop the dashboard, which can be found in chapter 4.2.1.

With the case company being a large organization with a broad product shelf, a limitation was imposed to only gather and visualize static test data as a demonstration of real data from a product. This study will not include interviews with all employees

at the case company. Only representatives of each development team (team leaders), working as software engineers, and software architects were asked to participate in the interview. The study focuses on gaining insight in which KPIs that could give value. The study also focuses on giving a proposition of how the KPIs identified should be presented on a digital dashboard.

(8)

Outline

This section gives a brief description of what information each chapter contains.

Chapter 2: This chapter contains previous works in the field of KPIs and dashboard

development.

Chapter 3: This chapter contains information regarding related works and describes

some categories which have been essential to understanding the subject. The categories are KPIs, data visualization, figures, and dashboard.

Chapter 4: This chapter describes the different methods that were followed during

the present work. The chosen scientific approach, project methods, and the frameworks that were used are described. The chapter also includes the technical process that was followed when developing a digital dashboard. The chapter also contains the analysis process that was followed during the collection of the empiric data. The chapter also include a description of how validity and reliability was followed to ensure that the empiric research achieved high quality.

Chapter 5: This chapter contains the empirical findings that were collected using the

different methods and tools presented in chapter 3. The findings are divided into three different categories which are: KPI selection, Visualization of KPIs and Pros, and cons of visualizing KPIs. The chapter also contain information about the final prototype that was developed based of the empirical findings.

Chapter 6: The empirical material collected in the present work will be evaluated

and discussed with the research area and different fields of work. The reason for this is to create a broader perspective and to see what differentiates and connects both sections. The contents of the dashboard prototype are also analysed.

Chapter 7: This chapter concludes thoughts and opinions regarding the study’s

result. The chapter also provides examples of future research to emphasize the importance of continued research on the subject.

(9)

2 Related Work

This chapter presents research related to this study and contains three categories. The first category describes what to consider when selecting appropriate KPIs to monitor. The second category explains the purpose of visualizing elements on a platform, and the last category contains previous studies on dashboard design and the mindset that should be followed during the creation process.

Selection of KPIs

The definition of a KPI is” A quantifiable measure used to evaluate the success of an

organization, employee, etc. in meeting objectives for performance.” (Oxford

Dictionary 2020). Hedin and Zander (2019) made a study that included a suggestion of which KPIs that would be most suitable to present based on different questions that were replied from different interviewees. The companies that were participating in the study were based in the construction industry. Therefore, the empiric data that were collected were used to describe the best solution for this specific industry only. Hedin and Zander (2019) used several methods to come to their conclusion. They used the qualitative approach to obtain the business angle of KPIs and what value they should bring. The authors later used case studies to obtain the results. Hedin and Zander (2019) states that the results they got helped them to draw the conclusion that there are KPIs that are commonly more suitable for different companies in the construction industry, such as pre-tax profit margin, cash liquidity, and chargeability.

However, it is difficult to find appropriate KPIs (Peral et al. 2017). A KPI that is associated with a specific company may not work for another company. Peral et al. (2017) describe that there is a lack of clear guidelines on how to define which KPIs you would like to visualize. There are different models to use when the aim is to find the KPIs that could give value. The structure of the models is clear, but the content is not (Peral et al. 2017). Therefore, it is essential to find knowledgeable people when working with KPIs. The most common way to identify suitable KPIs is by asking users and consumers about their needs (Peral et al. 2017).

Parmenter (2015) describes that the key to success with a KPI project is to have a well-trained in-house staff that works full time on the project. He also mentions that a mix of young and old individuals who has different characteristics is best suited to move the project in the right direction.

Purpose of visualizing core elements

Liff and Posey (2004) argue that visual management seeks to improve organizational performance. This is achieved by linking the organizational vision, its core values, objectives, and culture with other management systems and stakeholders.

Visual panels are visual elements that are composed of a platform. These visual elements appear in various kinds of literature but in different terms. Greif (1991), Imai (1997) and Tezel et al. (2016) call visual elements boards. Activity board is a term used by Kyokai (1996). Kattman et al. (2012) use the term panel, while Pauwels et al. (2009), Yigitbasioglu and Velcu (2012) use the term dashboards. In the present work, the term dashboard will refer to visual elements composed on a platform.

(10)

Yigitbasioglu and Velcu (2012), Gusnadi and Hermawan (2020) define the dashboard as primarily, an interactive and a visual tool that presents essential information. The authors also state that what the visualized information/elements should represent, is something the developer should determine together with the users. They argue that it is up to the company to, in an earlier stage, define the number of elements that intend to be visualized.

The purpose of visualizing different elements, according to Baalbergen (2020), is to simplify the information and make it easier to read. Also, the purpose of visualizing elements could be to achieve organizational goals. Baalbergen (2020) argues that tools used to show information allows different users to communicate, monitor, and identify various problem areas where intervention is needed. Pauwels et al. (2009) are also stating that monitoring is a core function of the dashboard. Different purposes and functions of visualizing elements are that it could enable planning and

communication (Pauwels et al. 2009). The authors refer the purpose of monitoring to

being able to execute performance analysis that makes it easier to obtain an overview of how well something is operating, for example, a team or a product. Planning refers to the identification of various strategies/objectives. An example of the purpose of communication refers to deliver information to different stakeholders, according to Pauwels et al. (2009).

As previously mentioned, the literature recommends that the dashboard should be used for monitoring, to communicate and to plan. In the present work, a digital dashboard has been developed with the recommendations stated earlier in mind.

Developing a dashboard

According to Bremser and Wagner (2013), the first step one must encounter when creating a digital dashboard is to evaluate the company's goals. A company's goal is often to increase their financial growth. Bremser and Wagner (2013) argue that this has led to more digital metrics that focus on the economy when developing a digital dashboard. A metric is a figure that measure results (Oxford Dictionary 2020). The authors also mention that this approach works for some companies but also that a mix with both financial metrics and other metrics are more suitable when designing a digital dashboard or visualizing data in general (Bremser & Wagner 2013). The authors state that the creation of a dashboard involves five activities. Figure 1 shows the contents within every stage in the development process using a flow diagram.

Figure 1. The process to follow when developing a dashboard (summarized from

Bremser & Wagner 2013).

Define the objective of the dashboard Define metrics and identify the content (KPI) Seek user input Create initial prototype Launch and monitor

(11)

It is important to choose proper metrics that are going to visualize the current state of a company, a product, or how well a team is solving different tasks. Grossmeier (2020) states that it is essential to know how metrics link to broader organizational objectives. Bremser and Wagner’s (2013) conclusion is that too many metrics can create confusion and therefore recommends that only 4 to 8 different metrics should be shown on a digital dashboard. Bremser and Wagner (2013) describe that one of the most common mistakes when creating a digital dashboard is to implement too many meaningless metrics. The idea is to develop a dashboard consisting of important KPIs of an organization or users, visualized through different kinds of metrics.

Bremser and Wagner (2013) also specify that different users have various needs. For example, users’ opinions often differ on questions like how often a dashboard should be refreshed. Bremser and Wagner (2013) argue that some users prefer data (KPIs) being refreshed in near real-time for some dashboards. Other users feel that other types of data, will not make sense to refresh more than once a day. Bremser and Wagner (2013) state that the users must be aware that the data latency issues might differ depending on how frequent the refresh rate is so that they know what to expect.

Bahl, Mccreadie and Stevenson (2007) made a study to be able to measure pharmacy costs. The authors outline that the aim when creating a new platform for data visualization should be to determine important KPIs. They explain that the first step in the development of their dashboard was to gather data obtained by different interviewees in health care. The collected data then got transformed into useful information about trends in drug use (Bahl, Mccreadie & Stevenson 2007). This information then helped to find the KPIs that were later displayed on the dashboard that was built. The study concluded that the dashboard provided an easier understanding of trends in drug use. Bahl, Mccreadie and Stevenson (2007) describe the visualized data as critical in formulating strategies to control drug expenses by each service. The mindset of first listening to the users and then move forward in the process that Bahl, Mccreadie and Stevenson (2007) used during the development of their dashboard was embraced in the present work.

(12)

3 Theoretical framework (Theory)

This chapter contains information regarding important topics of the study. The topics discussed in this chapter are key performance indicators (KPI), data visualization, and dashboards.

Key performance indicator

Key performance indicators (KPI) are becoming more important than ever (Hao et al. 2014). Gusnadi and Hermawan (2020) describe that institutions or organizations must be ready to face an era of digitalization when it comes to presenting information about a company, especially to see employee performance. Gusnadi and Hermawan (2020) describe that KPIs are measurement tools that primarily help to understand what is needed to achieve goals. KPIs can be used as different indicators that focus on certain aspects or parts of the performance in an organization. Hao et al. (2014) describe KPIs as a tool that makes a relationship between the performance of technical loops/components and high-level production qualities. It also provides data like production efficiency and data consumption, to name a few components.

According to Stricker et al. (2017), performance measurement with important KPIs could also be used as an instrument to detect changes or errors in production system performance. The reason for this is to be able to coordinate relevant countermeasures. Fanning (2016) state that a good understanding of what is important to an organization, is needed to be able to find essentail KPIs. The KPIs could also be used to identify fault diagnoses by monitoring them (Shen et al. 2015). It is vital to use the advantage of identifying the diagnoses to inform the process engineers whether the fault will influence the product quality reflected by KPI argues Gusnadi and Hermawan (2020). This case could apply to different companies that operate in these fields of work. Gusnadi and Hermawan (2020) describes that if some of the identified diagnoses are critical, it could result in dissatisfied customers and long-term financial losses.

Stricker et al. (2017) describe that the main challenge in planning KPI systems consists of determining relevant KPIs and that enough KPIs must be selected for valuable information measuring. Peral et al. (2017), on the other hand, state that it is vital to identify the relationships between KPIs characteristics and how valuable they are.

To keep a high efficiency, which leads to an organizational profit, KPI based monitoring plays an important role (Hao et al. 2014). Hao et al. (2014) describe that there is a continuously increasing interest in the KPI based monitoring and diagnosis techniques because of the profit and efficient work process that monitoring brings. Zhang et al. (2017) state that industrial KPIs could be categorized into three different groups:

1) Engineering

KPIs referring to the technical performance of something. For example, the quality of a product.

2) Maintenance

(13)

3) Economic

KPIs referring to business profit. For example, energy consumption.

Data visualization

Creating data visualization is challenging and requires different tools for analysis and development (Walny et al. 2020). Data can be visualized in a lot of different ways. Data is usually presented through plain text but can also be presented as figures, tables, and all sorts of diagrams (Yang 2013). The opportunity to see beyond the numbers makes data visualization an excellent tool for decision-making and to understand the effect of a new strategic approach. Yang (2013) argues that even if many different tables and diagrams can be combined, it is important to not lose focus on the main advantages of using such tools. This is something that was taken into consideration during the creation phase of the dashboard.

Depending on the aim and the data, different tools are better suited to the specific task than others. Yang (2013) mentions that figures and tables can be used when presenting complicated relationships, while the text is often used to describe the data of the structures. The use of proper tools increases the readability, which allows information to transfer to the reader without consisting of an enormous text structure (Yang 2013). Yang (2013) argues that graphical presentations are preferred when the purpose is to increase the overall engagement of any sort of analysis. Diagrams are a great tool to enhance the interest and motivation of people who have not been studying the numeric material of a table (Yang 2013). Walny et al. (2020) and Ljungberg and Larsson (2012) agree that the way data is presented is crucial for the enhancement of the comprehension.

Lempinen (2012) describe that colour is important when presenting information on a dashboard, besides choosing different charts. The process of visualization gets improved using colours, but if colours are overused, it will affect the user in their decision-making. Lempinen (2012) argues that careful use of colour is advised when the purpose is to visualize KPIs. Overusing colours can disorient the user, that is why a limited use of colour is required for the dashboard.

Kerzner (2017) describes that the placement of elements (KPIs) determine the outcome of how successful data visualization through dashboards become. The first phase should be to talk with users and identify what information is important for them. Kerzner (2017) argue that the placement of KPIs on the dashboard should be based of its importance. The majority of users read from left to right and start at the top. As a result of this, the KPIs could be placed in this order. The idea is that if the users only have a few seconds to view at a dashboard, their eyes will first catch the elements that are of most importance. However, Kerzner (2017) describe that the potential users have the last call on how to structure important KPIs on a dashboard.

Five different activities (Figure 1) to consider when creating a dashboard were described in the related work. The second activity is to define which metrics would be the most optimal to use. Data visualization projects often include collaboration between people with different skills (Walny et al. 2020). It must be good symmetry among these individuals to increase the chance to succeed with a project. Something that also is stated in the related work is that Bremser and Wagner (2013) argue that developers often make mistakes when creating a digital dashboard and make it

(14)

complicated by visualizing too many KPIs. The focus should primarily be on visualizing more easy-read metrics. Yang (2013) also describes that the metrics should be user-friendly and describes that there are various types of metrics to be used for different purposes. For example, tables are used to present raw data like summary statistics, and boxplots are a convenient way to depict the quantitative variables according to Yang (2013).

The main purposes of data visualization, which also were stated in the related work are to monitor (KPIs), communicate and identify various problems within a problem area (Yigitbasioglu & Velcu 2012). Useful metrics to visualize data with the purpose of monitoring are the Pie chart, Bar chart, and the Linear graph due to the simplicity and because they are easy to read (Yang 2013). As a result of this, different versions of the following metrics in chapter 3.2.1 were used to display KPIs on the dashboard that was developed for the purpose of the present study.

3.2.1 Figures

Three figures will be presented below.

Pie chart

Bertini et al. (2016) describe that by highlighting different aspects of a presentation, colours are often used to capture people’s attention. Pie charts are often consisting of different colours to represent the proportion of various categories (Yang 2013). Figure 2 displays an example of how a pie chart could be visualized.

Figure 2. Example of a pie chart.

Bertini et al. (2016) states that a pie chart is most useful if all categories combined to achieve a sum of 100%, even if it sometimes can be challenging to make comparisons between them. It could be hard to visually compare the sizes. To achieve a more comprehensive pie chart, percentage is often applied.

Bar chart

Bar charts consist of bars that are vertical and equal in size. This tool is often used when the purpose is to show changes in information (Vanderplas et al. 2020).

(15)

Figure 3. Example on a bar chart (Geckoboard 2020).

Figure 3 illustrates how a bar chart can be visualized on a dashboard. Pre-defined categories are often placed on the X-axis and their volume on the Y-axis. A bar chart is a valuable tool when comparing categories over time (Yang 2013).

Linear graph

Linear graphs are also used when comparing changes in information over time. Like the bar chart, it has one X-axis and one Y-axis and consists only with a single line that represents many different data points that are connected. An example of how a linear graph could be displayed on a digital dashboard is visualized in Figure 4.

(16)

A linear diagram makes it easy to identify trends that can be very useful in future decision-making (Yang 2013).

Digital dashboards

The usage of digital dashboards varies due to the community becoming more information depended, as previously mentioned in the present work. The primary usage is though still to provide valid information to end-users. Digital dashboards have dramatically increased in population during the last decade (Oeser et al. 2018). Organizations use dashboards to display uptime, errors, and other essential KPI values with the purpose to get more details between relationships (Grossmeier 2020). The purpose of dashboards is to simplify the way of finding comparisons and to have more insight into the findings (Dobraja et al. 2020). Bremser and Wagner (2013) describes digital dashboards as a tool that focuses on goals and objectives. An effective digital dashboard should also utilize visualization techniques and cues to engage a user in the information processing experience (Bremser & Wagner 2013). Digital dashboard usage is extensive, and organizations adapt the usage to their needs and the KPIs they find valuable to monitor, which makes the dashboard flexible to use (Yigitbasioglu & Velcu 2012). The structure, design, and simplicity of the dashboards are fundamental to gain advantages over competitors. Few (2006) describes the phenomenon:

“Most dashboards that are used in businesses today fail. At best, they deliver only a fraction of the insight that is needed to monitor the business.”. – (Few 2006, s.2).

The primary problem within the world of digital dashboards is poor data presentation (Grossmeier 2020). To maximize the potential of the dashboard, it must display the most valuable information (KPIs). Companies often invest in expensive data warehouses to access real-time reporting, but they struggle to structure the data into meaningful information that generates value (Grossmeier 2020). The need of techniques can be ascertained to enhance the process of adding value to data and to find important KPIs (Peral et al. 2017).

There are various types of useful designing tools that can help a developer to build a platform to display analytic data. While society is being more dependent on technologies based on big data, companies also see an opportunity to break into a new market. In early 2016, Amazon introduced a new solution to gather data on one single platform based on cloud computing (Amazon Web Services 2020). Amazon

QuickSight was one of the first technologies of its kind when it launched and is now

one of the biggest products in the field of cloud computing. After the success of QuickSight, more companies started to detect the value of displaying various kinds of data, and there are now numerous services that provide their customers with similar software. Examples of companies and their services are Google (Google Analytics) and Microsoft (Power BI).

(17)

4 Method

This chapter includes a brief description of the chosen scientific approach. The chapter does also describe the development process of the dashboard prototype. The chapter includes a description about the selection of interviewees, the empiric context, and an explanation of why interviews were used as a method to collect empiric data. The chapter ends with a discussion about how the validity and reliability

of the empiric workflow, and the findings were followed during the whole process.

Scientific approach

There are several ways to approach a specific problem and to gather scientific data. Creswell and Creswell (2018), Johannesen et al. (2020) argues that the most common methods used in conducted studies are the quantitative and qualitative approach. The qualitative approach is used to gather detailed (qualitative) data. The qualitative research method is also used by researchers to investigate the quality of social life using different interviewing options and tools (Holliday 2016). For this reason, the qualitative approach was chosen to be followed in the present work. The intention was to gather detailed data about different kind of topics and to understand every individual that participated in the study, and this became possible by using the qualitative approach. Creswell and Creswell (2018) also describe that a more open and welcoming interview brings more detailed data, which ultimately indicated that the methodology did fit perfectly for this study.

Developing the first prototype

A prototype of a digital dashboard, which is shown in Figure 5, was created as a complement to try to answer the first research question using the related works as references. The idea of developing the first prototype was to lay the groundwork for what a visual representation of KPIs might look like. The idea was to, after laying the foundations, be able to develop and gradually add information after the empirical studies were carried out and the results were analysed.

(18)

Figure 5. Illustration of the initial dashboard prototype.

Five different activities (Figure 1) to consider when creating a dashboard were described in the related work. These activities have been followed during the creation of both the initial (Figure 5) and final prototype (Figure 9). One of the key activities is to develop and to implement a design thinking mindset which has been implemented in the present work. The last activity of the five presented in the related work is to launch and monitor the dashboard (Bremser & Wagner 2013). This was also followed in the present work using Amazon CloudWatch as a monitoring service (3.2.1).

4.2.1 Framework and software

Before starting the development phase of the initial prototype, the potential tools that were going to be used were identified. After investigating different types of programming languages, the choice became to use C# due to its compatibility with the Amazon ecosystem.

Something that was understood early in the development process was that daily code execution would not be possible using a static, desktop computer. The reason for considering this was that previous works stating that it is important to determine how frequently the dashboard and the KPIs should be updated. As a result of this, a structure got implemented in Amazon Web Services that provided the tools needed. These tools were Lambda functions, S3-bucket and CloudWatch. An external service called Geckoboard was used to visualize the KPIs. This due to its compatibility with services provided by Amazon Web Services. Visual Studio was used as development environment because it supports the programming language C#.

4.2.2 Dashboard development process

Initially, a console application was created with the functionality to generate static objects in C# to JSON objects. The console application was created using Visual Studio as a framework. This application was then uploaded to Amazon Web Services using Amazon Tool Kit, which is an addon to Visual Studio. The console application was in this stage transferred from a static application to a cloud-based Lambda

(19)

function. When executed, this Lambda function automatically triggered the Amazon Web Service product called S3 bucket, which acts as a storage space for objects created by the Lambda function. The external service (not Amazon product) Geckoboard was used as a visualization service of the data that got into the S3 bucket, which previously was mentioned. The information comes to Geckoboard through a second Lambda function. It works as the link between Amazon Web Services and Geckoboard. An implementation of scheduled events using CloudWatch was made with the purpose of being able to manually adjust and set how frequently the first Lambda function would be executed.

This way of developing the dashboard was used during the creation of the initial prototype (Figure 5), as well as the final prototype that can be found in Figure 9.

4.2.3 Design thinking theory

Brown (2012) describes that the designers and developers today need to use their sensibility and to choose proper design methods to match user and customer needs. A way that encourages this approach is the design thinking theory. The method is commonly used to create ideas that better meet consumer needs and desires, according to Brown (2012). Design thinking is a discipline that also allows the practitioners to be more innovative since they are passing through different stages throughout the lifetime of their work (Brown 2012).

The first phase is called inspiration, according to Brown (2012). The reason for this is that circumstances could motivate the search for solutions. The next phase is called

ideation, which is used in the process of developing and testing ideas that may lead

to different solutions. The last phase is called implementation, and the purpose of the phase is to chart a path to place the product in an environment so it can be used by the user. The projects will loop back and forth through all these phases and particularly the first two argues Brown (2012). The reason for this is that ideas are refined, and new directions are taken (Brown 2012). A visual representation of the three phases can be found in Figure 6.

(20)

Figure 6. Illustration of the three phases of design thinking (adopted from Brown

2012).

Design thinking theory fits perfectly in the mission of the work because of the flexibility it encourages. After obtaining the tools needed for the development phase, a method was needed to make it easier for the stakeholders to have their say about the design, the necessary data being used, and the prototypes being created. The reason for this was to be able to ensure that the dashboard got the highest quality possible. The design thinking theory was used during both the initial and final developing process of the dashboard to obtain the highest quality possible. By following the theory, it made it possible to think more innovatively. By using the process, it encouraged to think outside the box and try different ways to come up with solutions. An example of adapting this theory and passing through the different phases of the theory was when various approaches were made to push static data objects to the first Lambda function (Chapter 4.2.2). The first way of pushing objects was made by constructing a form application in the programming language C#. The reason for this was that by using a form application, it enabled build-in functions that made it convenient to easily change different values. The form application made it possible to choose between 1 to 100 objects. The JSON-object data were then randomized. The second approach to push the objects to the Lambda function was a console application that was identical to the solution that later got implemented in the Amazon cloud.

The purpose of sending randomized, static objects that were not taken from a database was to make everything to work manually. The reason for the process was to, in the future, make it easy to change a couple of rows that enables a user to set how frequently the dashboard updates.

A second example of the design thinking theory being used practically in the present work was when the final prototype was developed. As the empiric data got obtained, the results and the KPIs that were found were successively implemented to the

(21)

dashboard. More information and details about the final prototype can be found in chapters 5 and 6. A visual illustration of the final prototype can be found in Figure 9.

Data collection & analysis

This chapter includes information about the selection of interviewees and the empiric context that was followed. The chapter also elaborates on the reason behind selecting the interviews as a method for data collection.

4.3.1 Selection of interviewees

The case company in this study has five different development teams. The individuals that participated in the study are representatives for their team (team leaders) and are working as software engineers and software architects. The individuals in this selection are responsible for their team and have daily conversations with other group members. By working closely with the rest of the employees they obtain a picture of what their opinions are, their needs, and their wishes. The interviewees have many years of experience and by using a selection of this kind, an assumption was that it could provide empiric data of high quality.

The interviews were conducted at different times with five different interviewees. The reason for choosing the specific number of interviewees is that Creswell (1998), Creswell and Creswell (2018) argue that 5-25 interviewees are optimal for qualitative research. The authors state that this sample size is large enough to obtain enough data saturation to sufficiently describe the phenomenon of interest and to address the research questions.

4.3.2 Empiric context

The empiric context, which was followed, was the intensive approach. Johannessen et al. (2020) describe that the context is optimal in studies where the purpose is to gain a better understanding of a subject or to capture descriptions of the phenomenon in its natural context.

4.3.3 Interview

The chosen method to collect the necessary data was semi structured interviews with the purpose of learning more about the company, its organizational needs, their thoughts about visualizing data, and how the design of the dashboard could be structured. The focus during the creation of the questions was to understand the organizational view and to collect empirical data that could help to answer the research questions. The interviews were recorded, and notes were taken during the whole interview. Later, the material got decoded and analysed. The interviews were conducted between April 8 and April 16 in the year of 2020 in consultation with the interviewees. A document regarding consent was sent to the interviewees well in advance of the interview taking place. This document can be found in Appendix 1. Creswell and Creswell (2018) argue that it is essential to provide information that reminds the interviewees about the purpose of the study. This was in mind during the creation of the appendix. The questionnaire that was followed during the interview can be found in Appendix 2. The interviews were performed using Google Hangouts. The reason for this was that the interviewees participating in the interviews were working from home.

(22)

Something that was stated in the related work section was that it is very important to first find out what information (KPIs) that should be presented before moving forward to implement the content on a dashboard (Yigitbasioglu & Velcu 2012). A common way of identifying suitable KPIs is by asking potential users Peral et al. (2017). As a result of this, the interviewees answered questions about the topic at the very beginning of the session. After that, the interviews continued with the interviewees answering other sections regarding the design and placement of information. Another reason for choosing the questions (Appendix 2) was to get a better understanding of the organization, the people that work there and their views on KPIs. The purpose was also to form an understanding of what the different stakeholders see in the different visual concepts that were developed (An example of this is the initial dashboard prototype). The questionnaire that was followed while conducting the interview gave answers that were used as guidelines when developing the dashboard. The dashboard eventually helped with answering the first research question, which was how the KPIs could be visualized.

The interviews that were conducted were semi structured interviews. Creswell and Creswell (2018) describe that there are several ways and theories to use while creating and performing an interview. The most common ones are the open interviews and structured interviews. Johannesen et al. (2020) argue that qualitative interviews are the most common way of collecting data in the qualitative scientific approach. An open interview has a small number of guidelines to help the interviewer remember the topics that should be discussed (Creswell & Creswell 2018). Creswell and Creswell (2018) portrays the semi structured interview as an approach similar to the open interview, but with a couple of limited answers. By using the semi structured interview, the interviewer is given the opportunity to both obtain static and qualitative answers Creswell and Creswell (2018). Therefore, the choice became to conduct a semi structured interview.

The materials that were left at the end of the data collection phase were the recorded interviews. Jacobsen (2017) describes that the qualitative analysis process could be divided into three separate categories, which were followed in the present work. The interview should be described as detailed as possible. Because of this, a detailed transcription was made of the recorded interviews. This is called a thick description, according to Jacobsen (2017), and this was the first phase that was entered during the analysis process. The second phase in the analysis process that was followed is called

systematization and categorization, and the purpose of this phase was to reduce data

that were found redundant. The reason for this is that the thick description is very detailed, and some data that had been collected were not relevant to the subject or the questions. The last phase in the analysis process followed in the present work is called

combination (Jacobsen 2017). The purpose of the phase was to interpret the data, so

it became readable. Johannessen et al. (2020) argue that the meaning of interpretation is to put the data you have got into a larger context. When the data was interpreted, exciting views on different subjects emerged, which gave answers that helped achieving a connection with the purpose of the work. The process followed is also presented graphicly in Figure 7.

(23)

Figure 7. Illustration of the qualitative analysis process that was repeated to analyse

all interviews (adopted from Jacobsen 2017).

Coding is mainly used in qualitative studies by researchers that perform interviews (Parameswaran et al. 2019). The execution of the coding in the present work has been conducted with the help of the categorization of data in tables and charts. However, the coding and categorization have taken place at several different times, and new ideas have emerged during the process of this study, which has created new perceptions that later got analysed.

Bremser and Wagner (2013) argue that gathering information from the user is an essential part in the creation of a digital dashboard. The authors describe that the users must have input on the dashboard since they are a potential consumer. The authors describe that seeking user input is one of five activities to follow when developing a digital dashboard. As a result of this, empirical studies were conducted with the help of different interviewees that potentially could be the users of a dashboard. All activities are being presented in the related work section (Figure 1).

Validation and reliability

The study, the data, and the research procedure should primarily be transparent, argues Yin (2013). The reasoning behind this is to boost the validity and reliability of the study. There are many ways to strengthen the validity and reliability (Jacobsen 2017). For example, the researcher should consider working methodically and to stick to the evidence. This way of working has been inspiring and implemented in the present work.

The most valid way to obtain data differ depending on the data collection method that is being used argues Johannessen et al. (2020). This was considered in the early phases of the present work. Before deciding the research questions that would be investigated, a decision was made upon which methodology was going to be followed. The reason for this was to be able to adapt the research questions to the scientific approach. The intention of working like this was also to simplify and adapt

(24)

the research questions to the choice of data collection method. Creswell and Creswell (2018) argue that this is the right way of working and that the research questions should carefully be selected with the methodology and a potential data collection method in mind.

The reliability differs depending on which data collection within the scientific approach is being used, according to Creswell and Creswell (2018). This was also something that was considered during the early phases of the present work. As a result of this, the chosen data collection method became a semi structured interview (Chapter 4.3.3), which also was stated in the present chapter. The reliability of the interviews could, therefore, be affected by the presence of the interviewer and the chosen location (Creswell & Creswell 2018). This was taken into consideration when conducting the interviews. All the interviews were, as mentioned before, performed using Google Hangouts.

Creswell and Creswell (2018) addressing the importance of the location of the interview. The authors describe that the interview should be taking place in a natural and non-hostile environment. As a result of this, the interviewees were asked to be in a peaceful and non-hostile environment before every interview. The individual participating in the study may thus feel secure and be himself.

Something that also was done to ensure high reliability was to always be comparing the data from the transcripts left from the interviews. Memos were then written, and descriptions applied to them so it later would not be any misunderstandings about the definitions of the memos. This is also an approach that Creswell and Creswell (2018) describe as a way of ensuring that the approach is reliable. Johannessen et al. (2020) also describe that by using this kind of approach, also called open coding, essential thoughts that might be interesting for the study can be found. This was also a decisive factor in using this approach.

Another way of ensuring higher reliability, which was implemented in the present work, two people decoded the same material. The findings were then compared to show similarities and differences, as previously mentioned. By working like this, Johannessen et al. (2020) describe that there is less chance of missing important data.

(25)

5 Findings and results

This chapter presents the empirical results of the interviews. Three different themes have been created based on the related work section, the research questions, and the questionnaire to increase understanding of various aspects of the subject. A description about the contents included on the dashboard prototype that was created can be found in this chapter.

KPI selection

Something that emerged during the interview was that some of the employees already use KPIs in their daily work. The KPIs that they use are more general and have been brought up by the management to measure the performance of the whole department. And because of the KPI being brought by the management as guidelines, the interviewees felt like it gave some value to their daily work. The interviewees did not directly work with KPIs during their daily work, but they have been aware of them, its significance, and what is expected from them. The interviewees did also not produce any new KPIs, at the time the interviews took place.

A common KPI that 3 out of 5 interviewees raised in the course of the interviews were indications of numbers of users interacting with their product. The majority of the interviewees felt like this KPI would have benefited the entire department. This KPI and the following KPIs that was found valuable, can be found in Table 1. Another beneficial KPI to visualize that the interviewees mentioned was how many users that are logged in at the same time. The interviewees said that the reasoning behind their opinions was that they felt like they were “in a bubble” at the office, working with their code. Some interviewees stated that the focus is more on putting through the daily work, and the purpose of the work disappears. Another interviewee believed that this KPI is an excellent way to use for gaining morale and pride in the daily work.

” […] So, I would like to build pride with that. I would like to show that there are many users we have produced a salary for this month and so much money has been paid out through our system. It is a huge sum.

- Interviewee 2

When the question was asked about what the most beneficial KPI was (if they already had implemented a KPI that they found valuable) or would be for every interviewee personally, the results varied. One interviewee argued that for their team, measurement of performance would be preferred. For example, how long it takes to load the welcome page on their web browser. As previously mentioned, interviewee 2 highlighted the aspect of building pride. By having a KPI that shows the number of salaries paid through their system, this interviewee argued that it could enhance the team spirit. The interviewee also said that they are developing a new feature and that it would have been interesting to visualize the KPI so that you could monitor it and see a curve with the help of a linear diagram of how many new customers are using this feature. It was also emphasized that a KPI that counts the number of issues and errors their product produces could be very valuable and appreciated among the colleagues. The interviewee said that the KPI could be important because their associated teams could resolve the error and act quickly before the user of their

(26)

products notice it. However, the compiled time for how long it takes to run a program seemed to be a recurring topic among the interviewees working with development.

” Above all, we want to have KPIs that show the longest run time and have them as a form of references so that you can see if it differs from what it usually takes for that specific customer.”

- Interviewee 3

Following table was created to give an overview of valuable KPIs that emerged from the conducted interviews.

Table 1: Overview of KPIs.

To find important KPIs in the future, some interviewees considered workshops as an appropriate tool to use. The interviewees said that the method is appropriate to use to generate ideas and to come up with creative solutions. Thus, a challenge could be found in converting what they want to show into a visual representation and to aggregate data. However, finding the KPIs was not considered a problem. One interviewee thought that the best way is to try over and over to obtain the best possible result in finding suitable KPIs and metrics.

Visualization of KPIs

The interviewees also got to answer on how they interpreted the word dashboard and data visualization in general. One interviewee got thinking about the dashboard of a car, and another interviewee said that they got thinking about the dashboard of an airplane. The remaining three interviewees thought that it was something to use when overviewing and observing the important data (KPIs) referring to the performance of a product or a team.

The interviewees also got to answer how they would visualize the KPIs they found valuable. As a complement to that question, they got to answer how many KPIs they would prefer displaying on a dashboard (See Figure 8 for a graphical overview of the results). Between 4 and 8 KPIs were the most optimal number to visualize according to the answers by the majority. However, one of the interviewees thought that a number between 2-4 was better and proposed the usage of more slides that changes every minute. In that way, the interviewee stated that it would become much easier to read, even for people who were not familiar with the subject. One interviewee shared previous experiences with using too many metrics as a nightmare since the dashboard creator used more than fifty different kinds of graphs and diagrams. The interviewee

(27)

stated that it was impossible to read and interpret any of it, and therefore the KPIs should rather be of high quality and have a purpose than of high quantity. The interviewee stated that it should not be an illustration of pointless elements that no-one understands.

A common thing that 4 out of 5 interviewees raised was that the resolution of the display can shape the answer. If the display is the size of a regular TV, a number between 4 and 8 KPIs are optimal. If the display size is bigger than a regular TV, one of the interviewees proposed that the number of KPIs could be the same but visualized in a bigger size.

Figure 8. Number of KPIs preferred by the interviewees.

When asked which metrics should be used, most interviewees responded that it depends on what is displayed. All interviewees believed that it was essential to be able to see trends (Ups and downs), and by visualizing information, it would be easier to do this. The interviews revealed that a linear diagram would be useful for showing trends and that tables and numbers were not perceived as equally good for showing trends. Most of the interviewees described that they would prefer using pie charts to visualize data. The reason is that the interviewees felt that pie charts are easy to read and could be used to draw conclusions. The interviewees all mentioned that they did not consider it optimal to use large sections of text to describe information on a dashboard. The reason for this was that they felt that it would have made it more difficult to interpret and read information.

“I think it is good to be able to see trends. Is it going up or down? Not just show numbers. A single number does not mean much if you should be able to see some kind of trend.

-Interviewee 5

The case company receives issues from their customers from time to time, and this was something that some of the interviewees said would benefit them to show on a

(28)

dashboard. A linear diagram was not preferred to use to visualize such KPI. All the interviewees thought that in this case, numbers would be more appropriate, and that the choice of metrics depends entirely on what KPI you want to visualize. The reason for this was that they felt that it would be easier to read the specific KPI by using numbers.

The interviewees were asked how frequent the refresh rate of a dashboard should be. The answers varied, but the interviewees agreed that it depends on what data is included on the dashboard. The interviewees commonly described that if critical data are being visualized, it must be updated in real-time. One interviewee stated that a KPI that count the length of a salary calculation might not be suitable to refresh more than a couple of times a day. As previously mentioned, most of the interviewees thought that it could be beneficial to show how many people use their system. One interviewee gave an example that this type of KPI could be refreshed every tenth minute. The reason for this was that the KPI is not referring to performance measurement and therefore, real time updates would not be necessary.

For the question of where the interviewees think the most important information should be placed on the dashboard, the answers varied. One of the interviewees thought that it did not matter how it was going to be placed. Other interviewees supported the idea of placing the KPIs at the centre of the display, with the rest of the KPIs surrounding it. The last interviewee was positive to a placement in the top left corner of the display. All interviewees believed that the most important data should be prioritized in size, and the second most important data should be visualized as the next biggest and so on.

Pros and cons of visualizing KPIs

On the question of what the interviewees thought about having KPIs visualized on a dashboard, the majority were on the positive side. They all thought that a dashboard with essential KPIs would have a big impact on their daily work. Another interviewee argued that an implemented dashboard would be positive because then the managers might have a bigger insight on how many problems they are solving. The interviewee described that if there are any problems, they will hear about it immediately, but if there are no bugs and the product is up and running, then it is quiet. The interviewee stated that if you can see that the system actually works to 99.9 percent, and there only are minor problems, it would result in an increased morale boost.

Despite all the positive thoughts about a digital dashboard at the office, some interviewees saw difficulties and threats that could emerge in the event of a possible implementation. One interviewee thought that if numbers point to certain errors for long enough, they will eventually become a kind of truth. The interviewee continued by describing that you can think that it should be wrong, but, in the end, you hear the opposite by someone else. Another interviewee explained another negative effect that may arise, but still capable of causing negative consequences in the long run.

” If something negative, statistics are just statistics. I know that one of our teams has much more complex issues, which takes a much longer time to solve. It might look bad on a dashboard. One team solves only three and the other solves eight […] You might not compare the teams

(29)

in that way, but maybe you should take everything together to create more coherency...

-Interviewee 1

The interviewees were also asked questions about how the information and the case management process look like today. This question was asked to receive thoughts regarding if a dashboard visualizing of KPIs could add something to the current flow. The interviewees described that they have a case management system where all errors or new features are added. After that, the cases become distributed. They also stated that API documentation and other technical things are stored in one of their services. To communicate internally, the employees use an external communication service. One interviewee was unsure if the case management should be on a dashboard. The interviewee thought that it might not fit the purpose of the dashboard. The same interviewee said that there are some dashboards in the stairwells in their building where one can find information, so there are ways to spread HR-based information. The interviewee stated that they would like to show more technical data on a dashboard and not so much HR information. Another interviewee thought that the dashboard could enhance information flow. The interviewee argued that if a dashboard could show statistics of the length of a salary calculation, they could be more proactive and decrease the time of finding the problem before the customer even notices the problem.

In the quote below, an interviewee answered how the information process works today and if there is a way of presenting data in metrical forms.

“If there is something in real-time, if we have any problem in the production environment, we have different channels, so it reaches the right people. Jira is more just for all kinds of information. But I think it works well. We do not have the same metrics we talk about here…” -Interviewee 1

The last question that the interviewees were asked was how they dealt with the detection of errors and if they thought that a dedicated KPI presented on a digital dashboard could help them with finding errors in an earlier stage. All interviewees believed that a dashboard could give them indications of whether an error had occurred and which area the error affected. As it works right now, it is sometimes the customer who detects a problem. One interviewee argued that if their associated team could have detected these errors more rapidly. For example, when they have released a new feature (errors often occur), they could be more proactive and spend less time on researching. The interviewee further stated that the employees could then call the customer before the customer calls them to say that they have an ongoing problem, and they are currently putting resources into trying to resolve it.

Dashboard design and its relation to KPIs: A qualitative case study on a software company