Identifying Challenges in Cybersecurity Data Visualization Dashboards

(1)

Identifying Challenges in Cybersecurity Data Visualization Dashboards

Patrick Shirazi

Information Security, master's level (120 credits) 2020

Luleå University of Technology

Department of Computer Science, Electrical and Space Engineering

(2)

Abstract

Nowadays, a massive amount of cybersecurity data-objects, such as security events, logs, messages, are flowing through different cybersecurity systems. With the enormous fast development of different cloud environments, big data, IoT, and so on, these amounts of data are increasingly revolutionary. One of the challenges for different security actors, such as security admins, cybersecurity analysis, and network technicians, is how to utilize this amount of data in order to reach meaningful insights, so they can be used further in diagnosis, validation, forensic and decision-making purposes.

In order to make useful and get meaningful insights from this data, we need to have efficient dashboards that simplify the data and provide a human-understandable presentation of data.

Currently, there are plenty of SIEM and visualization dashboard tools that are using a variety of report generator engines to generate charts and diagrams. Although there have been many advances in recent years due to utilizing AI and big data, security professionals are still facing some challenges in using the visualization dashboards.

During recent years, many research studies have been performed to discover and address these types of challenges. However, due to the rapid change in the way of working in many companies (e.g. digital transformation, agile way of working, etc.) and besides utilizing cloud environments, that are providing almost everything as a service, it is needed to discover what challenges are still there and whether they are still experiencing the same challenges or new ones have emerged.

Following a qualitative method and utilizing the Delphi technique with two rounds of interviews, the results show that although the technical and tool-specific concerns really matter, the most significant challenges are due to the business architecture and the way of working.

(3)

Acknowledgement

I wish to express my sincere appreciation to my supervisor, professor Elragal, of the Department of Computer Science, Electrical and Space Engineering at Luleå University of Technology. His fabulous insights with the great mentoring style of him helped me not only performing this particular research activity but also acquiring an in-depth understanding of research methods in general. His willingness to give his time so generously has been very much appreciated.

Special thanks should be given to all the other professors and faculty members, particularly Dr.

Awad for his professional guidance and valuable support during the period of my studies.

I will forever be thankful to Dr. Shahram Khadivi for the unforgettable lessons I learned from him during my other master studies, not just in the context of scientific matters, but also the humanity, altruism, and virtuously caring for the others. His enthusiasm and love for teaching and supporting are contagious.

My special thanks are extended to all the experts who were passionately participated in conducting this study.

Finally, I must express whole-heartedly appreciation to my wife for providing me with unfailing support and continuous encouragement throughout my years of study and the process of researching and writing this thesis. This accomplishment would not have been possible without her support.

(4)

Abbreviations

ATP Advanced Threat Protection

CSIRT Computer Security Incident Response Team CTA Cognitive Task Analysis

DSRM Design Science Research Methodology

FW Firewall

IaaS Infrastructure as a Service IDS Intrusion Detection System IPS Intrusion Prevention System PaaS Platform as a Service

SaaS Software as a Service

SIEM Security Information and Event Management SME Subject Matter Experts

SOINN Self-Organizing Incremental Neural Network

UI User Interface

UX User Experience

(5)

List of Figures

Figure 1. Connection river (Chen, et al., 2014) ... 2

Figure 2. Visualization of a security incident (Fan, et al., 2019) ... 3

Figure 3. Abstract relationship between the SIEM and visualization dashboards ... 4

Figure 4. Enhanced SIEM architecture (Sarno, et al., 2016) ... 4

Figure 5. Various input for a SIEM system (Bryant & Saiedian, 2020) ... 4

Figure 6. Sample SIEM Architecture (Lee, et al., 2017) ... 5

Figure 7. Qualitative research iteration (Streubert & Carpenter, 1999) ... 6

Figure 8. Main steps of the research ... 9

Figure 9. Framework for literature review (Brocke, et al., 2009) ... 9

Figure 10. Conceptualization of the topic ... 10

Figure 11. Process of literature review (Brocke, et al., 2009) ... 10

Figure 12. The process of selecting review sources ... 12

Figure 13. A framework of event visualization (Li, et al., 2020) ... 16

Figure 14. VisAct framework architecture (Wu, et al., 2020). ... 19

Figure 15. Evaluable components of a visualization practice (Staheli, et al., 2014) ... 20

Figure 16. Instructional efficiency measurement (Gerven, et al., 2003) ... 21

Figure 17. Components of SvEm Model (Garae, et al., 2018) ... 22

Figure 18. The traditional process of visualization (Paul, et al., 2015) ... 22

Figure 19. Sample decision-tree for choosing the right graph (Marty, 2008) ... 23

Figure 20. Visualizing approx. 10,000 records of a firewall (Marty, 2008) ... 24

Figure 21. Generation of a treemap chart using Bubble Treemap method (Görtler, et al., 2018) 24 Figure 22. A Time table graph that shows behaviour patterns (Marty, 2008) ... 25

Figure 23. Technical root causes vs. Organizational ... 34

(6)

List of Tables

Table 1. Network Visualization Categories and data source coverage (Sharafaldin, et al., 2019) . 1

Table 2. An alternate quantitative research method... 8

Table 3. An alternate constructive research method ... 8

Table 4. Literature Search Materials ... 11

Table 5. Panel member selection criteria ... 12

Table 6. Panel members demography ... 13

Table 7. Comparison of famous SIEM tools (Sönmez & Günel, 2018) ... 18

Table 8. Evaluation criteria conducted based on (Sharafaldin, et al., 2019) ... 20

Table 9. Visualization challenges according to (Wagner, et al., 2015) ... 26

Table 10. Details of experts’ consensus on identified challenges ... 33

(7)

1. Introduction

Data visualization, in general, is an approach to demonstrate knowledge and insights regarding a specific concept. Another usage of the visualization could be illustrating the correlation mapping of two or more entities, such as customer profiles, company products, and sales. Besides, visualization can also be used to display the correlation mapping, which shows different segments of one entity (e.g. customers) toward another concept such as sales (Olshannikova, et al., 2015).

A study by Raghav, et al. (2016) listed a number of benefits of data visualization for companies, such as improvement in decision making, improvement in ROI, information sharing, and time- saving.

1.1 Cybersecurity Data Visualization

As security professionals constantly need to monitor a variety of tools, devices, and environments on a regular basis, it is quite necessary to have efficient data visualization dashboards which can help them cumulating different sources of input data objects (logs, events) and providing a proper understanding and awareness of the environment they are working with.

The usage of security data visualization could be categorized in different ways. On example could be the categorization made by Vieane, et al. (2016), in which some major categories addressed as consumers of cybersecurity data visualization:

• Network analysis

• Threat analysis

• Situational awareness

Recently, some other researchers tried to expand the usage categories and investigate what kind of security-related data can be visualized in each category Sharafaldin, et al. (2019). Table 1 demonstrates different categories for cybersecurity data visualization, besides a minimum of data source coverage for each.

Table 1. Network Visualization Categories and data source coverage (Sharafaldin, et al., 2019)

Visualization category Minimum data source coverage

Host/Server monitoring Network Trace, Security Events, Network Activity Context, Network Activity Context, Non-log Information, Application Logs

Internal/External monitoring Network Trace, Security Events, Network Events, Network Activity Context, Non-log Information, Application Logs Attack patterns Network Trace, Security Events, Network Events, Network

Activity Context, Non-log Information, Application Logs Routing behaviour Network Events

IDS Monitoring Security Events Configuration visualization Non-log Information Steganography visualization Network Trace

Proxy server monitoring Network Activity Context Encrypted traffic monitoring Network Activity Context

(10)

In any domain, visualization helps security professionals analysing the current status of different environment components, finding anomalies, performing forensics, and having a security posture practice. That’s why the concept of cybersecurity visualization becomes vital, as it eases the human understanding of raw data and makes the decision-making process faster.

For example, Chen, et al. (2014) developed a visualization system to address the complexity of analysis and providing more in-depth insights and understanding. As part of their results, Figure 1 displays an example of how security data visualization can be used to monitor the live stream of network data. As an illustration, it appears that most of the huge traffic with source port greater than 200 are heading the destination port 80.

Figure 1. Connection river (Chen, et al., 2014)

In another work, Fan, et al. (2019) tried to implement a network visualization tool for real-time monitoring. Utilizing machine learning, their tool could display the anomalies. For example, Figure 2 by that visualizes a DDoS attack.

Each section of Figure 2, reveals one piece of information. For instance, Figure 2a displays the abnormal pattern node of the DDoS attack in SOINN. Figure 2c demonstrates the importance of port 80, as multiple IP addresses are connected to that are having heavy traffic, this might indicate an abnormal situation. If the operator is curious to see more details, he/she can track the traffic over the nodes in Figure 2b, or view them in Figure 2d to observe one of the nodes which are under heavy traffic. The thickness of the connection lines shows that there are simultaneous connections to this node from some other nodes, which indicates a suspicious behaviour.

(11)

Figure 2. Visualization of a security incident (Fan, et al., 2019)

1.2 Security Information and Event Management (SIEM)

Before studying security data visualization, we need to have an understanding of the SIEM tools, as data visualization needs to be performed over the manipulated data by the SIEM engine. Figure 3 displays an abstract demonstration of how different components of a SIEM tool will look like.

As it is shown in the diagram, the visualization is normally based on the previous manipulation (in the SIEM engine) over the raw data (logs).

(12)

Figure 3. Abstract relationship between the SIEM and visualization dashboards

There could be various architecture and implementation of the SIEM engines. Figure 4 displays an enhanced architecture proposed by Sarno, et al. (2016) in which “source” means a system that needs to be monitored.

Figure 4. Enhanced SIEM architecture (Sarno, et al., 2016)

The source of the input data could vary. Figure 5 displays some examples of input data for a SIEM system.

Figure 5. Various input for a SIEM system (Bryant & Saiedian, 2020)

(13)

SIEM tools are having different functionalities. The common functionalities could be listed as follow (Bryant & Saiedian, 2020; Sönmez & Günel, 2018; Cinque, et al., 2018; Novikova, et al., 2017; Sarno, et al., 2016):

• Data collection, aggregation, and normalization

• Orchestration of alerts and alarms

• Providing monitoring capabilities

• Evaluation of different perimeters

• Threat hunting

• Forensic analysis

• Assessment and of security policies

There could be many different architectures for a SIEM, but usually, they are some components that are fundamental such as Event collector, Event Processor, and Storage. Figure 6 displays a sample architecture of a SIEM (Lee, et al., 2017). As it is demonstrated, the SIEM engine, storage, and user interface are three different components that are functioning in separate layers, while integrated with each other. One important point to mention here is that in the SIEM engine, this architecture is using a message queue for both handling the input data while sending messages (alerts). Although in more modern software architecture, different other approaches could be considered.

The visualization happens in the user interface layer. That is where the dashboards and visual elements will appear. The important factor here is if we have proper segregation of layers, the SIEM tool can be integrated into the different other visualization products, as they only need to use the APIs.

Figure 6. Sample SIEM Architecture (Lee, et al., 2017)

(14)

2. Research Process

This research utilizes a qualitative research form. This research method is selected due to the fact the we would like to discover, understand, and explore a particular phenomenon. The phenomenon in our case can be described as the challenges that cybersecurity professionals are facing today while utilizing the cybersecurity data-visualization dashboards. One clue could be we are looking for “what” are the challenges, that indicates we are looking for how a phenomenon behaves in reality.

By definition, “qualitative methods are often regarded as providing rich data about real-life people and situations and being more able to make sense of behaviour and to understand behaviour within its wider context. However, qualitative research is often criticized for lacking generalizability, being too reliant on the subjective interpretations of researchers and being incapable of replication by subsequent researchers.” (Polonsky & Waller, 2018)

Figure 7 shows a typical cycle of qualitative approach iterations.

Figure 7. Qualitative research iteration (Streubert & Carpenter, 1999)

2.1 Research Scope

Following items are in the scope of this research:

• Cybersecurity data objects (security events, logs, alerts, etc.) that are produced by cybersecurity-related components such as SIEM, FW, IPS/IDS, etc.

• Visualization dashboards that cover the cybersecurity data objects

• Both cloud and on-premise environments The following are out of scope:

• Visualization in different IT domains, such as financial data.

• Information Security in manufacturing or production lines

• Information Security in very old legacy systems, such as mainframes

(15)

2.2 Delphi Technique

The approach which is planned to follow by this study is a tailored version of the Delphi method.

This method is conceptually based on the consensus development techniques (Vernon, 2009), (Avella, 2016).

In a systematic review, 100 Delphi studies reviewed from the Web of Science and Elsevier database (Diamond, et al., 2014). The results of this study showed approximately 98 percent of those Delphi studies were conducted in order to evaluate the group consensus. Of course, only 72 of them provided a definition for the consensus. This review also showed that 25 pieces of research were considered the percentage of common agreement as a metric to assess the consensus and the median of what they considered as a group consensus was about 75%.

The major design characteristics of Delphi can be listed as follow (Avella, 2016; Keeney, et al., 2001):

1. Being anonymous: This is really important, as having separate discussions with panel members prevents some issues such as group-pressure or being influenced by dominant participants (Avella, 2016). On the other hand, the researcher should be extremely cautious regarding induvial based communication with panel members. it is recommended that participants should not know each other’s participation in the research, even if they know each other (Avella, 2016).

2. Capturing Feedback: utilizing SMEs’ knowledge and expertise in the relevant field 3. Iteration (rounds): having rounds and analysis

In a recent study, Tan, et al. (2020) provided one example of the Delphi methods in the real-world on COVID-19 pandemic as a new challenge to the societies. In their work, they mentioned that although there is a tremendous amount of information being published daily, it is not easy to gather a pearl of collective wisdom on the most relevant ones while filtering the others. Therefore, they utilized the Delphi method to absorb wisdom and provide more appropriate care standards at the national level.

2.2.1 Pros and Cons of the Delphi Technique

Delphi technique has various useful characteristics which make it a very good alternative for many research-works. Some of them can be listed as following (Avella, 2016; Keeney, et al., 2001; Fink- Hafner, et al., 2019):

1- Answering uncertainty 2- Flexibility

3- Cost-effectiveness 4- Freedom of expression 5- Geographically unlimited

There have been some discussions regarding various challenges in the Delphi technique. For instance, the following major challenges are addressed by (Keeney, et al., 2001; Avella, 2016):

1- Reliability: If two different researchers perform the same research, is there any guarantee that they will come to the same conclusion?

2- Validity: How can we validate the results?

(16)

3- Researcher bias and shortcomings impact: How can we make sure the researcher does not lead the research towards a wrong direction by biasing the initial phase of Delphi (Keeney, et al., 2001; Avella, 2016)? This could, of course, happen while the results are being narrow down and selected.

2.2.2 Alternative Research Methods

Although a qualitative research method appears to be a perfect fit, there might be other alternatives as well. Table 2 and Table 3 summarize the pros and cons of some alternative methods.

Table 2. An alternate quantitative research method

Quantitative

For example, designing a survey with accurate questions, so the feedback from those Pros

Could be spread faster and easier in less time Statistical analysis will be easier

Cons

It does not match the research question. We are looking for exploring a problem to see what is that? To design surveys, we need to know the problem very well

Table 3. An alternate constructive research method

Constructive

For example, performing qualitative research first to understand the issue, then designing surveys to capture more accurate and in-detail information

Pros

More value will be added to the research, so the community (cybersecurity community) will get more benefits.

The results could be generalized better.

Cons

Generally, this will take a lot of time and effort, which exceeds the boundaries of the planned timeline.

2.2.3 Tailored Delphi Technique

This research tries to tailor the Delphi technique. As the pre-step, a decent literature review needs to be conducted to give the researcher an in-depth understanding of the problem domain and related work. This literature review could be a combination of both academic works (such a visualization scientific foundations, former qualitative works, etc.) and also, whitepapers published by global IT market leaders. The main steps of this research are shown in Figure 8.

(17)

Figure 8. Main steps of the research

2.3 Literature review

A literature review needs to be conducted, following a proper method. It needs to be in a way to reveal the previous related works which have identified the results that are related to the research motivation. For example, any recent research study that indicates specific challenges in a case study, or a CTA work.

There are different approaches to conduct a literature review. The selected approach will be based on the approach introduced by Brocke, et al. (2009), as it defines a clear process for conducting a literature review. Besides, this approach contains a very important step called “conceptualization of topic” which is a great basis for further searches.

The purpose of the review is:

- Getting insight regarding the research questions - Help us adjusting the research questions

- Finding out the gaps in previous research studies - Help in better conducting on the panel interviews

Figure 9. Framework for literature review (Brocke, et al., 2009)

5.

Seconrady rounds of feedback gathering 4.

Review &

Narrow Down Factors 3.

Brainstorming (round one of the interviews) 2.

Identify &

select Panel Members 1. Literature

Review

(18)

The first step, definition of review scope, is the fundamental step in the review, as we need to be clear regarding what needs to be done in order to conduct a review that helps to achieve the research target. The scope of the review will be the same as described earlier in this chapter. In the second step, a conceptualization of the topic is needed, as it will be a basis for finding the relevant search terms in the next step. The conceptual map of the research is presented is demonstrated in Figure 10.

Figure 10. Conceptualization of the topic

For the next step, literature search, it is needed that search keywords will be prepared, as well as the list of journals and databases to perform the search on them. Figure 11 displays this concept.

Figure 11. Process of literature review (Brocke, et al., 2009)

Visualization Challenges

Visualization Techniques

Dashboard

Reports

Charts, Gauges, etc.

Cybersecurity

Cloud

Visualization Tools

Azure Sentinel

Splunk etc.

(19)

The materials for conducting the review are presented in Table 4. At first, we started by identifying the journals that are related to conceptualized topics.

Table 4. Literature Search Materials

Selected Journals

Asian Spine Journal

Elsevier Journal of Network and Computer Applications Elsevier: Computers & Security

IEEE Communications Surveys & Tutorials IEEE Computer Graphics and Applications IEEE Journal of computers and security

IEEE Transactions on Intelligent Transportation Systems IEEE Transactions on Visualization and Computer Graphics Information Systems Research

International Journal of Doctoral Studies Journal of Clinical Epidemiology

Journal of Computer Graphics forum

Proceedings of the Human Factors and Ergonomics Society Annual Meeting Springer Journal of Computing in Higher Education

Springer Journal of Digital Imaging

Springer Journal of Information Systems Frontiers Springer Journal of medical systems

Springer Journal of Visualization Databases

and Search Engines

ACM Digital Library Elsevier

Google Scholar

IEEExplore Digital Library Microsoft Academic Search Scopus

SpringerLink

The IEEE Symposium on Visualization for Cyber Security (VizSec) Selected

Keyword

Cybersecurity Visualization Challenges Design Security Dashboard

Security Analytics SIEM Visualization Visualization Challenge Visualization Dashboard Visualization Evaluation

The selection of journals has been based on different criteria, such as relevance, accessibility to the researcher, year of publishing, number of citations, and so on. Figure 12 displays the process of selecting the sources to conduct the literature review.

(20)

Figure 12. The process of selecting review sources

In case of the literature analysis and synthesis step, a deep study of the selected dataset of collected papers takes place. Backward and forward referencing have been a great help in many cases, specifically in case of analysing review papers. The outcome of this step is explained in detail in the chapter 3.

Finally, in research agenda step, structured output of the previous step in regards to further researches and extending the performed review.

2.4 Identify and Select Panel Members

In the Delphi method, we keep running the iterations until we feel we have achieved a group consensus (Keeney, et al., 2001). Since its almost all about the group consensus/wisdom, it is vital to select proper SMEs as panel members that are holding the right qualifications.

Table 5 demonstrates the mandatory and “nice to have” qualifications for selecting panel members.

It worth mentioning that in some sources, this is called the “knowledge resource nomination worksheet (KRNW).” (Okoli & D.Pawlowski, 2004)

Table 5. Panel member selection criteria

Qualification Rational Type

Professional role in one of the following:

• Information Security domain or tightly related, such as Cybersecurity Architect, security analyst, Infrastructure Architect, network technician, etc.

• Software Architect within Dashboard Design projects

• Dashboard specialist / Visualization tool specialist

• Relevance of experience help capturing relevant input

• It is directly related to the research questions: Challenges of users

Mandatory

At least 3 years of experience In such a period, SMEs have faced some real-world challenges of the problem

Mandatory

Knowledge and expertise in the cloud environment project

Be able to compare on-prem vs.

cloud

Mandatory

Analytics skills A better understanding of the data

and how it can be gathered and processed

Nice to have Being a member of an Incident response

team

Have more in-depth knowledge regarding the reporting process

Nice to have

Performing Keyword search in each journal

and Databse

Selecting top 10 (approx.)

screening abstracts and removing old/irrelevant results

Filteringout based on Research Questions

(21)

According to the qualifications, 12 experts selected. All experts have been working in large enterprises (more than 1000 employees). Except for one expert who was located in the USA, the rest were in Sweden. Table 6 demonstrates the panel members' demography. The average experience of the participants is 11.9 years.

It worth mentioning that the diversity of a range of experts was considered purposefully to prevent some issues such as getting just technical-oriented or management-oriented answers. The participants were all asked not to just consider the current organization they are engaged with but elaborate on their past knowledge and insights as well. The insight and lessons learned that gathered during the interviews are presented in appendixes on this document.

Table 6. Panel members demography

Nr. Current Role Years of

Experience

Related roles (former/present) 1 Senior Security Adviser 10 CISO, GDPR deployment expert

2 Cybersecurity Analyst 5

3 Senior Security Architect 10 Network Admin and Technician 4 Enterprise Security Architect 25 Network Security, Infra Admin

5 Cloud Architect 7 Security Architect

6 Infra Solution Architect 19 7 Cybersecurity specialist 20 8 IT Security Specialist 10

9 Security Analyst 8

10 Monitoring Architect 6 Managing a team of +10 people who design monitoring dashboards

11 Dashboard engineer (Splunk Specialist)

3 12 Senior Security Professional

(OT)/ Managing consultant

20

2.5 Brainstorming

This can be considered as the round one of the Delphi process. Usually, it starts with some pre- prepared questionnaire, to initiate the process of brainstorming and idea-generation (Keeney, et al., 2001). At this step, the researcher needs to make the research topic clear to the individual panellists, then address the issues and ask for feedback. Primarily, it starts with a set of open-ended questions (Keeney, et al., 2001).

To perform this round of interviews, a list of potential questions prepared based on the performed literature review. The questions were mostly targeting the experience and opinions of experts regarding different challenges in security visualization dashboards.

At the first round of interviews, we focused on the open-ended questions that were designed based on the research questions, so the experts could express their thinking and ideas in a broader way.

They have been encouraged to elaborate as much as possible. In the process of interviews, the experts assured that their expressions will be anonymous and no personal/organizational information needs to be provided. Each interview took almost between 45-75 minutes.

(22)

Some questions (and not limited) to below:

• Tell me about your experience of using visualization dashboards: Your way of usage, common strong points you see in the visualization dashboards, common weak points, etc.)

• What are the major challenges in security visualization?

• Have you had a problem with understanding what a security visualization dashboard displays? What were the root causes?

• Generally speaking, do you see any difference in cloud security visualization vs. On-prem?

Why?

• What features do you think are missing in the current security visualization tools?

• In your opinion, what features are we missing in nowadays visualization tools?

• Have you heard other people complaining about visualization dashboards? What do you think are their concerns?

• If you would like to buy a visualization tool, which features are the most important to you as a customer?

2.6 Review and Narrow Down Factors

The result from the previous phase needs to be collected, cleansed, and clustered. There might be some irrelevant items addressed by panel members that could be omitted. On the other hand, the researcher will have some idea regarding different clusters of opinions which are coherent. These could be used later on to design more particular questions.

The practical approach for us in this research would be the following process:

1. Gathering all the statements that are expressed by the panel members in the brainstorming (round one interviews)

2. Removing the duplicates, or irrelevant statements.

3. Categorization of the remaining statements by labelling the similar ones 4. Reviewing the categories and merging them in case of similarity or overlap

After performing the above process, we expect that major categories of challenges will be evolved.

2.7 Secondary Rounds of Feedback Gathering

Having more accurate factors in hand from the previous step, now it is time to have the second run of the feedback gathering. In this phase, the questions will not be open-ended and panel members are more required to express their opinions on specific items. In some cases, they need to approve/reject or rank the items.

(23)

3. Related Works and Research Gaps

There have been many kinds of research in different aspects of visualization. Although not all of them are related to the field of cybersecurity, the identified challenges in them could be useful.

As a relevant example, Olshannikova, et al. (2015) provided an intensive overall review of the different aspects of data visualization, including big data visualization, different types of visual elements (e.g. colour, shape, size, etc.), and even cognitive psychology principles. Having such a universal review, they deducted that an optimal visualization method had to optimize the usage of different criteria, otherwise the results might be too complex in case of an overload of visual elements and eventually, not being human readable.

3.1 Inherent Data Visualization Challenges

There are some inherent challenges in many kinds of data visualization, including cybersecurity data objects. These challenges are agnostic to the tools, development approaches, and the data domain. There is no wonder that in many related works, these kinds of challenges such as the high volume of data, the complexity of the data, and so on, are common.

In a study, Li, et al. (2020) worked on the implementation of an event-sequence visualization tool called SSRDVis. The tool supposed to visualize and detect abnormal cases using a “rare detection module”. One challenge they addressed during their work was extracting meaningful information from a massive amount of sequential data, as sequential data are highly complex. They mentioned that although there are some methods for the analysis of event sequences, they are still experiencing a challenge in creating an overall view on several event sequences. For instance, they mentioned the challenge of applying machine learning methods (e.g. clustering) to sequences directly (despite they work well with vector space).

Eventually, they proposed a framework for visualization of events considering the variance of time. According to them, users would like to know “What does a typical event look like? How do the features and events interact with each other?”

In their work, besides the challenge of processing a massive number of features and timesteps, they addressed the complexity of event processing over timesteps as well. As the ratio of occurrence for events varies, besides having many identical events at different intervals. Another challenge would be the detection of events based on the features and spanning up to hundreds of timesteps. Figure 13 demonstrates their proposed framework and the process of visualization of the events.

Data quality seems to be another challenge in the context of visualization. This challenge addressed by Borovina and Ferreira (2017), besides identifying data defects. For instance, detecting some data defects need the knowledge of the data context, meaning this needs to be supervised by a human. On the other hand, just visualizing the data might not necessarily help to detect some of the data defects. However, in the case study, they tried to follow a qualitative approach to achieve a set of implications in order to have a visual assessment system for data quality.

(24)

Figure 13. A framework of event visualization (Li, et al., 2020)

3.2 Human Factor Challenges

A fundamental challenge of how to transform the raw data into a “human-understandable format”

while not losing the meaning, is addressed by some researchers (Paul, et al., 2015).

Besides that, many of the recent related works are addressing a common challenge: in the traditional way of the data-visualization design process, the end-user is either ignored or not involved. For instance, Sethi and Wills (2017) mentioned that the development of visualization tools in cybersecurity is significantly suffering from not having enough involvement of the end- user, as well as not standardizations and guidelines in both design and evaluation. They also addressed the issue of the generic design of visualization tools, which in some cases makes it ineffective for some users.

In another research, Vieane, et al. (2016) tried to address the human factor gaps in different areas of cybersecurity. A major gap they mentioned was that we are still very beginner in cognitive human aspects in cybersecurity and more researches are needed in this area.

Investigation researches in which an implementation of a tool besides its evaluation is included could provide significant knowledge or insight for discovering challenges that users might have nowadays even with market-leading products. In some cases, researchers start implementing a tool having a tool architecture in their minds, but after implementation, they did not really get the results they wanted from the users. In other words, users did not necessarily behave as they have expected.

Discovering various root causes of such failures could be a great help in order to identify visualization challenges.

Chen, et al. (2014) not only addressed the issue of “traditional single expert analysis system”, but they also designed and launched an online tool to highlight the importance of collaboration and higher engagement of more people. In their development, they mentioned two major challenges:

the complexity of input dataset including interrelations, besides the dynamic handling of threshold settings for the input.

A visualization tool called Charticulator implemented by Ren, et al. (2019), determined from a constraint-based layout approach. The visualization tool implemented utilizing HTML5 and other front-end technologies in order to enable a user to design his/her own visualization in an interactive

(25)

way. They tried to transform the specification of the desired chart to some mathematical constraints, then trying to computer and generate the visualization layouts in a way that it satisfies the constraints, using a constraint-solving algorithm.

After implementation, they performed a user acceptance test and noticed three challenges (besides a natural challenge of low speed due to the constraint-solving algorithms):

• Usability: how the user can place, align, rotate the text, etc.

• Conditional visibility of texts

• Legend creation: users had the challenge to add legends to the charts

They tried to fix these challenges by optimizing the tool itself. However, they noticed even with the fixes they applied; the third challenge is still there. Although their tool had a button for adding the challenges, the users were still experiencing difficulties in adding the legends. A valuable insight by one of the users could give us a very important clue: “I'm still thinking in a regular [Microsoft] Excel format.” (Ren, et al., 2019). This clearly shows how the human factor and the user mindset could have an impact on the ways of working.

3.3 CTA Researches

Cognitive task analysis (CTA), as a method that focuses on unobservable task activities of people (Wei & Salvendy, 2004), might be an option for understanding user behaviour and eventually, designing effective visualization based on that. Of course, we should consider that CTA results might significantly vary based on the training time, required skill, and so on (Wei & Salvendy, 2004).

One example is a CTA work by Champion, et al. (2012) that identified three different team performance factors in cybersecurity:

• The structure of the team

• The communication of the team

• Information overload.

The last item is a very important factor for us as this huge load of information in not only addressed in other CTA works such as Gutzwiller et al. (2016), but also in other reviews and tool developments (Chen, et al., 2014; Sethi & Wills, 2017; Sharafaldin, et al., 2019; Staheli, et al., 2014; Bakirtzis, et al., 2018).

Although the mentioned CTA works were not directly regarding the visualization, the findings could indirectly be a great insight for any visualizer designer to show how a user could be drawn in a massive amount of information to perform a simple task, and this will have an impact on their performance. The other important key finding of Champion et al. is that in cybersecurity tasks, the situation awareness is moderate-to-low (Champion, et al., 2012). Besides Gutzwiller et al. (2016) addressed another key finding: as security analysts need the information just in time of the decision formulation, they have to keep pieces of the information “in the mind”. These points again could be an indirect vital insight for visualization designers, as the capabilities of feature selection, tailoring and customization are mentioned in other research studies (Chen, et al., 2014; Fischer &

Keim, 2014; Sharafaldin, et al., 2019; Wagner, et al., 2015).

(26)

3.4 Tool Specific Behaviours

It seems some challenges raise due to the specific behaviour of the visualization tools. In some cases, the approach to design a dashboard could be improved (e.g. implementing a new approach by adding an action history layer (Wu, et al., 2020)). In some other cases, the challenges could be resolved by optimization and fixes.

Considering the vital role of any SIEM tool, Sönmez and Günel (2018) performed an evaluation of the famous commercial SIEM tools in the current market. Table 7 demonstrates their comparison summary. As it clearly appears, the tools are having different behaviours in various areas, which might cause some challenges for the users. For instance, according to their comparison, some tools are more difficult to be integrated with custom data.

Table 7. Comparison of famous SIEM tools (Sönmez & Günel, 2018)

According to a study by Sarno, et al. (2016), SIEM tools are having “three principal weaknesses”

while being used in critical infrastructure protection:

• In case of having different security policies, SIEM tools are often do not have capabilities to resolve those policy conflicts.

• Not all the current SIEM tools are able to monitor, identify, and control the universal possible data flows for a perimeter.

• Ensuring the integrity of the history and logs, which are going to be used for further forensic activities. This integration could be sone via encryption or signing the data.

Many different visualization tools investigated by Wu, et al. (2020). In their work, they noticed that most of them are not recording what actions had been performed over the process of conducting data visualization. This could be interpreted as a challenge, as users are limited to some undo/redo actions, and not having a semantic chain of actions. Identifying such a challenge, they

(27)

proposed a new framework called VizAct consisting of three different layers, including an “Action Tracker”. Figure 14 demonstrated their proposed framework and its different layers.

Figure 14. VisAct framework architecture (Wu, et al., 2020).

A noticeable challenge that evolved during the implementation of the Charticulator tool by (Ren, et al. (2019), was people’s mindset regarding the direct manipulation of the charts. According to them, people that have worked with vector graphic tools (e.g. Adobe Illustrator) believe in

“everything is manipulable”, so they expect to be able to modify the outcome at any time, while the visualization tool created by them (based on the constraint-solving algorithms) could not possibly do that, as it was designed to generate the layouts mathematically. They also mentioned the user interface challenges that their users had, for example “It was sometimes difficult determining what I needed to click to reveal other properties/options.”

VisComposer is the name of another tool for information visualization by Mei, et al. (2018). In their work, they focused on the programmability capabilities as well as the user interface and user interactivity. After the evaluation of their work, they had interviews with the participants.

Although they got some positive feedback, half of the participants had experienced UI challenges due to the complexity of it, as it required a lot of user interactions in order to perform the visual mapping and transformation of the data.

3.5 Evaluation Challenges

In a study, Elmqvist and Yi (2015) worked on the evaluation of data visualization. They gathered a list of 20 evaluation patterns, in either qualitative or quantitative form, in 5 different categories:

• Exploration

• Control

• Generalization

• Validation

• Presentation.

(28)

In their work, they tried to provide some approaches in order to evaluate different visualization solutions. Although this is an important matter, it still seems to be at an abstract level and, putting that into practice is not an easy task.

Evaluation of visualization approaches and dashboard tools is also another challenge. This studied by Staheli, et al. (2014), and they addressed eight evaluable components in a visualization practice.

Figure 15 demonstrates the components and their connections.

Figure 15. Evaluable components of a visualization practice (Staheli, et al., 2014)

In a related work, Sharafaldin, et al. (2019) introduced an evaluation framework for network security visualization. Table 8 demonstrates the summary of their criteria.

Table 8. Evaluation criteria conducted based on (Sharafaldin, et al., 2019)

Evaluation Criteria Description

Data source coverage How sophisticated is a visualization tool to handle multiple sources of input data

Interoperability How sophisticated is a visualization tool to integrate its services with the other tools e.g. exchange the information

Flexibility and Interactivity

How easy the users can interact with the system?

Scalability How a tool can handle big data, besides how can it visualize big data in an efficient way

Machine Assistance How the tool really facilitates users to solve their problems? Is it popular among users?

Validation evaluation: Having a set of use cases, is the tool evaluated in practice? The use case contains:

o How an individual cybersecurity user uses the tool?

o How a team of cybersecurity users is collaborating using the tool?

o How can it be used to solve a real-world problem?

o Does the tool have documentations?

Attack Coverage How many attack patterns are covered by the tool?

(29)

Figure 16 displays a classic approach to measure the effectiveness of mental activity needed to understand a phenomenon. In this way, high efficiency is achieved when the reading performance is high while the mental level is low (Gerven, et al., 2003). Considering efficiency score (E) for a user as a perpendicular distance between the learning performance and the mental effort. Now, if we compare three different groups (A, B, and C), and measuring the group means, then when E=0 line could be considered as a measure.

𝐸 =ZPerformance− ZMental Effort

√2

Figure 16. Instructional efficiency measurement (Gerven, et al., 2003)

SvEm is a security visualization effectiveness measurement framework introduced by. In their proposed framework, the main effectiveness metrics are:

• Visual clarity

• Visibility

• Distortion rates

• User response (viewing) times.

They argued that effectiveness measurement in many other related works is usually based on technical measurements such as performance, image quality, clarity, etc. While effectiveness does not necessarily mean that. According to them, more suitable visualizations are those that the viewer user can understand the story and the rationale behind it, without needing external help in interpretation.

To do so, they tried to bring cognition, perception and insight contributed to the account. Figure 17 displays the components of their proposed framework, and their relations.

(30)

Figure 17. Components of SvEm Model (Garae, et al., 2018)

3.6 Design Challenges

In a study, Paul, et al. (2015) worked on. Opposite to the “traditional” approach with a design based on answering pre-defined problems, they tried first to develop a visual concept having the end-user (human) in focus, regardless of the input data or user requirements. They mentioned that this approach works better for new problems that do not have a strong solution in places, and is not supposed to replace traditional approaches.

Figure 18. The traditional process of visualization (Paul, et al., 2015)

Another research by Bakirtzis, et al. (2018) addressed a simple, yet the vital challenge of the in the design phase in a visualization tool: lack of access to the real security data for the system designers.

They also highlighted the fact that sometimes the lack of information id due to historical logs, such as applied patches to resolve a vulnerability in an older version of a software system.

Some related visualization challenges addressed by Marty (2008), such as poor data-quality that might not contain the information it is supposed to have for the visualization. According to him, another challenge is deciding on the amount of data to be visualized in each graph in a dashboard.

In his book, Marty mentioned how vital it is to select the proper tool (chart) to display the correct information. Figure 19 displays a simplified decision-tree for choosing the right type of diagram to be used in a dashboard. As demonstrated in the figure, it is primarily important to know what is going to be explained (from the tool perspective) and needs to be understood by the user (from the user perspective).

Data Analytics Visualization Context Human

(31)

Figure 19. Sample decision-tree for choosing the right graph (Marty, 2008)

An important aspect of this approach is considering the number of dimensions of the input data.

For example, a histogram might be a good option for comparing 3 different dimensions (e.g.

comparing different urban development factors in three big cities), but it will not be the best option for comparing hundreds of items. In that way, a Treemap diagram is a much better approach.

Figure 20 displays a treemap chart that visualizes the status of 10,000 records of a firewall, in case of either they are passed or blocked.

(32)

Figure 20. Visualizing approx. 10,000 records of a firewall (Marty, 2008)

A related challenge addressed by Görtler, et al. (2018) considers the generating treemap diagrams in the visualization of the deep hierarchical data including uncertainties. To resolve the issue, they developed a model to visualize the data (hierarchical in nature) that are impacted by uncertainty.

In Figure 21 they presented how their developed method is generating a treemap chart using a model that propagates the characteristics of uncertainty data in a hierarchical structure.

Figure 21. Generation of a treemap chart using Bubble Treemap method (Görtler, et al., 2018)

(33)

Figure 22 displays a time-table graph that is used to visualize the comparison of traffic over time.

As it appears, such a diagram is capable of highlighting gaps and patterns which are the result of periodic behaviours.

Figure 22. A Time table graph that shows behaviour patterns (Marty, 2008)

3.7 Other Challenges

Nine different categories of network security visualization identified by Sharafaldin, et al. (2019), based on the classifying recently published works:

1. Host/Server monitoring 2. Attack patterns

3. Internal/External and Internal/Internal monitoring 4. Routing behaviour

5. IDS monitoring

6. Configuration visualization 7. Steganography visualization 8. Proxy server monitoring 9. Encrypted traffic monitoring

Although this list is covering many aspects of visualization, it clearly shows that cloud-specific visualizations are missing. For example, according to Sharmaa, et al. (2016), cloud providers usually provide security “as a service”, such as monitoring of identity and access management.

A significant challenge addressed by Wagner, et al. (2015), performing a survey on visualization systems with a focus on malware analysis, is the tremendous amount of newly identified malware, besides the extensive growth rate of them.

They divided malware visualization into three categories:

• Individual malware analysis,

• malware comparison

• Malware Summarization.

As a result of their work, they addressed the “future challenges” conducted in Table 9.

(34)

Table 9. Visualization challenges according to (Wagner, et al., 2015)

Challenge Description

Overlap of Malware visualization categories

Different categories (mentioned above) are overlapping. The visualization tool has to be very sophisticated to switch between individual malware analysis and comparative analysis.

Handling multiple data sources

How a tool needs to get the input data from multiple sources (integration), and combine the data.

Capability of customization and tailoring

How users can tailor and customize the tool to create their own visualization dashboard

Capability of interactive adaptive changes

Does the visualization tool have the capability of detecting and using the knowledge of expert users’ behaviour?

Segregation of visualization and data analytics

In most systems, these two are so tied together, making it very difficult for scalability and customization based on the problem.

3.8 Research Gaps

Considering the previous related works, it seems considerable research studies have performed over the visualization challenges in the field of cybersecurity. However, cybersecurity and eventually its visualization is rapidly changing nowadays due to the fact of expansion of the cloud environments and digital transformation.

Thus, one obvious gap is how the visualization challenges look like in the current situation, while many companies adopted cloud technologies while keeping their on-prem infrastructure? Are they still experiencing the previous challenges that already covered by some of the related works, or there has been a change in either major challenge headlines or their priority?

Considering advances in cloud environments and those many companies that are migrating their infrastructure and applications to the cloud-based environments, how does it differ from classical on-premise infrastructure? Is there any difference between the visualization of the cloud data, compared with on-premise data? Considering the existing challenges, how likely they might happen in the real-world?

3.9 Research Questions

The research gaps show that we need a more updated investigation on the current situation, in the case of cybersecurity data-visualization challenges. From the previous related works in the past, we cannot infer which challenges are existing today, or whether new challenges have emerged.

Therefore, answering the following research questions will greatly help to verify known challenges, besides identifying any recently evolved major challenge:

1- What are the main challenges that security professionals are facing in using the current tools that are visualizing security-related data objects?

2- How significant are the challenges in real-world scenarios?

(35)

4. Empirical Work

This section elaborates on the details of the empirical research study.

4.1 Results from the First Round of Interviews

After performing the first round of interviews, explained in chapter 25, more than 120 statements gathered from the panel members. Applying the narrow down step, explained in chapter Review and Narrow Down Factors, the results condensed into 42 group statements in 13 categories. The categories are covered alphabetically as the following sub-chapters.

4.1.1 Business Context Correlation

Correlation of dashboards with the company's strategy, vision, risk, and decision-making processes seems to be an obvious challenge to many experts. This is also regarding how to connect the outcome of the visualization dashboards to the business context in order to use them for a better understating of the organization’s status. This looks more like management and organizational challenge. Some experts mentioned the organizations are unable to answer some key success factor questions such as:

• Finally, are we having a proper security practice in place or not?

• Is out security posture satisfactory?

• Are we suffering from the skills gap in the field of cybersecurity?

• Do our assets have proper protection?

• How can we “see” the spent budget on internal security training, in the dashboards?

The other major topic here is how the results could be used in decision-making processes. For example, if the management of an organization is about to decide whether or not they should prioritize a specific section (e.g. risk management, security incident team, etc.)?

According to the first round of results, another discovered challenge in this context is missing the

“management view” as security visualization dashboards are usually designed for the technical security professionals.

According to one of the experts: “Managers are interested in how secure we are, and not how many incidents or vulnerabilities we had covered”

4.1.2 Customization of general-purpose tools

Discussions show that there is a fundamental challenge in nowadays dashboard visualization tools.

The vendors try to build a product that could be sold to as many customers as possible. Hence, it has to be generalized in a way that could cover so many different usages. On the other hand, experts believe that custom products are usually a better fit for their domains, as they designed and adjusted specifically for those domains.

In case of having so many different custom products for each domain of usage, other challenges such as price, maintenance, and the integration of the tools to get the big picture will emerge.

The outcome of the previous discussion would be general-purpose systems that security experts need to customize them to their needs, which of course take time and effort.

(36)

Experts mentioned that usually, visualization tools come with some pre-defined templates for the most common reports and alerts, but in many cases, they need to spend time on creating their custom reports.

They also said that requesting vendors for making changes in the products will cost a lot as it is being considered as customer-specific requests. The chance of having changes in the famous visualization tools is not that much, as it has dependencies on so many different factors such as cost, time to delivery, backward compatibility, market, and so on.

4.1.3 Dashboard Design

This category illustrates the challenges in designing the dashboards. It covers both the challenges in which designers are facing, besides those challenges that end-user thinks are caused due to the design.

As an end-user, the following challenges addressed:

• Sometimes there is no meaningful story behind the visual elements, and the end-user needs some explanation to understand the chart.

• Sometimes the visual elements are not clear, or they are missing the explanation for abbreviations.

• In case of very general dashboards, people might need to export the data and create and their own visualization.

In the case of designing a dashboard, Experts addressed the following major topics:

• People’s preference:

People are having a vast variety of preferences. For example, some people really like pie charts while others avoid them.

• Customization:

This is very similar to the challenge which is already covered in the chapter 4.2.1.

In big enterprises, due to the vast number of required visual reports, there is a conflict between the need for general purpose reports versus particular usage reports. One challenge of creating these custom reports is more load of work for maintenance, troubleshooting, change management, and so on.

• Change of requirement and scope creep:

In visualization projects, experts feel new requirements and change requests sometimes lead to the scope creep issue.

4.1.4 Data Quality

Experts addressed various aspects of data quality challenges. Some of the challenges could result in low trust in the visualization tools. However, some could be due to the visualization tools behaviour themselves.

1. False positives: False positives are very common in visualization dashboards. Of course, they seem to be the challenge of the input data or the processing engine of SIEM tools.

2. Dependencies: While gathering data from different sources, there is always a dependency on them. For example, if an external link to a system that provides potential threats will be

(37)

broken, the data will not eventually land in the SIEM tool, thus, the visualization dashboard will not be showing the current information.

3. Data cleaning and pre-processing: Processing of the massive input data is a challenge.

4. Aggregation and standardisation: data that are gathered from different sources are not necessarily having the same quality standards. They also might have different formats that need to be standardised.

5. Data quality challenges in master data. Sometimes, the master-data system suffers from low-quality data (e.g. in an asset management system, some assets might not have the correct data fields such as ID, owner, usage, etc.)

4.1.5 Human Factor

The addressed factors in this category are related to the behaviour of the users. Experts mentioned that Although technical integrations are possible, different teams sometimes not collaborating enough. This could be due to the team loads, organizations’ internal processes and bureaucracies, and even organizational politics.

Another identified challenge is that people's mindset is usually conditioned to react mostly to negative situations in data visualization. For example, people tend to double-check when the dashboards are showing red flags. The risk in such behaviours could be due to an integration technical difficulty, some negative results are not being detected or transmitted to the visualization tool, but the end-user thinks there is nothing negative.

This factor was also a difficult one. As some participants with more years of experience have seen more human factor challenges while the other participants had neutral ideas.

4.1.6 Information Overload

Two major challenges addressed by the panel members:

1. How to get the right level of details to present to the users?

In different scenarios, users get much more details that they need to perform a specific task, and they would like to scale out some details.

Note: Some tools use a “drill-down” capability, meaning end user sees a high-level report, then they can click on a part of the report and see more details. Although this is a very good approach according to the users, in some cases after one or two levels of drilling-down, the details would be either confusing or just too much. This is besides taking a long time to prepare and display the details.

Some users mentioned that some tools or some designed dashboards, do not support the capability of removing the columns of data that are no needed. Therefore, they have to do it manually in a third-party tool like MS Excel.

2. Too many visual elements

This is very common while users have to use “all in one” dashboards. Sometimes, there are very general-purpose dashboards that display too many elements (e.g. for monitoring a network).

In such situations, the user might feel some conflicts/paradoxes between different visual elements, and making meaningful correlation among those elements will need mental processing. According to one expert: “I need visualization to make it easier, not more complicated”.

(38)

4.1.7 Integration and Interoperability

Nowadays, each company is utilizing many different IT components in a variety of its domains.

Hence, it is inevitable to make integrations among these components in order to aggregate the data and make meaningful results. This is more obvious in the case of having master data.

Experts identified some different challenges as follow:

• In big enterprises, it is common that different teams can have their own tools and dashboards. Although this diversity could be good for using customized tools, it will add complexity in order to aggregate the data and integrate different software components for automated data gatherings.

• Integration of the old legacy systems is challenging, as they might not support modern integration techniques.

• Delay of getting data from different sources, especially external ones.

• Tools might overlap in some areas. One example could be using a tool for monitoring some network devices, and another tool for updating some of those devices. Hence, some devices need to be registered twice. In such cases, conflicting entries in tools could reach an integration issue (as they need to be correlated and aggregated).

4.1.8 KPI Definition

Key Performance Indicators (KIPs) are very common in many organizations. They are supposed to aggregate the data and display the status of a specific parameter. Thus, there are highly being used in the visualization dashboards.

Experts mentioned that there are some obvious challenges in this category:

• Designing KPIs based on the available data rather than the business problem

In some cases, people/organizations do not know what to monitor, but they are looking for KPIs that show them the overall status.

• Wrong KPIs for answering questions

Some designed KPIs are misaligned with realities. For example, just having the number of patched devices does not necessarily show whether the company is in a good situation or not. A better approach could be working on the patch curve of last quarter.

• Wrong questions, looking for KPIs

Sometimes, people focus on finding KPIs for answering questions that do not really bring any useful value. This is correlated to the business context as well.

• Focus on one parameter only

Sometimes, it is needed to include different factors to answer a business question. For example, to have a better view of vulnerability management, the organization should not deduct it from the progress perspective during a specific time period.

4.1.9 Manual Work

Experts mentioned that in some cases, which might cause due to the lack of automation/integration, there is still a need for manual work to extract, check, and build custom reports. There are many different scenarios that users need to export pieces of information from the different course and use a third-party application, e.g. MS Excel, in order to aggregate and build a custom report.

Identifying Challenges in Cybersecurity Data Visualization Dashboards