• No results found

A systematic mapping study on Quality of Service in industrial cloud computing

N/A
N/A
Protected

Academic year: 2021

Share "A systematic mapping study on Quality of Service in industrial cloud computing"

Copied!
50
0
0

Loading.... (view fulltext now)

Full text

(1)

aster˚

as, Sweden

DVA423

Thesis for the Degree of Master of Science (60 credits) in Computer

Science with Specialization in Software Engineering 15.0 credits

A SYSTEMATIC MAPPING STUDY

ON QUALITY OF SERVICE IN

INDUSTRIAL CLOUD COMPUTING

Malvina Latifaj

mlj19001@student.mdh.se

Examiner: Jan Carlson

alardalen University, V¨

aster˚

as, Sweden

Supervisors: Federico Ciccozzi

alardalen University, V¨

aster˚

as, Sweden

everine Sentilles

alardalen University, V¨

aster˚

as, Sweden

(2)

Acknowlegments

First and foremost, I would like to express my deepest gratitude and highest appreciation to my supervisors, Federico Ciccozzi and S´everine Sentilles, without whose consistent feedback and persistent support, this thesis would not have been possible. To Federico, who so uniquely pro-vided me the intellectual freedom to pursue my research, while making sure I did not diverge from the core of the study, and whose guidance and expertise during this journey have been invaluable: Infiniti ringraziamenti! To S´everine, who trusted me with this thesis, who so generously supported my academic growth from the very beginning, and whose passion for research was contagious: Je ne vous en remercierai jamais assez!

I would like to extend my most sincere thanks and recognition to the Malardalen University’s professors, whose constructive advice and insightful suggestions have enriched my educational background. Special thanks also go to my home institution, the Polytechnic University of Tirana for providing me with the opportunity to pursue my studies abroad.

Outside the academic circle, I would like to gratefully acknowledge the people with whom I shared this journey and experienced times of joy and despair. Their assistance and encouragement, while simultaneously carrying a heavy load of their own on their shoulders, is deeply appreciated.

This last word goes to my family, whose never-ending love and profound belief in my abilities are worth much more than I can express on paper. As I know they are the proudest to witness my every achievement, I dedicate this work to my parents, who so selflessly provided me with the opportunities and experiences that have made me who I am, and encouraged me to seek my destiny. Because of them, I have the strength to grow, live and take the world by storm.

(3)

Abstract

Context – The rapid development of Industry 4.0 and Industrial Cyber-Physical Systems is leading to the exponential growth of unprocessed volumes of data. Industrial cloud computing has shown great potential as a solution that can provide the necessary resources for processing these data. However, in order to be widely adopted, it must provide satisfactory levels of QoS. The lack of a standardized model of quality attributes to be used for assessing QoS raises significant concerns. Objective – This study aims to provide a map of current research on QoS in industrial cloud computing, focusing on identifying and classifying the quality attributes that are currently most commonly used to evaluate QoS.

Method – To achieve our objective, we conducted a systematic mapping study of the state-of-the-art of QoS in industrial cloud computing. Our search yielded 1063 potentially relevant studies that were subject to a rigorous selection process, resulting in a final set of 42 primary studies. Key information from the primary studies was extracted according to the categories of a well-defined classification framework.

Results – The analysis of the extracted data highlighted the following main findings: (i) research largely focuses on providing solution proposals that require a more solid validation, (ii) the adoption of cloud technologies is closely related to performance indicators, while research on other quality attributes is quite limited, (iii) there is a lack of research on security in industrial cloud computing, (iv) approaches are in most cases not targeting explicitly a specific industrial domain, (v) there is a strong focus on the impact of virtualization solutions on QoS, and (vi) research efforts are oriented towards the improvement of QoS through scheduling.

Conclusion – These results can help the research community identify trends, limitations, and re-search gaps on QoS in industrial cloud computing, and reveal possible directions for future rere-search.

(4)

Table of Contents

1. Introduction 1

2. Background 3

2.1 Industrial cloud computing . . . 3

2.2 Quality of Service . . . 3 2.3 Quality model . . . 3 2.4 Quality attributes . . . 4 2.5 Quality metrics . . . 4 3. Research Methodology 5 3.1 Planning . . . 5 3.2 Conducting . . . 5 3.3 Reporting . . . 5

3.4 Research goal and questions . . . 7

3.5 Search and selection process . . . 8

3.5.1 Initial search . . . 8

3.5.2 Merging and impurity removal . . . 9

3.5.3 Application of selection criteria . . . 9

3.5.4 Backward snowballing . . . 10

3.6 Classification framework . . . 11

3.6.1 Publication trends and contribution . . . 11

3.6.2 Software quality attributes and metrics . . . 12

3.7 Data extraction . . . 13

3.8 Data synthesis . . . 14

4. Conducting the systematic mapping study 15 4.1 Search and selection process . . . 15

4.1.1 Initial search . . . 15

4.1.2 Merging and impurity removal . . . 15

4.1.3 Application of selection criteria . . . 15

4.1.4 Backward snowballing . . . 16 4.2 Classification framework . . . 16 4.3 Data extraction . . . 17 4.4 Data synthesis . . . 17 5. Results 18 5.1 Vertical analysis . . . 18 5.1.1 Results analysis of RQ1 . . . 18 5.1.2 Results analysis of RQ2 . . . 21 5.1.3 Results analysis of RQ3 . . . 23 5.1.4 Results analysis of RQ4 . . . 24 5.2 Horizontal analysis . . . 26

5.2.1 Quality attributes and quality metrics . . . 26

5.2.2 Factors and quality attributes . . . 27

5.2.3 Quality attributes and metrics across different domains . . . 28

5.2.4 Objective of primary studies and means of reaching it . . . 28

5.2.5 Contribution type and validation type . . . 29

6. Discussion 31

7. Threats to validity 33

8. Related work 34

(5)

References 37

Appendix A Primary studies 38

Appendix B Classification framework 41

(6)

List of Figures

Figure 1 Systematic mapping study process . . . 6

Figure 2 Search and selection process . . . 8

Figure 3 Detailed description of the search and selection process . . . 10

Figure 4 Keywording and data extraction process . . . 13

Figure 5 Publications trend per year . . . 18

Figure 6 Publications venues . . . 19

Figure 7 Distribution of publication types per year . . . 19

Figure 8 Contribution types . . . 20

Figure 9 Validation types . . . 21

Figure 10 Distribution of primary studies per quality attribute . . . 21

Figure 11 Factors affecting QoS . . . 22

Figure 12 Domains . . . 23

Figure 13 Quality metrics . . . 23

Figure 14 Objectives of the primary studies . . . 24

Figure 15 Means of reaching the objectives . . . 25

Figure 16 Mapping of quality attributes and quality metrics . . . 26

Figure 17 Number of metrics with only one occurrence per quality attribute . . . 27

Figure 18 Mapping of quality attributes and factors . . . 27

Figure 19 Mapping of quality attributes, quality metrics and domain . . . 28

Figure 20 Mapping of the objectives and means of reaching it . . . 29

(7)

List of Tables

Table 1 The selected digital libraries . . . 8

Table 2 Search string based on PICOC . . . 9

Table 3 Contribution types according to [1] with the addition of a new type . . . 11

Table 4 Research types according to [2] . . . 12

Table 5 Validation types according to [3] . . . 12

Table 6 Search string for each library . . . 15

Table 7 Number of publications per type of search . . . 16

Table 8 Classification framework overview . . . 17

Table 9 Publications per research type . . . 20

Table 10 Detailed means of reaching the objective . . . 25

Table 11 Detailed classification framework . . . 41

(8)

List of Acronyms

EFP Extra-Functional Property

ICPS Industrial Cyber-Physical System

IIoT Industrial Internet of Things

ISO International Organization for Standardization

KPI Key Performance Indicator

NFP Non-Functional Property

NFR Non-Functional Requirement

NIST National Institute of Standards and Technology

PICOC Population, Intervention, Comparison, Outcome, Context

QoS Quality of Service

SLA Service Level Agreement

SLO Service Level Objective

SP Service Provider

(9)

1.

Introduction

“Industry 4.0” represents the fourth industrial revolution through the digitization of manufactur-ing. It has shown rapid development, and it is now an established trend in automation and data exchange for manufacturing technologies and processes. A step further in the same direction is taken by Industrial Cyber-Physical Systems (ICPS), which integrate computation and networking with physical processes and aim to control the latter in real-time [4]. The potential impact of these systems is undisputed, and as the popularity of “Industry 4.0” increases, so does the volume of data that needs to be processed. However, the hardware and software needed to process these data cannot be offered by traditional approaches [5].

A solution to limitations of hardware and software is Cloud Computing, defined by The National Institute of Standards and Technology (NIST) as “a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal man-agement effort or service provider interaction” [6]. Due to its rapid advancements, benefits, and relatively low cost, cloud computing has been considered as a favorable solution for multiple en-terprises. Nevertheless, its adoption encounters several challenges when it comes to providing interconnection between different enterprises and domains to allow them to communicate and col-laborate. For this reason, the concept of Industrial Cloud Computing was introduced, and it aims at providing industrial digital information integration and collaboration between enterprises, based on a shared understanding of concepts [7]. Industrial cloud computing can facilitate the cyber part of ICPS by providing higher efficiency and scalability and meeting the dependability demands of industrial applications.

Dependability and software quality are crucial factors in the success of the industrial applica-tions [8]. Thus, the appropriate quality of service (QoS) must be attained for cloud platforms that will supply the necessary resources for processing industrial data. According to ISO/IEC 25010 [9], the quality of a system is “the degree to which the system satisfies the stated and implied needs of its various stakeholders, and thus provides value”. The stakeholders’ needs are defined in a quality model, which helps determine the quality attributes needed to evaluate the product.

Even though quality attributes play a crucial role in software engineering, they have received little attention, and there exists no formal definition or a complete list of them [10] [11]. There have been some attempts to the standardization of these attributes, but the classification schemes are inconsistent with each other terminologically and also categorically, and therefore their defi-nitions are still unclear and lack consensus. Without that standardization, each specific system defines its own terminology.

Moreover, the lack of a standardized terminology regarding quality attributes has a high and negative impact on the Service Level Agreement (SLA), which is the contract negotiated by a cloud provider and cloud consumer, to ensure a certain agreed level of QoS. The SLA contains Service Level Objectives (SLOs), which describe the set of quality attributes for the specific ser-vice. The lack of standard definitions of quality attributes leads to lack of clarity in SLAs and consequently limits their credibility, as there is no clear understanding of the established objectives and they cannot be compared to the actual performance.

Need for a systematic mapping study

All of the aforementioned considerations can negatively influence QoS in cloud-based platforms. That being said, even though cloud computing is one of the most important technologies of the 21st century and has been the center of attention of researchers in the last decade, it still encounters multiple challenges regarding QoS. The lack of a generic model of quality attributes that can be used as a reference for specific domains leads to the lack of a fundamental set of quality attributes to be used to evaluate QoS in industrial cloud computing. This specific set of quality attributes is necessary due to the peculiarities of industrial cloud computing. The existing literature described

(10)

in Section 8. does not provide the necessary evidence that could help in identifying the appropriate quality attributes to evaluate QoS in industrial cloud computing. Therefore, there is a need to conduct a study that can answer the following question:

What is the state-of-the-art on QoS in industrial cloud computing?

More specifically, we aim at answering the research questions defined in Section 3.4. The overall goal is to identify and classify the quality attributes that are used to evaluate QoS in industrial cloud computing, and the ways to assess them.

The main contributions of this thesis are the following:

1. An overview of the publication trends on QoS in industrial cloud computing and the venues that address the matter.

2. A classification of the quality attributes used to evaluate QoS in industrial cloud computing and the quality metrics used to assess the quality attributes.

3. A classification of the factors that have an impact on QoS in industrial cloud computing. 4. A classification of the domains in which QoS in industrial cloud computing is investigated. 5. A classification of the primary studies’ objectives with respect to QoS in industrial cloud

computing and a description of the means by which these objectives are achieved.

The remainder of this study is organized as follows. Section 2. describes the background of our research and Section 3. details the research methodology of our study. Section 4. describes how we conducted the systematic mapping study, and Section 5. provides the main findings. Section 6. discusses the results and Section 7. analyzes the threats to validity. Section 8. describes related work and Section 9. concludes the paper and highlights future work.

(11)

2.

Background

In this section, we describe the core ground concepts of the thesis.

2.1

Industrial cloud computing

Cloud computing is a cutting edge technology due to its flexibility and low costs. It makes possible the sharing of resources between multiple users. The popularity that cloud computing has obtained over the last decades is due to five characteristics that make a significant difference compared to traditional approaches. First, cloud computing offers on-demand self-service, which allows the con-sumer to acquire the needed service, automatically, without human interaction from the provider. Second, it offers broad network access, meaning that its capabilities can be accessed through stan-dard mechanisms over the network. Third, it offers resource pooling where the provider puts together all its computing resources and serves it to multiple consumers in a multi-tenant environ-ment depending on their demands. Fourth, cloud computing offers rapid elasticity, which provides a sense of unlimited resources because the consumer can elastically provision and release resources based on its needs. Last but not least, cloud computing offers measured service, which provides transparency because resource usage can be controlled and optimized [6]. All these characteristics provide the driving force of cloud computing, which has stimulated consumers to adopt it. How-ever, during the last few years, the digital information produced by industry has increased rapidly. Enterprises try to handle this flow of information using their existing structures, which, of course, have boundaries on their capabilities. Furthermore, industry nowadays shows the need for col-laboration between different enterprises and disciplines, which crosses their individual boundaries. Cloud computing has proven to be a solution that can tackle the complexity of such collaborative approaches, but none of its deployment models can overcome all the occurring obstacles. For that reason, the concept of Industrial Cloud Computing was introduced as a solution in the form of a platform to exchange, process, and analyze digital information. For the enterprises to be able to communicate and collaborate in an industrial cloud, they must share the same understanding of concepts. First, they need an open and shared architecture so that participants from different enterprises can connect and, second, they need common ontologies and unified data standards to facilitate data exchange amongst one another and ensure that the approach on their meaning of data is mutual [7].

2.2

Quality of Service

To stay competitive as technology evolves, a certain QoS should be developed to meet the cus-tomers’ expectations. QoS in cloud computing indicates “the levels of performance, reliability, and availability offered by an application and by the platform or infrastructure that hosts it ” [12]. The main objective of QoS is to make the services acceptable for users, and it is a mutual necessity for both cloud consumers and cloud providers. Cloud consumers need to make sure that the quality of service advertised to them when acquiring the service is delivered, and cloud providers need to find the right trade-offs amongst operational costs and QoS levels [13]. Furthermore, the aforemen-tioned need to ensure that the resources cloud consumers request to provision can be supported at any time and that these resources meet the customers’ requirements and expectations consistently.

2.3

Quality model

A quality model is “a defined set of characteristics, and of relationships between them, which pro-vides a framework for specifying quality requirements and evaluating quality” [14]. According to Dromey [15], to increase the software quality, there should be defined a quality model that can make clear and direct links amongst the high-level quality attributes and specific product charac-teristics. Additionally, the model must ensure a systematic structure on how to build high-quality software, how to identify and classify software characteristics and make sure this is comprehensible, refinable, and adaptable at multiple levels. One of the first software quality models is McCall’s quality model [16], introduced in 1977 and where quality is decomposed in a hierarchical way to a measurable level. Several other models have been introduced in the following years, such as

(12)

Boehm’s model [17], Dromey’s quality model [15] etc. The International Organization for Stan-dardization (ISO) established the 9126 standard, which also describes a software quality model and which was afterward revised in the ISO 25010. The ISO quality models can be used to iden-tify relevant quality characteristics that can be further used to establish requirements, criteria for satisfaction, and the corresponding measures.

2.4

Quality attributes

Quality attributes in software engineering can be used to define software quality. They are also frequently referred to as quality properties, non-functional properties (NFPs), extra-functional properties (EFPs), etc. Although the aforementioned have a crucial impact on software develop-ment, they have been left in the background compared to functional requirements, and the results are scattered [10]. This leads to quality attributes not being addressed correctly, software of poor quality, higher costs, and customer dissatisfaction. There exist multiple reasons as to why quality attributes, which are so pivotal to the quality of software systems, can be so hard to address cor-rectly. Their nature is subjective as different stakeholders interpret and evaluate them differently, leading to ambiguity. Secondly, their achievement is relative, as there might always be different ways to reach a satisfactory level. Third, quality attributes can be interacting and affect one an-other for better or for worse, so there can not be a localized solution. This set of issues makes it difficult to deal with and even more challenging to measure and verify quality attributes.

2.5

Quality metrics

Quality metrics in software engineering can be used to provide a quantitative basis regarding software quality and to objectively measure quality attributes [18]. They measure different aspects of quality and have multi-fold objectives. Through this quantitative measurement, the actual QoS can be compared to the expected QoS and future steps for improvement can be determined. Providers can assess the progress and performance of their service. Moreover, quality metrics provide a quantitative description of how QoS is affected by various factors. According to ISO/IEC 9126 [19], software quality metrics can be split into three categories; external quality metrics that measure the behavior of the system, internal quality metrics that measure the software, and quality-in-use metrics that measure the effect of using the software in a specific context.

(13)

3.

Research Methodology

Quality of service and industrial cloud computing are both very active research areas, with a large set of primary studies that contribute to their body of knowledge. However, to our best of knowledge, there is no evidence of a secondary study that investigates QoS in industrial cloud computing with a focus on quality attributes and highlights future research prospects. Thus, to form a good understanding of the academic state-of-the-art on QoS in industrial cloud computing with a focus on quality attributes and to fill the research gap, we carry out a systematic mapping study. According to Petersen et al. [1], a systematic mapping study “provides a structure of the type of research reports and results that have been published by categorizing them and often gives a visual summary, the map, of its results”. The process of this systematic mapping study can be divided into three well-established phases: planning, conducting and reporting. The steps are illustrated in Figure 1, and a description of each phase is provided in the following sections.

3.1

Planning

The planning phase of a systematic mapping study includes the following activities: (i) identify the need of performing it, (ii) define research goal and questions, (iii) define the protocol to be followed for conducting the study, and (iv) review the protocol and if it needs to be refined go back to step (iii). The output of the planning phase is the final reviewed protocol.

3.2

Conducting

The output of the planning phase, which is the final defined protocol, will be set into practice in order to conduct the systematic mapping study through the following steps.

1. Search and selection: During the search and selection process we (i) apply the identified search string to the selected digital libraries, (ii) merge and remove impurities, (iii) apply the selection criteria to identify primary studies to include, and (iv) apply backward snowballing in order to expand the set of primary studies. At the completion of these steps, the final set of primary studies is obtained. In Section 4.1 we provide a detailed description of this step. 2. Classification framework : We define the parameters of the classification framework based on (i) the research questions and (ii) by applying the keywording method to the set of primary studies. This phase generates a classification framework. In Section 4.2, we provide a detailed description of this step.

3. Data extraction: We iterate through the set of selected primary studies and extract infor-mation from them according to the facets of the classification framework. The framework is refined in case of additional relevant information that does not belong to any of the existing facets. At the end of this process, we obtain the extracted data. In Section 4.3 we provide a more detailed description of this step.

4. Data synthesis: We analyze and summarise the extracted information that is obtained in the previous step using vertical and horizontal analysis, in order to provide answers to the research questions defined in Section 3.4. In Section 4.4 we provide a more detailed description of this step.

3.3

Reporting

The final phase is composed of the following activities: (i) a discussion on the extracted data and the systematic mapping with the purpose of reporting the main findings, (ii) a classification and analysis of the validity threats, and (iii) the writing of the report that describes the performed study.

(14)
(15)

3.4

Research goal and questions

This study aims to identify, classify, and evaluate trends in the current literature concerning QoS in industrial cloud computing. The fundamental goal is to achieve a classification of the quality attributes addressed the most in industrial cloud computing and to provide researchers and practi-tioners with a mapping of key QoS attributes and methods to assess them. This will enable them to make informed decisions in the context of QoS for industrial cloud computing and will highlight research gaps, laying the groundwork for future research.

First we apply the Population Intervention Comparison Outcome Context (PICOC) criteria in accordance with [20] as follows:

• POPULATION: Industrial cloud computing

• INTERVENTION: QoS in industrial cloud computing • COMPARISON: Not applicable

• OUTCOMES: A classification of the primary studies that reflects the current state-of-the-art of QoS in industrial cloud computing

• CONTEXT: Academic peer-reviewed publications with a software engineering perspective

The research questions to be answered from this systematic mapping study are as follows: • RQ1. What are the publication trends regarding the quality of service in industrial cloud

computing research?

Rationale: To identify the existing state of research on QoS in industrial cloud computing and assess the density of scientific publications over the years.

Outcome: An illustration of the current state of research on QoS in industrial cloud com-puting in terms of publication venue, contribution type, research type, and validation type.

• RQ2. What are the most addressed quality attributes in industrial cloud computing re-search?

Rationale: The concept of QoS is quite vast, and it includes a multitude of quality attributes, therefore it is important to identify the ones that are used in industrial cloud computing. Outcomes: (i) A set of software quality attributes addressed in industrial cloud computing research, (ii) a set of factors that affect specific quality attributes and the overall QoS (e.g., network), and (iii) the industrial domains in which cloud computing is used (e.g., gaming).

• RQ3. What are the most common quality metrics used to assess quality attributes in indus-trial cloud computing research?

Rationale: To identify the metrics used to assess the quality attributes in industrial cloud computing and illustrate how they are mapped to the latter.

Outcomes: (i) A set of metrics to assess the Qos in industrial cloud computing, and (ii) a mapping of the latter to the respective quality attributes.

• RQ4. What are the research objectives of the primary studies in relation to QoS in industrial cloud computing and how are those objectives achieved?

Rationale: The research objectives vary between studies, and it is important to understand the trends of what the research community hopes to achieve.

Outcomes: (i) A classification of the objectives to be achieved by the primary studies in relation to QoS (e.g., improve), and (ii) ways on how these objectives can be achieved (e.g., task scheduling).

(16)

3.5

Search and selection process

In a systematic mapping study, the search and selection process is a decisive multi-stage process that provides wide coverage of the topic under investigation. As such, it needs to be documented thoroughly to allow the study’s replication from researchers who can hopefully corroborate the results.

Our search and selection process begins with an initial search over the chosen digital libraries. The next step is the merging and impurity removal of publications, and the application of selection criteria to assess the relevance of the publications. Backward snowballing serves as a complemen-tary strategy to identify relevant literature and data extraction finalizes the process. Figure 2, illustrates the complete search and selection process and provides information on the size of our corpus during the five stages.

Figure 2: Search and selection process

3.5.1 Initial search

In order to obtain our initial set of primary studies, we carry out automatic searches on two of the largest and most complete electronic databases in software engineering: IEEE Xplore Digital Library and SCOPUS, that can be accessed through the URL listed in in Table 1. Both databases are well-established and include a wide spectrum of peer-reviewed publications. The search was performed considering the title, abstract, and keywords of studies and it was conducted on 25 February 2020.

Digital Library Number of results IEEE Xplore Digital Library 442

SCOPUS 621

Table 1: The selected digital libraries

Both databases provide two crucial features that facilitate the search and selection process. 1. Advanced command search: It provides the possibility to combine different data fields and

operators in order to form complex queries and narrow the scope of our search. IEEE Xplore allows up to 20 search terms, while SCOPUS has no limitations on the amount of the search terms. Both databases support single-character(?) and multi-character(*) wildcards, and the use of quotes for an exact phrase.

(17)

2. CVS export: Both databases support the export of the search results, in a well-defined format such as CVS to be used in the following phases. Moreover, they provide the selection of fields to be exported so that the excel spreadsheet meets our requirements.

The search string to be used in the initial search is created using the PICOC criteria as illustrated in Table 2.

POPULATION “cloud” AND “industr*”

INTERVENTION

(“quality of service” OR “QoS” OR “quality model” OR “software qualit*” OR “quality propert*” OR “quality attribut*” OR “non functional” OR “extra functional” OR “NFR” OR “EFR” OR “NFP” OR “EFP”)

COMPARISON Not applicable OUTCOME Classification scheme

CONTEXT Academic peer-reviewed publications with a software engineering per-spective on QoS in industrial cloud computing.

Table 2: Search string based on PICOC

The syntax of the resulting search string is as follows:

“cloud” AND “industr*” AND

(“quality of service” OR “QoS” OR “quality model” OR “software qualit*” OR “quality prop-ert*” OR “quality attribut*” OR “non functional” OR “extra functional” OR “NFR” OR “EFR” OR “NFP” OR “EFP”)

3.5.2 Merging and impurity removal

Different digital databases and indexing systems often offer the same publications. Through the use of the search results spreadsheet exported from both databases and based on an exact match on the title, authors, and publication year, duplicates are found and removed. If the same study has two different versions, the oldest version is removed too. Furthermore, digital databases include a vast range of publication types such as proceedings, textbooks, editorials, magazines, etc., which are not research papers. These publications are also excluded.

3.5.3 Application of selection criteria

Potentially relevant studies need to be assessed to determine their relevance to our study as not all studies returned by the initial automatic search can be used. The selection criteria are used to filter the primary studies and to include only those that demonstrate significant relevance to the research questions. In this phase, the potentially relevant studies are filtered based on the inclusion and exclusion criteria, considering the title, abstract, and keywords. If the study satisfies all the inclusion criteria and none of the exclusion criteria, it is included in our corpus of primary studies, and if not, it is excluded. If a study cannot be clearly excluded considering the above-mentioned elements, a full-text investigation is carried out to make a decision.

Inclusion Criteria

I1. The study reports on an approach that aims to define, measure, analyze, evaluate or im-prove QoS in the scope of cloud computing.

I2. The study investigates quality attributes, quality metrics or quality models and their impact on QoS in cloud computing (e.g., minimize the request failure and improve quality of service). I3. The study reports on the use or need of cloud technologies in industry (e.g., banking) or it is published at an industrial venue.

I4. The study is written in English and is available in full text.

(18)

Exclusion Criteria

E1. The study does not focus on any specific quality attribute.

E2. Secondary and tertiary studies (e.g., systematic literature reviews, surveys, case studies) E3. Tutorial papers, short papers, poster papers, editorials, books, keynotes, tutorial summaries, tool demonstrations and panel discussions, introductory papers for books and workshops, technical reports and other non-peer-reviewed publications.

E4. The study is not in the computer science context (e.g., the search yields results in meteorology as a consequence of the use of the search term “cloud”).

3.5.4 Backward snowballing

The initial corpus of primary studies obtained after the application of the selection criteria is complemented with a fully-recursive backward snowballing activity, according to the guidelines in [21]. In backward snowballing, the reference list of each selected paper is used to determine new relevant papers to include in the final set of papers. The first step consists of going through the reference list and removing papers that do not fulfill the basic criteria such as language and publication venue. Next, papers that have been examined in prior stages are removed, and the remaining ones are examined based on the title. The relevant references are then found on the paper to research how and where they are referenced in order to retrieve additional information regarding their suitability. The abstract of the paper from the reference list is read, and if that is not enough, relevant sections or even the full paper is examined to make a final decision. Figure 3 illustrates the complete search and selection process.

Figure 3: Detailed description of the search and selection process

Upon completing the snowballing process, the studies from automatic search and backward snow-balling are combined. Duplicates from the snowsnow-balling process are removed, and earlier versions of publications are excluded. Finally, during the data extraction process in which the full text of the study is read, a study may still be excluded for not fulfilling the selection criteria.

(19)

3.6

Classification framework

As a means of extracting and classifying the relevant information from the primary studies, a classification framework is created. The framework is composed of four facets, each of which addresses a specific research question, as described in the following.

1. Publication trends and contribution – RQ1 2. Software quality attributes – RQ2

3. Software quality metrics – RQ3

4. Objective of the work in relation to QoS – RQ4

3.6.1 Publication trends and contribution

The facet regarding RQ1, publication trends and contribution, is composed of five sub-facets as follows: 1. Publication year 2. Publication venue • Conference papers • Workshop papers • Journal papers

3. Contribution type: The studies have been distributed in six categories, using the classification proposed by Petersen et al. [1] as a basis, and tailoring it to suit our needs by adding the framework category. A study can have more than one contribution type. The contribution types and their definitions are illustrated in Table 3.

Contribution type Definition

Metric The study provides specific metrics and measures to assess the quality attributes in industrial cloud computing.

Tool The study provides a tool or a prototype used to evaluate, analyze, assess or improve QoS in industrial cloud computing.

Model

The study provides an abstract representation that can be a model or a system used to evaluate, analyze, assess, or improve QoS in industrial cloud computing.

Method

The study provides a systematic approach which includes general con-cepts and working procedures to evaluate, analyze, assess or improve QoS in industrial cloud computing.

Process The study defines a systematic set of activities that interact to eval-uate, analyze, assess, or improve QoS in industrial cloud computing.

Framework

The study provides a layered structure on how to evaluate, analyze, assess or improve QoS in industrial cloud computing and it usually combines the aforementioned contribution types.

Table 3: Contribution types according to [1] with the addition of a new type

4. Research type: The studies have been distributed in six categories according to the classi-fication by Wieringa et al. [2], and each category has been defined to fit into our study. A study can be part of one or more categories. The research types and their definitions are described in Table 4.

(20)

Research type Definition

Solution Proposal

The study proposes a novel solution technique or a significant im-provement of an existing one. The applicability of the solution pro-posal is demonstrated through a proof of concept such as an example, argument, etc., but it does not include a full-blown validation.

Validation Research

The study investigates the claims and properties of a solution tech-nique, not yet implemented in practice. Possible techniques involve experiments, simulation, prototyping, etc.

Evaluation Research

The study evaluates the use of tools or techniques in practice by means of sound research methods such as case study, field experiment, survey, etc.

Philosophical Papers The study designs a new perspective of looking at things such as a new conceptual framework, taxonomy, etc.

Opinion Papers

The study provides the author’s personal opinion on a certain aspect of research, such as whether a technique is good or bad or how it should be carried out.

Experience Papers

The study provides a personal experience of the authors with an em-phasis in what and how an activity is conducted in practice, usually based on lessons learned from the participation in an industrial or academic setting.

Table 4: Research types according to [2]

5. Validation type: The studies have been distributed in six categories according to the classifica-tion of validaclassifica-tion types by Shaw [3], properly instantiated to fit our needs. This classification aims to provide a mapping of how presented approaches are validated. The categories are described in Table 5.

Validation type Definition

Analysis The study validates its approach by means of an extensive analysis such as: formal analysis, empirical mode, controlled experiment.

Experience

The study validates its approach by providing evidence of its correct-ness, usefulness or effectiveness through real examples provided by other individuals in the research community.

Example The study validates its approach by providing examples on how it works on ”toy examples” or a ”slice of life”.

Evaluation The study validates its approach by providing a qualitative or de-scriptive model of the phenomena of interest.

Persuasion

The study provides arguments regarding the benefits and advantages of the proposed approach but it provides no empirical results or ex-amples from its use.

Blatant assertion The study provides no serious attempts to validate the results.

Table 5: Validation types according to [3]

3.6.2 Software quality attributes and metrics

The parameters of the facets for RQ2, RQ3, and RQ4 are defined using the keywording method [1]. This method not only helps with the definition of the parameters but also with the data extraction from the primary studies described in Section 3.7. The final output is a classification framework. During this process, we will perform the following activities:

1. Identify keywords and concepts

For each primary study, we read the abstract and introduction section and identify keywords and concepts regarding software quality attributes and metrics for RQ2 and RQ3, and the

(21)

intended purpose of the authors regarding QoS in industrial cloud computing (e.g., improve, evaluate, define, etc.) for RQ4. After all keywords and concepts are identified, they are merged together to generate a set of keywords.

2. Cluster keywords and define parameters

The collected keywords and concepts are organized into clusters according to the identified characteristics. Given that the clusters are unknown at first, an iterative bottom-up approach is used to create them. The output of this step is the initial classification framework.

3.7

Data extraction

The purpose of data extraction is to obtain the relevant information from the primary studies according to the classification framework defined in Section 3.6. During this process we will perform the following activities:

1. Extract data from the primary study

An iteration through all the primary studies to be analyzed is conducted in order to: (i) extract information, (ii) collect information according to the defined parameters of the initial data extraction form and (iii) collect additional information which does not fit in the initial classification framework, but is considered relevant.

2. Refine the classification framework

The collected additional information is reviewed for the following reasons: (i) the initial interpretation of the additional information may not be correct and has to be refined, and (ii) the defined parameters of the initial classification framework may not be representative enough for the primary studies. For these reasons, the initial framework is refined to suit our needs, and the primary studies are re-analyzed to match the redefined parameters.

Keywording and data extraction process are illustrated in Figure 4.

Figure 4: Keywording and data extraction process

After the information from the set of primary studies is extracted and refined in a precise and consistent way, the keywording and extraction process is finished, with a complete classification framework as an output. The facets of the classification framework undergo a more in-depth analysis in order to identify possible correlations between parameters and define their values. The

(22)

framework can contribute to the classification, comparison, and evaluation of the software quality attributes and metrics in industrial cloud computing.

3.8

Data synthesis

The purpose of the data synthesis activity is to analyze and summarize the extracted data from the primary studies in a meaningful way, in an effort to understand and classify the current research on QoS in industrial cloud computing. This activity is conducted according to the recommendations in [22]. The data synthesis process is divided into two main phases: vertical analysis and horizontal analysis.

Vertical analysis: During the vertical analysis activity, we analyze the extracted data from the primary studies in order to collect information regarding each defined parameter of our classification framework. By applying the line of argument synthesis [23], we first analyze each primary study in isolation, in order to document and tabulate its main features according to the classification framework and then analyze the set of studies as a whole, to identify potential patterns and trends.

Horizontal analysis: During the horizontal analysis activity, we analyze the extracted data from the primary studies in order to discover possible relations among the different categories defined for each research question. We cross-tabulate the data in an effort to find similarities, noteworthy differences, and recurring patterns. The results that do not contain sufficient data points nor exhibit relevant patterns are excluded.

(23)

4.

Conducting the systematic mapping study

Following the research methodology described in Section 3., we describe in this section the execution of our study, which begins with the search and selection process and ends with the data synthesis of the data extracted from the primary studies.

4.1

Search and selection process

4.1.1 Initial search

We ran an automatic search on the selected digital libraries: IEEE Xplore and Scopus. In Section 3.5.1, we introduced the generic search string using the PICOC criteria. We included a considerable amount of search terms related to the QoS and quality attributes; we narrowed down our search through the “industr*” search term. For each digital library, the search string is adapted to fit the library’s search constraints. The exact search string used for each library can be found in Table 6.

Digital

Library Search String

IEEE Xplore

“All Metadata”:“cloud” AND “industr*” AND (“quality of service” OR “QoS” OR “quality model” OR “software qualit*” OR “quality propert*” OR “quality attribut*” OR “non functional” OR “extra functional” OR “NFR” OR “EFR” OR “NFP” OR “EFP”)

SCOPUS

TITLE-ABS-KEY (“cloud” AND “industr*” AND (“quality of service” OR “QoS” OR “quality model” OR “software qualit*” OR “quality propert*” OR “quality attribut*” OR “non functional” OR “extra functional” OR “NFR” OR “EFR” OR “NFP” OR “EFP”)) AND (LIMIT-TO (DOCTYPE , “cp”) OR LIMIT-TO (DOCTYPE , “ar”)) AND (LIMIT-TO (LANGUAGE , “English”)) AND (LIMIT-TO( SRCTYPE , “p”) OR LIMIT-TO (SRCTYPE , “j”) OR LIMIT-TO (SRCTYPE , “k”))

Table 6: Search string for each library

The automatic search on IEEE Xplore returned 442 studies and the one in Scopus 621, which gives 1063 initial primary studies.

4.1.2 Merging and impurity removal

While conducting a systematic mapping study, it is pivotal to ensure a reliable set of studies for inclusion. A multi-database search provides a broader spectrum of studies and minimizes publication bias, but simultaneously leads to duplicates and repetitive studies. For IEEE Xplore and Scopus, we merged the results in an excel spreadsheet and identified 193 duplicates based on an exact match of title, authors, and publication year; we removed the duplicates. No studies with different versions were identified. After the completion of this process, 870 publications were left in our corpus of selected studies.

4.1.3 Application of selection criteria

The inclusion and exclusion criteria are crucial elements to the primary studies’ selection process as they set the boundaries on what should and should not be included. Therefore, an explicit definition and description of these criteria should be provided in order to allow the replication of the study. First, an initial set of selection criteria was defined and the papers were screened based on title, abstract, and keywords. However, there was a set of 150 primary studies that were uncertain with respect to the inclusion criteria. Therefore, after discussions with the thesis supervisors, the inclusion criteria were refined and the final criteria are introduced in Section 3.5.3. Moreover, one major challenge was identified during this process. The terminology regarding the quality of service with an emphasis on quality attributes and metrics between publications is inconsistent and lacks a well-established consensus. Often, quality attributes and metrics are missing proper

(24)

definition, therefore hampering the process of selecting relevant studies. For instance, few studies investigate performance as the overall satisfaction of the system, while others explicitly mention performance as a quality attribute. In this case, we only included the latter. After the application of the refined selection criteria, 34 studies were included in the corpus of primary studies.

4.1.4 Backward snowballing

Backward snowballing is the process of using the reference list of the selected studies to track down potentially relevant studies that are not part of the chosen digital libraries or not covered by the search string. By following the procedure described in Section 3.5.4, the backward snowballing process returned 790 results. However, after the removal of duplicates and application of selection criteria, eight publications were included in our corpus, for a total of 42 primary studies. The main reasons for the exclusion of publications were as follows:

1. The publications retrieved from the reference list of a specific study, investigate similar topics to the aforementioned (e.g., mobile cloud computing), but do not report on QoS, or quality attributes and metrics, therefore not satisfying I1 and I2.

2. The publications do not report on the use or need of cloud technologies in industry, therefore not satisfying I3.

Table 7 illustrates the number of publications for each type of search strategy.

Type of search Included Excluded Total

Automatic search 34 1029 1063

Backward Snowballing 8 782 790

Total 42 1811 1853

Table 7: Number of publications per type of search

4.2

Classification framework

In order to extract and analyze data from the primary studies, we needed a classification framework that defined categories and data items pertaining to specific research questions. The categories and data items for RQ1 are defined in Section 3.6.1, while for RQ2, RQ3, and RQ4 are defined through the keywording process. The complete classification framework is illustrated in Appendix B.

RQ2 consists of 3 categories as follows:

• Quality attributes: The data items in this category cover the quality attributes used in industrial cloud computing and addressed in the primary studies.

• Factors: The data items in this category cover the factors that impact quality attributes in industrial cloud computing and that are explicitly mentioned in each study.

• Domain: The data items in this category cover all the domains in which QoS in industrial cloud computing is investigated. A study is classified under a specific data item, only if the domain is mentioned explicitly.

RQ3 consists of only one category as follows:

• Metrics: The data items in this category cover all the types of quality metrics addressed in the primary studies, which can be used to assess QoS in industrial cloud computing. RQ4 consists of two categories, as follows:

(25)

• Objectives: The data items in this category cover all the objectives of the research community regarding QoS in industrial cloud computing, addressed in the primary studies. The majority of keywords for this category were collected from the abstracts of the publications, as they summarise the overall objectives of the study.

• Means: The data items in this category cover all the means that are addressed in the primary studies to help researchers reach the objectives defined in their studies. The labels of the data items (i.e., their names) are exactly as reported in the primary studies, without any interpretation from us to avoid bias.

RQ Category Value multiplicity Value examples

RQ2 Quality attributes Multi-value performance, security

Factors Multi-value virtualization, network

Domain Multi-value gaming, mobile

RQ3 Metrics Multi-value latency, jitter

RQ4 Objective Multi-value improve, define

Means Multi-value scheduling, allocation

Table 8: Classification framework overview

4.3

Data extraction

After the definition of the classification framework illustrated in Appendix B, we started the extraction of the relevant information from each primary study. The full text of the primary studies was read to collect information according to the categories of the defined framework, and additional information that did not fit in any category but was considered relevant. As expected, the categories with the most additional information were the quality attributes, quality metrics, and means of reaching the objectives. These categories had to be refined, as the keywording process’s initially defined parameters were not representative enough of the primary studies. Ambiguities and uncertainties of any kind, related to the extraction and classification of data, were solved through constructive discussion with the thesis supervisors. At the end of the data extraction process, no studies were excluded.

4.4

Data synthesis

The final step of the study is the synthesis of the extracted data. As introduced in Section 3.8, data synthesis was performed in terms of vertical and horizontal analysis. Each category of the classification framework was subject to the vertical analysis in isolation in order to discover trends over time and obtain quantitative results regarding each data item. However, the vertical analysis alone may not be sufficient to provide a complete understanding of the topic. Therefore, we performed horizontal analysis by mapping: (i) quality attributes with quality metrics, (ii) quality attributes with factors affecting QoS, (iii) quality attributes and metrics with specific industrial domains, (iv) objectives with means to reach them, and (v) contribution type to validation type. Other mappings were analyzed, but not included in the results since they did not provide interesting correlations.

(26)

5.

Results

In this section, we present the results from the vertical and horizontal analysis of the extracted data. The complete list of studies from which the data is extracted can be found in Appendix A.

5.1

Vertical analysis

The purpose of the vertical analysis is to provide quantitative results regarding each category in isolation. These results are provided in the following subsections in a tabular and graphical form.

5.1.1 Results analysis of RQ1 Publication years

Figure 5 illustrates the publication trends on QoS in industrial cloud computing over the years. Our initial search had no boundaries regarding the publication year, but as we can see from the extracted information, the primary studies are published in a limited time range from 2009 to 2020 with only five publications until 2012 and a growing interest which reached its peak in 2015. From 2016 to present, we have an average of 4 publications per year with no highlighted peaks, which indicates that we have a constant publication rate regarding QoS in industrial cloud computing. However, the search for 2020 is not complete as it is conducted in February 2020, and the search for 2019 might be partially complete as not all publications have been added to databases yet.

Figure 5: Publications trend per year

Publication venues

Figure 6 illustrates the distribution of primary studies by (i) type of publication (e.g., workshop paper, journal paper, or conference paper) and (ii) type of venue (e.g., industrial venue or other). The most common publication type is conference papers (23/42), followed by journal papers(17/42) and workshop papers (2/42). From the extracted data, we notice that only four primary studies pertain to industrial venues with an even distribution between conferences and journals, and only one venue hosted more than one primary study (P1 and P5 published at IEEE Transactions on Industrial Informatics). The complete list of publication venues is provided in Appendix C.

Figure 7 illustrates the distribution of publication types per year. We notice that conferences have a more stable trend with at least one publication per year since 2009 and a high peek in 2017, while journals show a more steep line, with no publications until 2012, a peek in 2015 and no publications in 2017. Conducting a more thorough analysis of these results led us to some interesting insights. The high peak of publications in 2015 is described in P31 as the time when cloud computing was moving from theory to practice. Indeed, the larger part of the selected studies seems to support this claim. The majority of the primary studies published before 2015 are part of what can be labeled as the “inception phase”. Authors in P25, published in 2009, investigate the possibility of using cloud technologies to fulfill the enterprises’ requirements for new information

(27)

processing techniques that could analyze large amounts of data. Moreover, the authors in P4 and P9, published in 2011 and 2012, respectively, predict the popularity of the use of cloud technologies in industry and the growing number of cloud providers. Thus, they investigate ways on how the

Figure 6: Publications venues

QoS of these providers can be evaluated so that users can make more informed decisions. In P21 and P34, published in 2013, we notice an investigation of the bottlenecks that are hampering the adoption of cloud technologies. Moreover, in P1 published in 2014, the authors propose a quality model named CLOUDQUAL, in which they define quality attributes that can be used to evaluate cloud services. Thus, to a large extent and up to now, authors are defining new concepts and future challenges regarding the use of cloud technologies in industry and are promoting key attributes to evaluate QoS. However, there are still barriers to the adoption of cloud technologies in industry. Overall, these results imply the need for more workshops for discussing new ideas and journals to gather the original concepts. After 2015, we notice another trend in how studies refer to cloud computing. Now, cloud computing has emerged as a dominant paradigm and has been accepted as an industrial standard that supports the need of multiple industries (P7, P8, P10, P16, P26, P30). This could be partially due to the launch of AWS IoT from Amazon in 2015, which can be used to build industrial IoT applications. The growing number of conferences and workshops in 2017 could be due to the release of Google Cloud IoT Core, which can be used to connect and manage IoT devices. A systematic literature review conducted on IIoT [24] provides evidence that, from 2015 onwards, the number of publications on this topic has rapidly increased. These results should lead to more conference papers after 2015 as frameworks become available for developing and testing new research ideas.

(28)

Contribution types

Figure 8 illustrates the total number of primary studies per contribution type. According to the extracted data, the main contribution types are model and method (18/42 each), with the rest distributed as follows: framework (4/42), tool (2/42), process (1/42) and metric (1/42). It is interesting to see that only one primary study proposes metrics to assess quality attributes, while others only briefly mentioned the existing ones. These results indicate that the majority of primary studies either contributed with systematic working procedures or models regarding QoS in industrial cloud computing.

Figure 8: Contribution types

Research types

Table 9 illustrates the total number of primary studies per research type. We intended to classify the primary studies into the six categories described in Table 4, but among our primary studies, there were only occurrences of solution proposal and evaluation research papers. The majority of the primary studies are solution proposals (95%), and they propose novel ways on how to define, measure, analyze, evaluate, or improve QoS in industrial cloud computing. Only three primary studies are identified as evaluation research: P28, P37, and P38. P28 reports on an experimental setup for the performance evaluation of SaaS providers, while P37 and P38 report on cloud gaming through a large-scale measurement study and an experiment in which they use a self-developed measurement tool, respectively. P38 is both a solution proposal and an evaluation research paper.

Category No. of papers Paper ID

Solution proposal 40 P1-P27, P29-P36, P38-P42 Evaluation research 3 P28, P37, P38

Table 9: Publications per research type

Validation types

Figure 9 provides information on how QoS approaches have been validated in industrial cloud computing. The majority of primary studies validate their approaches by means of evaluation (24/42), but these approaches are not validated formally nor empirically. The evaluation is mostly simulation-based, and the following tools are used: CloudSim (P2, P7, P8, P10, P15, P16, P20, P24, P26, P27, P31, P34, P36), Matlab (P29), CloudAnalyst (P6), NS2 (P12), and Java Mod-elling Tools (P42). However, a set of 11 primary studies conducts a more extensive validation of the proposed approaches through analysis. The analysis is conducted in the form of case studies using real-world applications (P1, P5, P35, P41), or experiments (P3, P13, P19, P28, P32, P37, P38). A smaller set of primary studies (4/42), focus on validation by example (P18, P21, P30, P40). For instance, the authors in P21 implement and load test two versions of booking movie ticket operations to provide an example of how their lightweight operation approach is able to serve more requests. Only two primary studies (P17, P25) fall in the persuasion validation category as

(29)

they only discuss the potential advantages of their approaches, with no other kind of validation. One primary study (P33) falls in the blatant assertion category as it makes no tangible attempt to validate the results. The absence of experience papers may suggest a lack of maturity in this research area, as existing solutions have not been validated by others yet.

Figure 9: Validation types

5.1.2 Results analysis of RQ2 Quality attributes

In this section, we consider the quality attributes that are addressed the most in industrial cloud computing. Figure 10 illustrates the details of our quality attributes categorization, as found in the 42 selected primary studies. The terms used in the list of quality attributes (i.e., their names) are exactly as reported in the primary studies, without any interpretation from us to avoid bias. Our investigation identified 18 considered quality attributes; the majority of the primary studies addresses performance (∼ 80%) as a relevant quality attribute in industrial cloud computing, fol-lowed by security (∼ 19%), efficiency (∼ 16%), reliability (∼ 14%), availability (∼ 12%), scalability (∼ 7%), and usability, elasticity, stability (∼ 5% respectively). A set of nine quality attributes is only mentioned once (∼ 2%) through all the primary studies, and seven of these attributes are mentioned in P35, which is also the study that identifies most quality attributes (14/18). An in-teresting approach is noticed in P2 where reliability and security are addressed as sub-attributes of trust and in P32 where stability is addressed as a sub-attribute of performance. The overall results indicate a significant gap between performance and all other quality attributes. This suggests that the adoption of cloud technologies in industry is very related to the performance indicators offered by these technologies.

(30)

Factors

In a cloud environment, there are multiple factors that can affect quality attributes. Figure 11 provides an illustration of these factors and their occurrences among the primary studies. Being that cloud technologies operate on the grounds of virtualization (24/42), it is not surprising that quality attributes are frequently affected by virtualization solutions. Virtualization is investigated both as a potential factor that can degrade QoS (e.g., P5 states that virtualization may induce significant performance penalties when facing highly demanding workloads) and as a potential so-lution to QoS challenges (e.g., P23 states that virtualization can improve the overall performance through proper allocation of resources). The primary goal of RQ2 was to identify these factors rather than attempt an in-depth analysis of their potential advantages and disadvantages. Virtu-alization is followed by data storage architecture (7/42). This is not surprising either, considering the fact that enterprises use cloud to store sensitive data, and besides the enormous benefits it provides, it also raises numerous security concerns that a proper architecture can mitigate. Next in line comes the network architecture (6/42), which is mostly investigated in cloud gaming. This is due to the fact that in cloud gaming, the majority of computational operations are performed in the cloud, and, in high-action games, network latency is the greatest concern. Furthermore, cloud providers often have to trade-off between offered levels of QoS and energy consumption; adequate energy-aware solutions can deeply affect offered QoS, especially performance and efficiency. A total of four primary studies (P1, P4, P11, P35) fall into the other category as they focus on providing quality models and mechanisms to rank service providers and are not affected by specific factors.

Figure 11: Factors affecting QoS

Domains

From the 42 primary studies analyzed, 32 (∼ 76%) investigate QoS in cloud computing by pro-viding insights on the use or need of cloud technologies in enterprises and industries, but without a clear indication of specific industrial domains. The other ten primary studies (∼ 24%) identify five industrial domains (i.e., sensor cloud, big data, power trading, mobile, and gaming). Figure 12 provides an overview of the distribution of the primary studies based on their industrial do-main. Authors in P12 state that the integration of cloud computing with sensor network provides useful solutions like efficient data handling for various monitoring applications. In P40, the au-thors emphasize the need for cloud technologies due to the increase in the amount of data and incapability of traditional databases to process unstructured data. Power trading cloud platforms are investigated in P27 because of their peculiarity and high requirements for QoS. The research in mobile cloud (P11, P13) and cloud gaming (P18, P19, P37, P38, P39) is driven by the limited computing capability of mobile devices and thin-clients, which are not able to process high-quality 3D graphics. Overall, the results indicate that even though cloud technologies are very beneficial to various industries and enterprises, research on their use in specific industrial domains still lacks maturity.

(31)

Figure 12: Domains

5.1.3 Results analysis of RQ3 Quality metrics

In this section, we consider the quality metrics that are most often considered in industrial cloud computing. Our investigation identified 28 relevant quality metrics illustrated in Figure 13. The results show that top three quality metrics are response time (∼ 28%), followed by resource uti-lization(∼ 21%) and make span (∼ 17%). Response time is a quality metric that is repeatedly emphasized for the impact it has on the performance of cloud technologies. Thus, being that per-formance is the quality attribute mentioned the most in our primary studies (see Section 5.1.2), this result is not surprising.

Figure 13: Quality metrics

(32)

worth mentioning that, while all other studies only suggest the use of specific metrics, P35 proposes new approaches for assessing quality attributes through an extensive set of formulas, but with no specific terminology regarding the quality metric (e.g., suitability can be assessed by the ratio of the number of non-essential features provided by service to the number of non-essential features required by the customer), thus they are not included as part of the quality metrics list.

5.1.4 Results analysis of RQ4 Objectives

This section investigates the objectives of primary studies with regards to QoS in industrial cloud computing. The results indicate that the most common objective is to improve the QoS offered by the service providers. Indeed, this objective is identified in ∼ 74% of the primary studies, and it is crucial for the adoption of cloud technologies. Even though a considerable difference is observed between the number of studies that try to improve and evaluate, research efforts on the latter do not go unnoticed. A total of nine primary studies fall into this category, where QoS is evaluated for specific purposes (e.g., authors in P3 seek to evaluate the quality of an Infrastructure-as-a-Service cloud, while the authors in P9 seek to evaluate the performance of a private cloud). Focusing on the increasingly large number of cloud providers, three primary studies (P4, P6, P35) investigate approaches on how to select the most suitable one according to specific requirements. Only two primary studies fall into the define category: P1, which defines a quality model named CLOUDQUAL composed of six quality dimensions, and P35, which provides definitions of quality attributes. The low number of primary studies in this category highlights the fact that the cloud computing community is still lacking agreement when defining QoS and quality attributes.

Figure 14: Objectives of the primary studies

Means

A total of 14 different approaches that can help researchers achieve the objectives specified in the primary studies are found in the literature, with 47 occurrences among our primary studies. As shown in Figure 15, the five top approaches are: scheduling (∼ 33%), security mechanisms (∼ 12%), load balancing (∼ 10%), error-prone conditions (∼ 10%), and allocation (∼ 10%). Even though some of the studies present a cause-effect relationship (e.g., authors in P3 investigate how their scheduling mechanism is capable of achieving load balance effectively), in this section we only consider the cause, as our goal is to investigate how they achieve a specific objective. These results imply that the most effort has been put into scheduling, which is not surprising since, with the increasing amount of data that is being processed in the cloud, effective scheduling mechanisms are crucial to provide satisfactory levels of QoS. Furthermore, being that security was the second most mentioned quality attribute in industrial cloud computing (see Figure 10), it was expected that the security mechanisms would be ranked among the top approaches.

(33)

Figure 15: Means of reaching the objectives

Table 10, provides a more in depth analysis of the data items for a more comprehensive under-standing. In the scheduling category, the most frequently mentioned approach is task scheduling (9/14), while in the security mechanisms category the most frequently mentioned approach is en-cryption (3/5). In the error-prone conditions and allocation category we have an even distribution of studies.

Data item Details Primary study

Scheduling Task scheduling P2, P7, P8, P10, P13, P23, P24, P31, P32 Resource scheduling P26, P27, P34, P40

Duty cycle scheduling P12

Security mechanisms Encryption P14, P17, P35 Security protocols P25

Secured data transmission P12 Error-prone conditions Failover system P21 Server failure simulation P20 VM failure simulation P20 Error prone conditions P5 Allocation Resource allocation P6, P19

Task allocation P15, P16

Load balancing N/A P25, P30, P31, P39

Edge servers N/A P18, P37, P38

Queuing model N/A P3, P9

Request traffic control N/A P29, P42

Workload variability N/A P28, P41

SPs coalition N/A P11, P22

Ranking cloud services N/A P4, P35

Quality model N/A P1

Dynamic optimal routing N/A P12

VM live migration N/A P36

Figure

Figure 1: Systematic mapping study process
Table 1: The selected digital libraries
Table 2: Search string based on PICOC
Figure 3: Detailed description of the search and selection process
+7

References

Related documents

46 Konkreta exempel skulle kunna vara främjandeinsatser för affärsänglar/affärsängelnätverk, skapa arenor där aktörer från utbuds- och efterfrågesidan kan mötas eller

The literature suggests that immigrants boost Sweden’s performance in international trade but that Sweden may lose out on some of the positive effects of immigration on

Both Brazil and Sweden have made bilateral cooperation in areas of technology and innovation a top priority. It has been formalized in a series of agreements and made explicit

Generella styrmedel kan ha varit mindre verksamma än man har trott De generella styrmedlen, till skillnad från de specifika styrmedlen, har kommit att användas i större

Parallellmarknader innebär dock inte en drivkraft för en grön omställning Ökad andel direktförsäljning räddar många lokala producenter och kan tyckas utgöra en drivkraft

I dag uppgår denna del av befolkningen till knappt 4 200 personer och år 2030 beräknas det finnas drygt 4 800 personer i Gällivare kommun som är 65 år eller äldre i

Den förbättrade tillgängligheten berör framför allt boende i områden med en mycket hög eller hög tillgänglighet till tätorter, men även antalet personer med längre än

In this research, we have seen how to quantify the performance of QoS of mobile cloud gaming using GamingAnywhere.Primarly, the game AssaultCube is streamed from GamingAnywhere