SYSTEMATIC LITERATURE REVIEW OF SAFETY-RELATED CHALLENGES FOR AUTONOMOUS SYSTEMS IN SAFETY-CRITICAL APPLICATIONS

(1)

School of Innovation Design and Engineering

Västerås, Sweden

Thesis for the Degree of Master of Science (120 credits) in

Computer Science with Specialization in Software

Engineering

SYSTEMATIC LITERATURE REVIEW OF

SAFETY-RELATED CHALLENGES FOR AUTONOMOUS

SYSTEMS IN SAFETY-CRITICAL APPLICATIONS

Miloš Ojdanić

moc16001@student.mdh.se

Examiner: Aida Čaušević

Mälardalen University, Västerås, Sweden

Supervisors: Elena Lisova

Irfan Šljivo

Mälardalen University, Västerås, Sweden

May 29, 2019

(2)

An increased focus on the development of autonomous safety-critical systems re-quires more attention at ensuring safety of humans and the environment. The main objective of this thesis is to explore the state of the art and to identify the safety-related challenges being addressed for using autonomy in safety-critical systems. In particular, the thesis explores the nature of these challenges, the different autonomy levels they address and the type of safety measures as proposed solutions. Above all, we focus on the safety measures by a degree of adaptiveness, time of being active and their ability of decision making. Collection of this information is performed by conducting a Systematic Literature Review of publications from the past 9 years. The results showed an increase in publications addressing challenges related to the use of autonomy in safety-critical systems. We managed to identify four high-level classes of safety challenges. The results also indicate that the focus of research was on finding solutions for challenges related to full autonomous systems as well as solutions that are independent of the level of autonomy. Furthermore, considering the amount of publications, results show that non-learning solutions addressing the identified safety challenges prevail over learning ones, active over passive solutions and decisive over supportive solutions.

Key Words: Safety challenges, autonomous systems, safety-critical system, auton-omy levels, safety measures

(3)

List of Figures

1 Steps in SLR research process . . . 13

2 The study selection process . . . 16

3 The classification of safety-related challenges . . . 22

4 Classes of safety-related challenges distributed throughout the years . 35 5 Safety measure by a degree of adaptiveness throughout the classes of safety-related challenges . . . 37

6 Safety measure by a degree of adaptiveness throughout the years . . . 38

7 Safety measure by a time of being active throughout the classes of safety-related challenges . . . 39

8 Safety measure by decision making throughout the classes of safety-related challenges . . . 40

9 Autonomy levels groups throughout the years . . . 41

10 Distribution of study-relevant publications throughout the years . . . 42

List of Tables

1 Publication distribution by digital libraries . . . 15

2 Publication distribution through the study selection stages . . . 16

3 Publication distribution after the selection Stage 1 . . . 17

4 Publication distribution after selection stage 2 . . . 17

5 Publication distribution based on system and life-cycle challenges af-ter selection stage 2 . . . 18

6 Publication distribution after selection stage 3 . . . 18

7 Publication distribution based on system and life-cycle challenges af-ter selection stage 3 . . . 19

8 Classes of safety-related challenges throughout libraries . . . 27

9 Publication distribution by sub-classes . . . 28

(5)

11 Classes of safety-related challenges throughout domains . . . 30 12 Distribution of autonomy levels groups throughout the years . . . 31 13 Distribution of autonomy levels groups throughout domains . . . 31 14 Safety measure by a degree of adaptiveness throughout the years . . . 32 15 Safety measure by a degree of adaptiveness throughout classes of

safety-related challenges . . . 33 16 Safety measure time of being active throughout classes of

safety-related challenges . . . 33 17 Safety measure decision making throughout classes of safety-related

challenges . . . 34 18 Appendix: Extracted data from publications . . . 67

(6)

1 Introduction

The human factor has been considered as one of the primary causes leading to accidents and triggering catastrophic consequences in safety-critical systems [1]. In order to address this problem, modernized safety-critical systems introduce automa-tion. One way to achieve automation is by using autonomous systems which take control over systems excluding human operator. Introducing autonomous systems into an ordinary service represents the current trend of system development, as they contribute to risk and cost reduction by substituting a human’s ability, or lack of one, to assess and control a system. Depending on the extent to which humans are substituted and how often they need to cooperate with the system, different autonomous levels have emerged corresponding to different levels of human-system interaction. Introduced autonomous systems in safety-critical applications cannot be thoroughly defined during design-time and tested during the test phase due to their unpredictable behavior. Thus, these systems have been mainly designed accounting on continuous environment assessment and environment learning at run-time. This can be often achieved by applying Artificial Intelligence (AI) solutions and various kind of sensors [2].

Safety-critical functions using autonomous technologies are difficult to assure using traditional safety hazard analysis techniques due to incapability of predicting a dynamic behaviour of these systems [3]. It is a well-defined safety principle that a system should be considered unsafe unless justifiably convinced otherwise [2]. For example, the challenge of assuring an autonomous car is similar to the challenge of guaranteeing that it is acceptably safe for a human driver to operate a car. The permission cannot be simply issued once but has to be also constantly re-evaluated. Just as we continuously monitor human drivers to ensure that it is adequately safe for them to drive a car, we need to continuously monitor an autonomous car behaviour and its environment to assure that it is still acceptably safe to operate. Therefore, suitable safety measures are required to be linked with a role that autonomous systems are playing in safety-critical applications [3].

The research focus on autonomous safety-critical systems has been increasing lately. Many industries are struggling with adopting such autonomous systems in safety-critical applications and in the same time keeping the safety property at a jus-tifiable and trusted level. The challenges they face relate to both system engineering as well as life-cycle-wide issues. One example of system engineering challenges is the issue of the interface between the driver and the semi-autonomous car. It is a system engineering problem that requires innovative solutions to keep such hybrid system

(7)

acceptably safe. There are a lot of challenges related to the system engineering part which need to be addressed. As well as there are life-cycle aspects such as safety as-surance, verification, validation, etc. Therefore, a classification of safety challenges related to autonomous safety-critical systems for different autonomous levels and the possible ways to address these challenges is needed to ease the adoption of au-tonomous systems in safety-critical applications. To build such a classification, this thesis proposes a cross domain survey, that covers various application areas such health, automotive, etc. In this thesis we are focusing on system engineering part. The goal is producing a relevant thorough classification of safety-related system en-gineering challenges. In particular, the thesis investigates existing research done in the domain of autonomous safety-critical systems to obtain a better comprehension of the phenomena. In order to extract information and achieve the goal, we con-ducted a systematic literature review. Among other things, we choose the systematic literature review as a research methodology following work done by The Safety of Autonomous Systems Working Group (SASWG) [4]. They stated that reviewing of previous publications in order to understand and identify safety challenges in SCSs applying AS is a valuable contribution for further research.

Furthermore, we investigate which automation levels are in the research focus of the state of the art in the past years, which safety-related challenges are faced with using autonomous systems in safety-critical applications as well as what kind of safety measures have been proposed to address the identified challenges. Extracted knowledge from the results provides an overview of the current state of research focusing on autonomous systems in the state of the art and thus is of interest for both industry and academia. On the one side, this research impacts the industry and serves as a catalog of identified solutions for common challenges when adopting autonomous systems in safety critical systems. On the other side, the thesis provides academic researchers with the insights of safety-related challenges in autonomous safety-critical systems, offering an opportunity to build on top of the domain-relevant publications.

1.1 Problem Statement

Industries nowadays invest a lot of effort and time to introduce autonomous safety critical systems [5]. In parallel, we can observe how much autonomous systems, in a form of artificial intelligence, have an impact on system development. As industries are striving to combine safety critical systems with autonomous systems, this raises a question of reaching justifiable and trusted safety properties of these types of systems, i.e., whether safety properties of these systems keep up with the pace of

(8)

accelerated development. Meanwhile, this includes a lot of challenges related to the system engineering part which need to be addressed, as well as the life-cycle aspects such as safety assurance, verification and validation, etc.

In this thesis we focus on system engineering part. An investigation of common safety-related challenges arising with using autonomous safety-critical systems and what are the possible proposed ways to address them, drive the topic of this thesis and set its overall goal.

Overall goal: To build a knowledge base around the safety-related system en-gineering challenges in autonomous safety-critical systems and the proposed ways to address them.

By addressing the overall goal, the thesis aims at providing a better understand-ing of the state of the art and in that way eases the adoption of autonomous systems in safety-critical applications at the system level. To efficiently address the overall goal, we are formulating research questions to direct the study focus and guide the research.

1.2 Research questions

The following research questions (RQs) are formulated to address the specified over-all goal:

1. RQ1: To which extent has research in the past years focused on exploring safety-related systems engineering challenges in full and semi-autonomous sys-tems?

2. RQ2: What are the safety-related system engineering challenges identified for using autonomous systems in safety critical applications for the two levels of autonomy?

3. RQ3: What kind of the safety measures are proposed to address the identified challenges?

Considering RQ1, the thesis tends to cover the scale at which groups of autonomy levels have been used in the state of the art as well as to what extent autonomous systems in safety-critical systems are elaborated through research in the range of 2010-2018 years.

Taking into account the influence and development complexity which the au-tonomous systems put on safety-critical systems in the range of 2010-2018, RQ2

(9)

contributes by identifying the safety-related system engineering challenges that need to be considered when using autonomous systems on full and semi-autonomous levels in applications that can endanger human life or threaten environment.

Considering in parallel RQ3, the thesis examines what types of safety measures have been proposed to address identified challenges. Furthermore, as part of this question, we further analyze the research trend in addressing different challenges in autonomous safety critical systems. It is important for us to emphasize that we are not going into technical details of those measures, but consider their characteristics.

1.3 Thesis outline

The thesis structure is organized with respect to the predefined systematic literature review development protocol.

Thus, at the beginning of the thesis, Section 1 provides a brief study introduc-tion followed with the subsecintroduc-tions of problem formulaintroduc-tion and research quesintroduc-tions. In order to introduce the technologies concerning the state of the art and to facilitate the understanding of the study, Section 2 covers its general background.

Section 3 describes the research methodology including the proposed systematic lit-erature review study phases. In Section 4 the thesis results are summarized.

Section 5 presents an analysis of the results and further discusses the research ques-tions and presents study observaques-tions. The end of the section covers the study validity threats. In Section 6 the related work is presented and finally, conclusions of the study can be found in Section 7.

(10)

2 Background

This section provides some essential background information needed for understand-ing the research behind this thesis. In particular, it gives introduction of safety-critical systems and the corresponding safety terminology, as well as autonomous systems and their levels of autonomy. Furthermore, this section elaborates on Sys-tematic Literature Review (SLR) research methodology by providing the standard process guidelines as well as a detail description of the process phases.

2.1 Safety critical systems

The concepts and terminology presented in this section are mainly obtained from In-ternational Electrotechnical Commission (IEC) 61508 Standard for Functional safety of electrical/electronic/programmable electronic safety-related systems [6]. However, in this section we are presenting definitions from the standard through our own understanding of those.

Safety Critical Systems (SCSs) are systems whose malfunctioning can lead to loss of human lives or cause damage to the environment [7]. These systems are being deployed in nearly every domain and our daily life would not be possible without them. We are relying on them to not only provide us with the correct service, but also to be acceptably safe while doing so. However, acceptably safe cannot be measured. We can observe it as a risk of jeopardizing a human or causing damage to the environment. Therefore, safety must be particularly addressed with the goal to reduce the risk to minimal levels. One of the ways of achieving acceptably safe systems is by decreasing failure rates to acceptable levels. Reduction of the risk can be done by mitigating the faults causing the service failure or tolerating the faults when their mitigation is not possible. Given that bringing the risk to zero is not possible, there is no absolute safety, i.e., a system cannot be completely risk-free. There is always same kind of a residual risk, which is the remaining risk when all failure causes are analysed and handled. Consequently, the process of developing a set of safety-related services is a challenging process. To achieve a safety-related development accuracy and provide an answer on how efficient a system needs to be in order to be safe, different metrics have been used depending on the domain. For example, the aviation domain introduces the metrics which state the roughly precision, in designing SCSs to be acceptably safe, is to lose one life per billion hours of operation [8]. To achieve metrics leading to such efficiency, SCSs must guarantee dependability in stages of the development process. Numerous stages of the SCSs development process introduce various challenges depending on the

(11)

objectives. Those stages create the life-cycle of SCS.

Through the development life-cycle, a safety case is developed and maintained. The safety case is a collection of documents, goals, arguments and evidences which provide reasoning why the system is justifiable safe enough to not produce a hazard [5]. As it is mandatory by the specification of the safety case, SCSs need to achieve a set of dependability attributes. These attributes relate to system performance and abilities required to not to jeopardize human lives or environment. One of the abilities is correct service continuity, which means that the system functions shall be performed on demand. Further, the system holds the ability to avoid a service failure in presence of faults caused by external or internal effects. In case of faults or errors, the system should be able to recover itself from the erroneous state into a dependable operational state. Additionally, among described system abilities there must be an ability to preserve a potential exposure of confidential system information to the unauthorized sides.

Overall, the entire collection of definitions, assessments, concepts, designs, devel-opments, and testing needs to be performed and included in the related safety-case in order to label the system as being acceptably safe. Accordingly, to understand safety and all previously defined collections, there are numerous terms and defini-tions to be familiarized with. Thus, in the following, we are presenting some of the safety terms and definitions relevant for our study.

2.1.1 Safety terminology

In this thesis, safety is considered as freedom of unacceptable risk [9], where a risk means probability of endangering human lives, environment or causing harm. Safety as a system property can only be justifiably evaluated by observing different aspects of the system as a whole, including hardware, software, humans, environment behavior etc.

A system consists of interacting units, i.e., components. Mutual cooperation produces system behavior and provides a user with an appropriate service. The service is perceived by the user as a set of external system states [10]. Now, dependability of that service is observed as a system ability to deliver the correct service and acceptable level of trustworthiness. For a system to be dependable, it needs to deal with a lot of different imperfections. These imperfections of systems can be perceived through a fault-error-failure model. A fault can be observed as an event causing a system state to change. This event, which can be external or internal, can influence system safe behavior by causing an error. An error is a part of the system state which deviates from the expected and correct one, with a

(12)

potential to reach external system behavior. Reaching the external system behavior means the error is not handled and if propagates can lead to a service failure. A failure means that functional interacting components of the system are unable to provide a required function [9]. With inadequate external system behavior, the system may produce a hazard that can harm people or the environment. A hazard is defined differently depending on a safety critical domain. However, in this thesis we consider a hazard to be "a set of external system behaviors that together with a set of worst-case environmental conditions, will lead to an accident" [11].

In order to create a safe system, i.e., acceptably safe system, various safety measures are applied. Safety engineering includes safety measures which should prevent potential hazards, increase safety and decrease corresponding risks. They can be characterized according to different categories. Categories which are of in-terest for this thesis research are classified by activity time, decision making and them being adaptive. Safety measures according to time of them being active can be active and passive. An active safety measure works continuously in order to prevent accidents. These are system features which stay active all the time while the system is operating, e.g., anti-lock braking system. In contrast, a passive safety measure does not work until it is called, e.g., airbags. According to their decision making, safety measures can be categorised on decisive and supportive. A de-cisive safety measure makes a decision about the exposure of a hazard on its own, taking necessary steps to prevent it. While a supportive safety measure helps an operator of the system to make the right decision. Considering the degree of safety measure adaptiveness, they can be learning or non-learning. A learning safety measure has the ability to provide knowledge gained through experience and distinguish what is the best suitable action to take in order to increase safety and prevent hazards. Meanwhile, a non-learning safety measure provides an ac-tion which is planned and anticipated by engineers of the system. Described safety measures can be further categorized by their goal. The goal of the safety measures is physical safety and data-safety. In circumstances where an individual is not ex-posed to hazard and cannot be harmed, the term physical safety is used. On the opposite side, data safety concerns the protection of losing the data. In this thesis, we are taking into consideration the physical safety.

2.2 Autonomous systems

Autonomous systems are systems that are able to "change behavior in response to unanticipated events during operation" [12]. An application and study of these sys-tems in various domains draw attention of industry and research. The reason for

(13)

this growth and applicability of the autonomous systems lies in multiple benefits they are providing. While a human factor is the main cause leading to accidents in traditional systems operated by humans, to try and address the human factor as the main cause of accidents, automation is increasingly used to replace or augment the human role in operating these systems. In particular, autonomous systems as a way of achieving automation provide a way to substitute the human factor in operating SCS. Considering operating conditions the main benefit of autonomous systems lies into replacing the human’s ability to operate complexities, i.e., to base decisions on wide amount of data and to make safety-important decisions quickly. Autonomous systems are mainly designed utilizing and relying on AI solutions. This type of technology enables a system to observe real-world environment as external data through various types of sensors, reason about it and learn from it so that in the next iteration the "reasoning" operation can be influenced by experience. Af-ter processing data or "thinking" operations, the system should produce a rational controlled behavior satisfying a specific goal. This cycle of mimicking human intel-ligence has been achieved with a multidisciplinary approach that combines different sciences, e.g., computer-science, mathematics, neuroscience, psychology etc. Once these disciplines are synchronised, they are capable of achieving comprehension of the unpredictable complex real-world environment. Thus, depending on a degree of automation and extent of mimicking the human intelligence, these systems can be categorized in to several levels.

2.2.1 Autonomy levels

The autonomy levels were first introduced by Society of Automotive Engineers (SAE). In this section we are describing each of those levels following categorization of SAE [13].

Different operational levels have been introduced to classify and present limi-tations of autonomous systems. These levels can be observed and represented like a progression of the technology reaching a fully autonomous system. A full au-tonomous system is a system which completely mimicking operator actions and makes decision on its own. On the other hand, different levels are providing us with answers on how much a system operator needs to control the system. A system where an operator is included into making decisions we consider as a semi-autonomous system. There are undesirable effects of the system’s uncontrolled behavior, which emerge into various system engineering safety challenges. With this in mind, au-tonomous systems can be categorized in the following levels:

(14)

• Level 0: No automation

At Level 0 there is no automation, which means the operator manages all control operations on his/her own.

• Level 1: Operator assistance

At this level, the operator has at least one system feature providing relevant assistance information regarding the control. However, control is still strictly in the hands of the operator.

• Level 2: Partial automation

In partial automation, the operator has several systems features providing the control management. Still, the operator needs to be fully attentive and monitor the system during divided control.

• Level 3: Conditional automation

At this level, the system is able to take all management of the control if the operator accepts it. However, in case of an obstacle, the operator needs to take over full control and neutralize a trouble.

• Level 4: High automation

A system has the ability to perform services on its own and take control without a need for the operator to intervene. However, systems in this category have potentially limitations involving loss of control in certain conditions.

• Level 5: Full automation

At this level, the system maintains control in all conditions with the ability to deal with potential obstacles and recover from certain errors.

In this thesis we are considering levels 1,2 and 3 together and refer to system which have these levels of autonomy as semi-autonomous. And by full autonomous systems we understand grouped together systems of levels 4 and 5.

2.3 Systematic Literature Review (SLR)

In this section we present the SLR research methodology based on the SLR guide-lines described by Kitchenham [14]. The SLR methodology is conducted in those situations when research requires identification of all relevant evidences concerning a particular state of the art.

(15)

Research studies can be classified as primary or secondary. Primary studies are singular performed studies addressing the specific research problems and contribut-ing to the existcontribut-ing knowledge by applycontribut-ing scientific methods. Secondary study uses the existing knowledge to make contributions and provide answers. SLR is a sec-ondary study where it thoroughly evaluates and in-depth defines the material of interest from all topic-relevant primary studies.

However, the huge number of primary studies has created an environment within which a lot of work is redone, rather than built on top each other [14]. SLRs represent one of the solutions for this problem. According to Kitchenham [14], the advantage is in summarizing evidences needed to enable insight into the current state of the art regarding studied phenomena. Furthermore, it identifies gaps in the existing research in order to suggest areas for further studies, providing an opportunity to build further upon the existing work without repeating it [14].

All these reasons make SLR differentiate from other literature reviews. In order to achieve all of its advantages, SLR thoroughly analyzes existing work. The SLR process itself is tailored based on the type of study and studied phenomena such that different phases may be used for different studies.

According to the guidelines from Kitchenham [14], the SLR process starts by constructing a development protocol, which among the other things first defines the research questions to be studied as well as what are the methods to be used to perform the review. This development protocol also specifies which search strategy needs to be used to thoroughly collect as many as possible topic relevant primary studies. The search strategy results with a set of topic relevant primary studies, which need to be documented in order to analyse the data of interest. Moreover, SLR has to have a proper selection criteria used for forming inclusion and exclu-sion strategies for studied existing works. On the one side, the incluexclu-sion strategy considers and provides an answer why the primary study can be incorporated into the systematic review and why it is considered as relevant. On the other side, the exclusion strategy provides reasoning why the primary studies are not considered as relevant for studied phenomena.

After filtering relevant primary studies, the next SLR phase is selecting which data is in focus and which data needs to be extracted from each study. The data sample depends on the research area and research questions. Still, independently from such, the SLR research process needs to include quality criteria for assessing the relevance of primary studies.

All of the SLR characteristics used during the research process development of the thesis, can be categorized into three high-level stages:

(16)

• The first stage includes planning the review . Within this stage a necessity for the review is evaluated. The discussion develops topic appropriate research questions, which further influence the development of the review protocol.

• The second stage is conducting the review process by filtering all pri-mary studies from the predefined scope of the work. The scope of the work includes all sources with the topic-relevant materials. This stage further in-cludes quality assessment evaluating material relevance, data extraction for selecting relevant data from identified material, and data synthesis where the extracted data is summarized.

• The last stage of SLR is reporting the review , during which the review results are disseminated and structured to be reasonable and easy followed by a reader.

(17)

3 Research method

In this section, we adjusted the SLR research methodology with the purpose to satisfy the objectives of the thesis.

This section is organized to follow the execution of the study process and presents data obtained by the conducted protocol. The protocol includes:

• The formulation of research questions • Description of a publication search process • A study selection criteria

• A quality assessment • Data extraction

The objective of this research process is to analyze all relevant papers associ-ated with the topic of interest in order to achieve the overall thesis goal stassoci-ated in Section 1.1. To achieve this goal, we classify safety-related challenges caused by using autonomous systems in safety-critical applications from publications identi-fied as relevant. Furthermore, we also identify safety measures proposed by those publications to address the identified challenges.

To achieve this, the thesis follows the aforementioned protocol. As stated, the created protocol offers various advantages for our study. The three of those are de-scribed in Section 2, providing adequate strategy for achieving our overall goal and answering on corresponding research questions. Besides those advantages, the most relevant one to us is that the SLR process provides an opportunity to thoroughly report the state of the art of the studied phenomena, with the intention to summa-rize the empirical data and gain knowledge based on a relevant data analysis [14]. The results from SLR, summarized by this thesis, are analyzed to provide a classi-fication of challenges related to the safety of autonomous systems in safety-critical applications.

However, the following process developed for this thesis is an adjusted version of Kitchenham [14] process of work. The difference is that Kitchenham process of work is a guideline process which besides developing strategies for the study selection phase, includes phases for synthesis of extracted data, a dissemination strategy as well as a review schedule. The main phases and steps proposed within guidelines of Kitchenham, based on which the development protocol for this study is created, are described in Section 2.3.

(18)

The developed process consisted of thoroughly designed steps is depicted in Figure 1. Additionally, these steps form a research direction which enable us to identify safety-related system engineering challenges arising by usage of autonomous systems in safety-critical applications including the proposed safety measures for addressing the challenges and preventing any cause of potential severe accidents.

Figure 1: Steps in SLR research process

Figure 1 presents the developed protocol adapted for this thesis. The protocol represents the SLR research process which consists of 6 sequential steps. Following the protocol steps from the bottom up, the process starts by identification of re-search questions. To achieve the thesis overall goal, the thesis covers three rere-search questions which will be further elaborated. After the identification of the research questions we created the adequate research string based on the research questions topic of interest. The search string is a Boolean expression, which represents a query used for searching through publication sources. During the next step of the process described in Figure 1, relevant publication sources are identified. These sources are identified assessing material quantity which is relevant for the topic of interest.

The next step of the SLR process is a thorough study selection criteria. The study selection criteria step further branches into several stages. In each stage is necessary to go through all recognized relevant publications, including the fact that in each stage different rules apply for labeling publications as study relevant. Each stage of the study selection criteria process includes or excludes publications from

(19)

the further study in the sequential order based on their titles, abstracts, full texts and relevant data collection. Furthermore, after creating a finalized set of the study relevant publications all relevant data is extracted and the publications topics are summarized in the process phase of Data extraction. After selecting and extracting data from the study relevant publications, the next step in the research strategy is performing data analysis. We analyse the data in the Data mapping phase by doing cross mapping between different attributes of the publications data. In particular, during the phase of Data mapping, which is the last phase of our SLR process, all papers are categorized based on their content. On these sets of papers, the descriptive statistics is applied. Next, through data analysis, the obtained results are analysed using descriptive statistics features likewise histograms, scatter plots to get an understanding of the data that has been collected [15].

Firstly, to start the process we need to determine research questions tightly correlated with the topic of interest. In the example of this thesis, we introduced the research questions in Section 1.2 and with those we will be guided throughout all phases of the process.

3.1 Search process

The SLR research process starts by identifying the research questions to support the overall thesis goal. When it is started, the next step is Specification of a research string. Note, the search string represents a logical statement of the keywords, which joined together are used to find relevant publications in the predefined sources. For the purpose of this thesis, identified relevant sources are digital libraries through which the specified search string is applied. Digital libraries used for the study are IEEE Xplore digital library 1_{, ACM digital library} 2_{, Springer Link}3 _{and Web of}

Science4_{. Meanwhile, these digital libraries form the scope of this study where each}

publication listed from a libraries search engine using a search string, is manually processed and analyzed. These libraries, are filtered by the year range of 2010-2018. Besides filtering publications by the year range, we are also focusing on peer reviewed publications, which include conference papers publications as well as journal articles. Furthermore, since it is difficult to focus only on the system engineering safety-related challenges based on the search string, we identified publications that focus on both the system and life-cycle related challenges, and differentiate between them

1_{https://ieeexplore.ieee.org/Xplore/home.jsp} 2_{https://dl.acm.org/} 3 https://link.springer.com/ 4 https://webofknowledge.com/

(20)

in the selection process starting from the Stage 2. The search string we use is:

"autonomous AND safety AND critical"

After applying the search string to the digital libraries, their search engines listed different numbers of identified publications. As demonstrated in Table 1, the number of publications listed by the IEEE database was 526, the ACM and the Web Of Science databases listed 69 and 474 topic relevant publications while the highest number of publications, 4309 in total, was listed by the Springer database. These obtained numbers of publications represent filtered publications which topic to some extent contains specified keywords from the search string. All those queried publications are included in the next phases of the SLR process, where their content is further analyzed.

After identification of publication sources, the next step in the SLR process is the Study selection criteria.

Table 1: Publication distribution by digital libraries

Digital library Address # of publications

IEEE https://ieeexplore.ieee.org/Xplore/home.jsp 526

ACM https://dl.acm.org/ 69

Web Of Science https://webofknowledge.com/ 474 Springer https://link.springer.com/ 4309

3.2 Study selection criteria

Study selection criteria is a step into the process within which the identified publica-tions from the previous step are filtered until the set of study-relevant publicapublica-tions is finalized. A publication is considered as topic relevant when its topic concerns the overall goal and contains data which concern the research questions.

Figure 2 depicts the Study selection process. The process contains several phases through which each publication is going through in order to be considered as rele-vant. Filtration is performed manually, taking into consideration inclusion criteria, declaring which publications are relevant enough to go into the next stage. After the initial stage of the study selection process that included applying the search string to considered digital libraries, the total number of identified publications is 5378. A publication distribution by study selection stages is depicted in Table 2. The next stage is applying the inclusion/exclusion criterion to the publication titles. In this

(21)

Figure 2: The study selection process

stage, the inclusion/exclusion factor concerns publications domain of interest as well as topic of interest defined by the research questions. If a publication belongs to a study relevant domain of interest and covers studied phenomena, it is considered acceptable for the next stage. In particular, publications considered relevant while being filtered in the first phase are all those publications which are covering safety in SCSs and which in the same time study applications of automation in safety critical systems. While excluding criteria is a filter by which publications are rejected for the next stage, i.e., if they are not in the domain of interest or do not concerns phenomena relevant for the research questions. After applying inclusion/exclusion criterion to the papers titles we have identified 913 relevant papers for the further study.

Table 2: Publication distribution through the study selection stages

Stage Activity Publications

0 Applied the search string to all the sources and gathered the results 5378 1 Applied inclusion/exclusion criterion to the papers titles 5378 2 Applied inclusion/exclusion criterion to the papers abstracts 913 3 Applied inclusion/exclusion criterion to the full texts 176 4 Finalised the set of included papers 108

A distribution of the papers after applying inclusion/exclusion criterion to the papers titles is presented in Table 3. This table contains the numbers of accepted publications categorized by digital libraries, i.e., numbers of all those publications which we consider relevant for the next stage of our study selection process. Fur-thermore, the table presents the numbers of rejected publications, i.e., which we considered as not relevant for our study. Furthermore, the table contains rates

(22)

of accepted and rejected numbers. These percentages serve with the purpose to compare the rate of the accepted and rejected publications from the total sum of papers, and like that provide a view on coverage each database holds concerning safety-related challenges topic of interest.

Table 3: Publication distribution after the selection Stage 1

Digital library Accepted Rejected IEEE 244 (46 %) 282 (54 %) ACM 27 (39 %) 42 (61 %) Web Of Science 163 (34 %) 311 (66 %)

Springer 479 (11 %) 3830 (89 %)

In the IEEE digital library we identified as relevant 244 publications. All those publications had satisfied conditions of inclusion/exclusion criterion for their titles and consequently went to the next phase so that their topics can be studied more in depth and data relevant for the study of this thesis extracted. Considering other digital libraries in the selection Stage 1 from Table 3, we can observe the number of accepted publications for the next phase of selection in the ACM digital library is 27, and the digital library Web Of Science correspondingly contains 163 relevant publications. Finally, in the digital library Springer, we accepted 479 publications for the next stage.

Table 4: Publication distribution after selection stage 2

Springer 208 (44 %) 271 (56 %)

After applying inclusion/exclusion criterion to the papers’ abstracts we identified the publication distribution presented in Table 4. During this stage, we identified various safety-related challenges in SCSs applying AS concerning system engineer-ing as well as life-cycle. Distribution between these papers is presented in Table 5. On one side, life-cycle challenges include all those papers whose topic covers safety during the system development process, e.g., system analyses or safety assurance. Next, the life-cycle category includes verification and validations techniques and approaches for assuring that a system meets its requirements and that the system operability is as expected. Another example of the topics classified as life-cycle is different kinds of hazard analysis, which means necessary processes for assuring

(23)

sys-tem safety. On the other side, the syssys-tem level covers those publications whose topic include safety-related system engineering challenges related to achieving higher level of automation in a system and their corresponding safety measures for addressing these safety challenges.

The reason we did not differentiate these two types of challenges in previous selection stage, i.e., filtering based on titles, is the lack of accuracy and information to base a decision on.

Table 5: Publication distribution based on system and life-cycle challenges after selection stage 2

Digital library System Lifecycle

IEEE 62 76

ACM 10 11

Web Of Science 26 24 Springer 78 130

Following the distribution of numbers depicted in tables 4 and 5 for the IEEE database, we identified 106 publications as not relevant for the further study, which gave us 62 publications accepted at the system level and 76 publications accepted at the life-cycle level. In digital library the ACM from 27 publications, we excluded 6 publications from the further study, and classified 10 publications as belonging to the system level and 11 publications to the life-cycle level. Next, going through 163 publications from the Web Of Science digital library, we rejected 113 publications and identified as relevant for the further study 26 publications at the system level and 24 publications at the life-cycle level. Finally, given 479 publications from Springer, we identified as not relevant for further study 271 publications on the one side, and on the other side, we identified 78 system level publications in contrast to 130 publications belonging to the life-cycle level.

Table 6: Publication distribution after selection stage 3

Springer 52 (67 %) 26 (33 %)

In the last stage of the selection process, we went through 176 papers in total. This further means we applied the inclusion/exclusion criterion to the full texts of publications identified as relevant for the study of this thesis until this step. Results

(24)

from this stage are presented in Table 6. While distribution of publications based on system and life-cycle challenges is presented in Table 7.

Table 7: Publication distribution based on system and life-cycle challenges after selection stage 3

Digital library System Lifecycle

IEEE 44 3

ACM 9 0

Web Of Science 13 0

Springer 42 10

Following the distributions in tables 6 and 7 the results are as follows. The digital library IEEE contains 44 system level publications identified as relevant for our study, out of 62 from the previous stage, where 3 publications are classified as belonging to the life-cycle level. From the ACM digital library, 9 system level publications are relevant for our study, wheres 1 publication is rejected and we did not identify any life-cycle level publication, which is the total sum of 10 publications from the previous phase. Next, from the total number of 26 publications in Web Of Science from the previous phase, we identified 13 system level relevant publications, all of which are belonging to the system level and rejected 13 publications as not relevant. In the end, in the Springer digital library, we filtered through 78 publica-tions from previous selection stage, and identified 42 as study relevant system level publications, 10 as life-cycle publications and rejected 26 publications based on their full text.

During the selection process, we found duplicates across digital libraries. All duplicates have been eliminated in the order of their appearance.

After completion of the selection phase, which is the fourth step into the SLR process, we identified 108 publications as relevant for our study and the thesis overall goal. The next phase in the research process is data extraction for collecting all data of interest from 108 identified as relevant publications. The extracted data is further mapped and analyzed in order to answer the research questions.

Before elaborating on which data was identified as relevant for extracting from each publication, the thesis provides a short overview of the principles of work we followed while conducting study selection stages in order to increase the quality of filtering.

(25)

3.3 Quality assessment

In order to be thorough during the selection process and increase the research qual-ity, we chose to follow a study quality assessment plan for study selections which was specified during the planning of the research process. The plan incorporates ad-ditional reviews of rejected papers from the study during the first two phases. Thus, each stakeholder of the thesis, i.e., the thesis author and supervisors, provides an additional review with the opinion which paper should be considered and accepted for further filtration in the selection stages. The decision to cover the first two phases with extra check came with the understanding of the study scope, concerns to not reject relevant papers in early phases and severity of studied phenomena. Note, that only the thesis author went through all the paper, the additional review conducted by supervisors concerned only rejected papers, in this way leaving the author opinion as the main and just additionally ensuring the quality of the selection.

Another decision emerged from the quality assessment plan, concerns decision-making while performing study selection filtering. We decided to be pessimistic while performing the filter process, more defensively deciding which publications are relevant for the studied research questions. This means that each situation of ambiguity where the topic was not clear by just considering studying stages focus (titles, abstracts) was approached by deciding to accept the paper to the next stage. Directed by these rules and the plan, we categorized and decided upon every publication and its topic of interest.

3.4 Data extraction

After completing the study selection phase, the next phase of our SLR process is Data extraction. According to the thesis goal, in this phase we focus on the publications dealing with the system engineering challenges. In this stage, we extract the following data from each publication identified as relevant in the final set of our study:

1. The publication Year: We constrained the study within the year range of 2010-2018. Thus, the field Year should be a year from that range.

2. The Domain: to which paper content or a considered application can be related to. After performing data extraction the identified domains are au-tomotive, air, maritime, construction, medical, robotics and at the end, the safety challenges which were not specifically related to a particular domain, were marked with a keyword generic.

(26)

3. The Safety category: This type of data has been extracted based on a considered safety challenge so that it can be further analyzed and classified during the next phases of the research process.

4. The Safety challenge: That has been in the focus of a paper identified as relevant.

5. The Solution to safety challenge: A solution proposed for addressing the corresponding safety challenge.

6. The Autonomy level: As we described in Section 2.2.1, we are considering systems that have autonomy levels 1,2 and 3 as the semi-autonomous systems. While for the full autonomous systems we are considering systems with au-tonomy levels levels 4 and 5. However, for safety related system engineering challenges which can emerge on all levels of autonomy, we are considering them as generic, i.e., we extracted that data with the keyword generic.

7. The Degree of safety mechanism adaptiveness: Proposed in the solution to the safety challenge. As stated in the Section 2.1.1, we were focused on learning safety mechanism and non-learning safety mechanism.

8. The Safety mechanism time of being active: This type of extracted data included active safety mechanism and passive safety mechanism. Both of the terms were elaborated in Section 2.1.1.

9. The Safety mechanism decision level. Depending on when a safety mech-anism is active, a safety mechmech-anism was marked as active or passive. Mech-anisms belonging to the former were also further classified as decisive and supportive safety mechanisms.

All results with the corresponding data and numbers, extracted from each iden-tified study relevant publication are further presented through tables and analyzed across the last phase of data mapping in our SLR process in Section 4.

(27)

4 Results

In this section, we present a classification of safety-related system engineering chal-lenges based on 108 publications identified as relevant. Note, we consider that each publication focuses primarily on one safety related challenge. Hence, each publica-tions has been mapped to one individual safety related class. Furthermore, we are presenting a safety-related challenges classification tree and the mapping of results based on the extracted data attributes.

4.1 Identified safety-related system engineering challenges

Figure 3: The classification of safety-related challenges

Based on the research, we identified and classified safety challenges which we found relevant from the SLR process. The classes of safety-related challenges are presented in Figure 3. We elaborate in more details each of the classes in the reminder of this section.

4.1.1 Human-Machine Interface (HMI)

The first class of safety-related challenges that we found relevant, incorporates the interaction between a human on one side and machine/system on other side. Based

(28)

on interaction objectives, different safety-related system engineering challenges can emerge, i.e., the interaction can produce faults that can propagate to failures that can lead to hazardous situations. We distinguish between two sub-classes of chal-lenges:

1. Operator assistance

Watching and supervising operator behavior, i.e., predicting operator actions, providing him or her with the support information about the environment and upcoming events as well as helping the operator by providing warning information about a potential threat imposed by the system to other humans.

2. Control handover

Providing a driver with information that a system needs action to be taken by the driver, i.e., cannot process event by itself and the driver has to take manual control over a system. It also include warning information about command takeover during potential hazards and system constraints. Satisfying the right time-frame for takeover is a challenging process. It is often achieved through sounds, lights, vibrations or different adjustments of a steering wheel.

4.1.2 System To System Interface (S2SI)

The next identified class of safety challenges address the interaction between systems. In order to efficiently operate, the systems need to exchange different information. From this interface, various hazardous scenarios can emerge causing different safety system engineering challenges. In general, the challenges which need to be addressed in this class relate to communication between systems through which they exchange information. The information includes experience encountered on the path, the data about accidents and warnings, the traffic density, approaching vehicles for perform-ing a lane change, formation of clusters or platoons, the satellite navigation for cooperative positioning, the inter-vehicle spacing, adjusting speed and braking. For example, this category includes organization for safe operation through intersections and roundabouts. The operation needs to be done by the systems informing sys-tems about the state in the loops and interchanging navigation and support in the sense of crossing priority, the traffic frequency, warnings, approaching time, the lane selection.

(29)

4.1.3 Environment Sensing (ES)

Another safety challenges class concerns the interaction between a system and the environment. The system ability to sense the environment in order to provide a correct acceptable safe service is influenced by different safety challenges that can emerge from different aspects of the environment. We distinguished between three sub-classes:

1. Object detection

This group is focused on a system ability to recognize and detect a different kinds of static and dynamic objects. This implies moving objects, e.g., pedes-trians, detecting and tracking other systems alert signals, e.g., breaking lights and other kinds of turn signals on the other systems. By static objects, we are assuming different kind of road signals, road lanes, etc.

2. Weather conditions

Various types of weather conditions can be challenging for an autonomous safety-critical system. This includes e.g., storms for the aviation domain; rainstorms, fog for the automotive domain; tsunamis and hurricanes for the maritime domain.

3. Positional uncertainties

This group incorporates safety challenges for localization, indicating a system need to know about its surrounding with a high precision, i.e., “where it is” at the moment. To be able to do this complex operation a system needs to perform local and global path planning.

The local path planning incorporates challenges for avoiding unexpected and unforeseen obstacles/collisions. A system must be able to sense and bypass obstacles and at the same time look forward to pursuing its intention [16]. On the other side, global path planning includes challenges concerning navi-gating a system route from the start point to the end point. Systems must be able to find an obstacle-free path which is planned beforehand [16]. Those paths besides necessity to be obstacle free, need to be fuel efficient, i.e., the path need to be optimal. Furthermore, navigation may need to be re-planned while the system is in operation in the case of critical situations.

(30)

4.1.4 Behavioral Adaptations (BA)

The last identified safety class concerns system performance. All possible behaviors of AS operability cannot be planned and known in advance. There are several steps to deal with the uncertainty that comes from the unpredicted environment and thus make a system adaptive. It is needed to monitor system operation at run-time, to adapt to uncertainties that may cause faults, avoid failures and thus keep system acceptably safe and adaptive. In particular, system run-time monitoring should be performed with the goal of identifying outcomes that may lead to violation of safety-critical properties of the system. This helps detecting that the system may potentially enter an unsafe state. In case the system enters an unsafe state, i.e., it was not prevented, appropriate actions should be taken to lead it to adequately safe state.

Based on the monitoring goal and the way the safety challenge is addressed, we distinguish between two sub-classes:

1. Adaptive fault detection

This group includes detection of violations that can bring a system into an unsafe state, different advanced monitoring approaches for fault detection and strategies to detect a fault. The system needs to have an ability for threat assessment, i.e., to possess adaptive nature while identifying there is a fault that can lead to a failure which can produce hazardous situations. Adaptive detection of faults is often achieved by conforming to a set of predefined rules.

2. Adaptive failure mitigation

This group incorporates an evaluation of a potential failure caused by un-certainties faults, performing necessary actions to address the problem and to bring a system into a safe state. This kind of safety challenges includes system’s need to be self-adaptive, i.e., necessity to control and to change be-havior based on experiences, build knowledge and make own decisions, a need to adapt to the uncertainties and operate correctly.

4.2 Data mapping

In this last phase of the SLR process, we are mapping data extracted from the 108 publications identified as relevant. The data is mapped through consideration of the different attributes, presented in the Section 3.4. In the following, we have identified several categories of mapping to present throughout tables:

(31)

• Classes of safety-related challenges throughout libraries • Publication distribution by sub-classes

• Classes of safety-related challenges throughout the years • Classes of safety-related challenges throughout domains • Distribution of autonomy levels groups throughout the years • Distribution of autonomy levels groups throughout domains • Safety measure by a degree of adaptiveness throughout the years

• Safety measure by a degree of adaptiveness throughout classes of safety-related challenges

• Safety measure time of being active throughout classes of safety-related chal-lenges

• Safety measure decision making throughout classes of safety-related challenges A description of the each category is provided below.

Classes of safety-related challenges throughout libraries

The first category of mapped data is shown in Table 8. In the table, we are pre-senting identified classes of safety-related challenges while using AS in SCSs across studied digital libraries. This data provides us with knowledge about safety-related challenges across different sources. Thus, the digital library ACM gave us 3 publica-tions concerning HMI challenges, the IEEE digital library resulted in 1 publication, the Springer database had 4 publications studying HMI, while the Web Of Science database didn’t provide us with topic regarding HMI challenges. All libraries in to-tal resulted in 8 publications on the HMI class of safety-related challenge. The next mapped class across different digital libraries is S2SI. This class of safety-related challenges resulted in a total number of 12 publications. This total number of pub-lications is distributed across 1 publication from the ACM digital library and 4 publications from IEEE, Springer had 6 publications concerning this class of safety-related challenges, while the Web Of Science gave us 1 publication concerning the S2SI class of safety-related challenge while using AS in SCSs.

Further, we mapped the BA class of safety-related challenges. This class contains 4 publications from the ACM library, then IEEE resulted in 17 of these publications,

(32)

Table 8: Classes of safety-related challenges throughout libraries HMI S2SI BA ES ACM 3 1 4 1 IEEE 1 4 17 22 Springer 4 6 12 20 Web Of Science 0 1 6 6 Total 8 12 39 49

whereas Springer had 12 and finally the Web Of Science provided us with 6, resulting in the total sum of 39 publications for the BA class of safety-related challenges. ES is the last class which we associated with digital libraries in this category of mapping. The ES class provided the highest number of publications. The total sum is 49 publications. This amount of publications is distributed as follows: 1 publication came from the ACM digital library, 22 of the total number of publications concerning safety-related challenges in the ES class were provided by IEEE. Springer resulted in 20 publications, while Web Of Science resulted in 6 publications covering ES safety-related challenges.

Publication distribution by sub-classes

Furthermore, we are presenting a distribution of papers covering identified sub-classes of safety-related challenges, incorporated into our classification. The distri-bution is presented in Table 9, where we can recognize identified classes of safety-related challenges and their associated sub-classes. Over the two sub-classes of the HMI class, we mapped 5 publications to the Operator assistance sub-class of safety-related challenges, while for the Control Handover sub-class we classified 3 publications. As we discussed in the section 4, the S2SI class is not decomposed fur-ther into sub-classes, which means the total number of publications for this class is 12. The EN class contains three sub-classes, where the publications are distributed on 14 publications for the Object detection sub-class of safety-related challenges, 1 publication for the sub-class Weather conditions and 34 publications for the Po-sitional uncertainties sub-class of safety-related challenges. In particular, while we were extracting and mapping data from the final set of publications concerning identification of safety-related challenges for the Positional uncertainties sub-class, we were able to recognize 15 publications for precise Local path planning safety-related challenges in the contrast of 10 publications investigating specific Global path planning safety-related challenges. The last number of publications concerns the BA sub-classes, which resulted in 15 publications for the Adaptive fault

(33)

detec-tion safety-related challenges in contrast to 24 publicadetec-tions for the Adaptive fault mitigation sub-class of safety-related challenges while using AS in SCSs.

Table 9: Publication distribution by sub-classes

Classes Sub-classes

HMI Operator assistance 5 Control Handover 3 S2SI 12 ES Object detection 14 Weather conditions 1 Positional uncertainties 34

BA Adaptive fault detection 15 Adaptive failure mitigation 24

Classes of safety-related challenges throughout the years

The next category of data we decided to present is the distribution of safety-related challenges classes published throughout the years. Thus, we mixed extracted data of publication years with corresponding safety-related challenges classes in order to obtain results of the state of the art in the safety-related challenges while using AS in SCSs across the year range 2010-2018. In Table 10 we observe the achieved results. From the presented table, we can notice that for 2010, we extracted 2 publications and both of them are regarding the ES class of safety-related challenges. Next, from 2011 we as well recognized 2 publications from which one concerns the HMI class while other concerns the S2SI class of safety-related challenges. From 2012, we iden-tified a total number of 5 publications, where 3 publications studied the BA class of safety-related challenges while 2 publications studied safety-related challenges in the ES class. Moreover, we recognized 2 publications from the year 2013, both focused on safety-related challenges in the ES class. In 2014, we can notice an increase in the number of publications, with a total number of 10. From this total number, we recognized 4 that are concerning the BA class, 4 were investigating the ES class, while 2 publications are regarding the S2SI class of safety-related challenges. The next year throughout we mapped classes of safety-related challenges is 2015, where we identified 10 publications. The highest number of those publications concerns the ES class of safety-related challenges. To be more precise 6 of them, then 3 were covering the S2SI class and 1 was about the BA class of safety-related challenges. From the year 2016, we identified 17 publications, where 11 of them concern the ES class, 5 were related to the BA class and 1 publication is regarding the S2SI class

(34)

of safety-related challenges. The next considered year is 2017, which resulted in 21 publications. The publications regarding safety-related challenges throughout this year are classified as 4 belonging to the HMI class, 1 to the S2SI class, 10 to the ES and 6 to the BA class. The last mapped year in our studied range is 2018. This year resulted in 39 of publications regarding identified safety-related classes. In this year we distinguished 3 publications as from the HMI class, 4 from the S2SI class, 12 from the ES class and 20 from the BA class of safety-related challenges while using AS in SCSs.

Table 10: Classes of safety-related challenges throughout the years

2010 2011 2012 2013 2014 2015 2016 2017 2018 HMI 0 1 0 0 0 0 0 4 3 S2SI 0 1 0 0 2 3 1 1 4 ES 2 0 2 2 4 6 11 10 12 BA 0 0 3 0 4 1 5 6 20 Total 2 2 5 2 10 10 17 21 39

Classes of safety-related challenges throughout domains

The next category in which we are mapping extracted data joins identified domains and classes of safety-related challenges. In this category, we are identifying which classes are included over which domains. The category with mapped data is pre-sented in Table 11. From the table, we observe that the first domain upon we mapped classes is Air, e.g., includes unmanned aerial vehicles. In this domain, we recognized 1 publication regarding the S2SI class of safety-related challenges, 16 including the ES class and 11 of the BA class. The total number publications cov-ering safety-related challenges across the Air domain is 28. Next, in the Maritime domain, e.g., unmanned vessels, we identified 3 publications distributed in 2 for the ES class and 1 for the BA class. From the table, we also can see that the high-est number of publications recognized as relevant are from the Automotive domain, e.g., autonomous vehicles, the total number of 58 publications. Amongst which 8 publications focused on the HMI class, 11 publications study on the S2SI class, 25 of the total number were addressing challenges from the ES class, while finally 14 of them belong to the BA class of safety-related challenges. Furthermore, we identified the Medical domain, e.g., surgical robots, in which 2 publications considered the ES class of safety-related challenges while using AS in SCSs. Alongside, we recognized 7 publications in the domain of Robotics. More precisely, 4 publications are covering the ES class in contrast to 3 publications covering the BA class of safety-related

(35)

challenges. Lastly, we recognized publications that can be allocated to all domains, i.e., publications covering safety-related challenges relevant for all domains. This kind of data was labeled as Generic, and 10 publications covering the BA class of safety-related challenges were recognized as belonging to this group.

Table 11: Classes of safety-related challenges throughout domains

Air Maritime Automotive Medical Robotics Generic

HMI 0 0 8 0 0 0

S2SI 1 0 11 0 0 0

ES 16 2 25 2 4 0

BA 11 1 14 0 3 10

Total 28 3 58 2 7 10

Distribution of autonomy levels groups throughout the years

Furthermore, we created a mapping category combining extracted data for autonomy levels and identified years. Thus, in Table 12 we are presenting autonomy levels on which the publications identified as relevant were focused across the studied year range. The autonomy levels we elaborated in Section 2.2.1, while in Section 3.4 we grouped them.

As it can be seen from the Table 12, the total number of publications concerning safety-related challenges in using the semi-autonomous levels of AS in SCSs, is 14. Moreover, from the year 2011, we identified one study. There was no study relevant publications until 2015 and 2016, where from each year we recognized one publication as relevant and focusing on the semi-autonomous levels of AS in SCSs. Further, from the year 2017, we identified 5 publications in contrast to 6 from year 2018. Considering the next category, 46 publications studied full autonomous systems. We recognized 2 publications from 2010, 2012 and 2013, then from 2014 we found 5 publications, 4 from 2015, 8 from 2016 and 2017 each, while from 2018 we found 15 publications. Besides, some safety-related challenges while using AS in SCSs are relevant for systems with different autonomy levels. Those kind of publications we labeled as Generic for this observed attribute. Of this type, we recognized 48 publications. From 2011 we found 1 publication, then from year after we recognized 3, 5 from the years 2014 and 2015 each, 8 publications both in 2016 and 2017 and from the last observed year 2018, we recognized 18 publications studying safety-related challenges distributed across all autonomy levels of AS in SCSs.

(36)

Table 12: Distribution of autonomy levels groups throughout the years

2010 2011 2012 2013 2014 2015 2016 2017 2018 Total

Semi-autonomous 0 1 0 0 0 1 1 5 6 14

Full autonomous 2 0 2 2 5 4 8 8 15 46

Generic 0 1 3 0 5 5 8 8 18 48

Distribution of autonomy levels groups throughout domains

The next category according to which we decided to map publications is a distri-bution of autonomy levels throughout domains. We found it interesting to see how autonomy levels are distributed across previously identified domains. Thus, we are presenting the mapped data in Table 13. From the table, we notice that under the Air domain, we recognized 2 publications relating safety-related challenges while using Semi-autonomous systems in SCSs, 22 of publications concerning the Full au-tonomous level and 4 concerning both, i.e., the Generic type. Across the Maritime domain, we recognized 3 publications covering the Full autonomous level. In the Automotive domain, we managed to distinguish 12 publications covering the Semi-autonomous level, 10 for Full Semi-autonomous systems and 36 publications concerning all autonomy levels. In the Medical domain, we recognized 2 publications considering systems with Full autonomous operability. From the domain of Robotics, we identi-fied 6 publications studying the Full autonomous type of systems, and 1 publication considering all autonomy levels. The last mapped attribute in this mapping category is Generic, i.e., it concerns safety-related challenges that can be faced throughout all identified domains. For this type, we identified 3 publications concerning Full autonomous systems, while on the other side we identified 7 publications concerning all autonomy levels, labeled as Generic.

Table 13: Distribution of autonomy levels groups throughout domains

Air Maritime Automotive Medical Robotics Generic

Semi-autonomous 2 0 12 0 0 0

Full autonomous 22 3 10 2 6 3

Generic 4 0 36 0 1 7

Safety measure by a degree of adaptiveness throughout the years

In Table 14 we are mapping data concerning safety-related challenges while using AS in SCSs and their corresponding safety measure adaptiveness degree. As outlined in previous sections, the degree of adaptiveness we are focusing in this study are