A Comparative Analysis of Argumentation Languages in the Context of Safety Case Development

M

U

S

I

,

D

E

V

,

S

Thesis for the Degree of Master of Science in Computer Science with

Specialization in Software Engineering 15.0 hp

A Comparative Analysis of Argumentation

Languages in the Context of Safety Case

Development

Sirishabai Govardhanrao

sgo18001@student.mdh.se

Examiner: Barbara Gallina

Mälardalen University, Västerås, Sweden

Supervisor: Faiz UL Muram

Mälardalen University, Västerås, Sweden

Supervisor: Jan Gustafsson

Abstract

The safety case creation has become an explicit requirement in most of the safety critical domains to ensure the safety of a system or an application. In the process of developing a safety case the foremost requirement is choosing an efficient argumentation language which fulfills all the functionalities needed to develop a safety case.

In general, there are text-based argumentation notations and graphics-based argumentation notations to represent a safety case. In this paper we are comparing and analyzing the graphics-based argumentation notations like Goal Structuring Notation (GSN), Claims Arguments and Evidence (CAE), Structured Assurance Case Metamodel (SACM, the standardized modelling language to describe the safety case), NOR-STA Services (software platform which support graphics-based notations), Resolute (which is both language and tool that supports graphics-based notations) and Dynamic Safety Cases (special type of safety case which supports graphics-based notations such as GSN).

In this thesis we compared the argumentation notations with respect to different aspects in the context of safety case development. We present our findings like the types of stakeholders gaining benefits from different types of notations, the list of domains where these types of notations are applicable. We also presented the major advantages and dis-advantages of using different argumentation notations considering certain features like understandability, standardization, consistency, maintenance, traceability and assessment in the context of safety case development.

Table of Contents

1

Introduction ________________________________________________________ 8

2

Background ________________________________________________________ 11

2.1 Structured Assurance Case Metamodel ____________________________________ 12 2.2 Goal Structuring Notation _______________________________________________ 14 2.3 Claims Arguments and Evidence Notation _________________________________ 16 2.4 NOR-STA Argumentation _______________________________________________ 17 2.5 Resolute ______________________________________________________________ 19 2.6 Dynamic Safety Cases __________________________________________________ 21

3

Related work _______________________________________________________ 24

4

Problem Formulation and Resolution __________________________________ 26

4.1 Problem Formulation ___________________________________________________ 26 4.2 Problem Resolution ____________________________________________________ 26

4.2.1 Historical Development of Argumentation Languages (RQ1) ____________________ 27 4.2.2 Analysis of Languages, Tools and Standards (RQ2) ____________________________ 28 4.2.3 Feasible findings from our study (RQ3) ______________________________________ 29

5

Method____________________________________________________________ 30

5.1 Research Measures _____________________________________________________ 30

5.1.1 Search Strategy __________________________________________________________ 31 5.1.2 Inclusion and Exclusion Criteria ____________________________________________ 31

5.2 Research Methods _____________________________________________________ 32

5.2.1 Manual Search and Establishing the QGS ____________________________________ 32 5.2.2 Web-Based Search _______________________________________________________ 34 5.2.3 Snow Balling Process _____________________________________________________ 35

5.3 Review of Papers_______________________________________________________ 35

6

Comparative Analysis _______________________________________________ 37

6.1 Historical Development of Argumentation Languages (1990-2019) _____________ 37 6.2 Comparison of Argumentation notations ___________________________________ 39

6.2.1 Modelling of Safety Case __________________________________________________ 39 6.2.2 Tool Support ____________________________________________________________ 41

6.2.3 Notations that are used by different Standards ________________________________ 44 6.2.4 Supporting Understandability ______________________________________________ 48 6.2.5 Consistency between System design and Safety case Development ________________ 50 6.2.6 Support for Automatic Updates in Maintenance of Safety Case __________________ 51 6.2.7 Beneficiary Stakeholders __________________________________________________ 52 6.2.8 Applicability of different argumentation notations in different domains ___________ 53 6.2.9 Advantages and Dis-advantages of Argumentation Notations ____________________ 55

7

Limitations ________________________________________________________ 62

8

Conclusion _________________________________________________________ 63

8.1 Overall Supporting Features of Argumentation Languages ___________________ 63 8.2 Extracted Comparative Studies __________________________________________ 68 8.3 Overall Conclusion _____________________________________________________ 68 8.4 Future Work __________________________________________________________ 69

\

List of Figures

Figure 1. Essential elements of safety case [1] ... 11

Figure 2. Toulmin model representation [58] ... 12

Figure 3. Example of SACM Representation [24]... 13

Figure 4. Example of GSN Representation ... 16

Figure 5. Example of CAE Representation [20] ... 17

Figure 6. Example of NOR-STA Argument Representation [21] ... 19

Figure 7. Syntax of Claim Function [23] ... 20

Figure 8. Syntax of Computational Function [23] ... 20

Figure 9. Binding Related Functions [23] ... 20

Figure 10. Connection Related Functions [23] ... 21

Figure 11. Example of Resolute Representation [3] ... 21

Figure 12. Lifecycle Overview of Dynamic Safety Case [48] ... 23

Figure 13.Steps in Research ... 30

Figure 14. Illustration of search process ... 36

Figure 15. Standards used in Studies ... 48

Figure 16. Inter-relation between argumentation notations ... 61

Figure 17. SACM features ... 64

Figure 18. GSN features ... 65

Figure 19. CAE features ... 66

Figure 20. NOR-STA features... 67

Figure 21. Resolute features ... 67

List of Tables

Table 1. Elements of GSN ... 14

Table 2. Elements of CAE ... 16

Table 3. Elements of NOR-STA ... 18

Table 4. Dimensions and keywords ... 31

Table 5. Including factors ... 31

Table 6. Excluding Factors ... 32

Table 7. Identified Conferences ... 32

Table 8. Identified Journals ... 33

Table 9. Searched databases ... 34

Table 10. Results of Web-Based Search ... 34

Table 11. Initial Evolution of argumentation notations(1990-2019) ... 38

Table 12. Modelling of Safety Case using different notations ... 41

Table 13. Comparison of tools in different notations... 44

Table 14. Argumentation notations used by different standards ... 47

Table 15. Support of argumentation notations in the context of understandability ... 49

Table 16. List of Beneficiary Stakeholders ... 53

Table 17. Applicability of different argumentation notations in different domains ... 55

Acronyms and Abbreviations

• SACM Structured Assurance Case Metamodel

• GSN Goal Structuring Notation

• CAE Claims Arguments and Evidence

• ASCE

• AGSN Assessable-GSN

• AdvoCATE Assurance Case Automation ToolsEt

• OMG Object Management Group

• AADL Architectural Analysis and Design Language

• OSATE Open Source AADL Tool Environment

• SAM Safety Argument Manager

• ISCaDE Integrated Safety Case Development Environment

• AMASS Architecture-driven, Multi-concern and Seamless Assurance and

Certification of Cyber-Physical Systems

• QGS Quasi Gold Standards

• SMACCM Secure Mathematically Assured Composition of Control Models

1 Introduction

The brief introduction of the thesis work is presented in this chapter, Section 1.1 presents the foremost motivation of this thesis and the context of thesis is explained in Section 1.2, the main contribution of thesis explained in Section 1.3 and the main structure of the thesis explained finally in Section 1.4.

1.1 Motivation

Safety case is defined as “A documented body of evidence that provides a convincing and valid argument that a system is adequately safe for a given application in a given environment” [19]. In the process of implementing a safety case we need to make claims regarding the system and provide evidence to support the claim which shows the mitigation of risks has been carried out.

The Toulmin model [58] proposed the argument and its essential parts. Based on Toulmin model specific types of argumentation notations are derived such as text-based or graphics-based languages. Graphics-based notations such as Goal-Structuring Notation (GSN) [1] and Claims

and Arguments and Evidence (CAE) [20] are popular argumentation notations for safety case

development. The NOR-STA [21] is a software platform which supports graphics-based notations, The Resolute [22] which is both a language and tool that supports graphics-based notations. There is also a standardized modelling language for describing the safety case, the Structured Assurance

Case Metamodel (SACM) [24] which is an abstract syntax for GSN and CAE. The new type of

safety case which is evolving is Dynamic safety cases [9] which also rely on graphics-based notations such as GSN.

The selection of appropriate safety argumentation notation is essential for ensuring the safety of a system or application, so choosing the argumentation notation as per the requirements of chosen domain is very crucial. Not only on the perception of the selected domain but also considering the stakeholders view of usability and understandability of argumentation notation to demonstrate the safety case must be considered for ensuring the system’s safety.

This thesis aims to provide certain assistance as a guide to researchers and stakeholders by doing comparative analysis of certain argumentation languages, this thesis tried at its best to answer the following research questions:

• What are the available argumentation languages that supports in the modelling of safety cases?

• What are the tools available for utility of different types of argumentation languages? • Which argumentation languages support in the context of understandability?

• Which argumentation languages have been using in different application areas (domains)? • What are the argumentation languages that have been used by different standards?

• Which stakeholders benefits more from different argumentation languages?

• Which argumentation language provides consistency between system design and developing safety cases?

1.2 Context

In this thesis, the comparative analysis of argumentation languages has been studied to understand the features of different argumentation languages, to discover different tools to use these argumentation languages, to know how the argumentation notations support in the

context of safety case development, to identify the domains that are using these argumentation

notations, to know the standards which are using these argumentation notations and to find out the list of stakeholders those are gaining benefits from different argumentation notations. Hence, the awareness of different argumentation languages has been provided in this thesis by doing appropriate comparison on different evaluation criteria, which helps user in the selection of certain argumentation language depends on the domain, requirement, accessibility, understandability to demonstrate an efficient safety case.

1.3 Contribution

The main contribution of this thesis is to conduct a comparative study for different argumentation languages in the context of safety case development. To achieve that, we have studied several papers related to argumentation languages and gathered broad information and provided wider picture of discovered argumentation languages and their features with respect to different criteria. The following outcomes are provided from this thesis:

• The assessment of different argumentation languages and their features. • The study of supporting tools for different argumentation languages. • The study of understandability feature with respect to different languages.

• The study of consistency between system design and safety case development with respect to different languages.

• The study of automatic updates in the maintenance of safety case with respect to different languages.

• The discovery of domains with respect to different argumentation languages. • The list of stakeholders who gets benefits from different argumentation languages. • The types of standards which used different argumentation languages.

• The assets and flaws of using different argumentation languages with respect to support of tools, maintenance, understandability, usability, reusability.

1.4 Thesis Structure

The thesis report structured as follows:

Chapter 2 provides the background knowledge needed to understand the features of different

Chapter 3 provides the state of art of the safety critical systems, safety cases, safety case

argumentation, safety case tools through the analysis of related work.

Chapter 4 provides the problem formulation and presentation of some questions which arise

for a safety case practitioner or common user.

Chapter 5 presents the research methods we used in this thesis that comprises defining research

problem which are refined into several research questions finding solution, collecting and evaluating data.

Chapter 6 presents the comparative analysis of argumentation languages that are studied in

this thesis that has been evaluated and presented.

Chapter 7 presents the limitations in this thesis work and possibilities of some inadequacy and

missing some relevant resources.

2 Background

This chapter provides the needed background and concepts to be able to understand the different argumentation languages mentioned in this thesis. For this, Section 2.1 presents the Structured Assurance Case Metamodel, Section 2.2 presents the Goal Structuring Notation, Section 2.3 presents the Claims Arguments and Evidence notation, Section 2.4 presents the NOR-STA Argumentation, Section 2.5 presents the Resolute, Section 2.6 presents the Dynamic Safety Cases. However, what is the necessity of an argumentation language. Since we studied the definition of safety case in Section 1.1 and according to that we understand that to implement safety case [19] we need to:

• Make Explicit set of claims of safety requirements & other objectives • Provide supporting evidence

• Make set of arguments that links claims to evidence

Figure 1. Essential elements of safety case [2]

According to the British philosopher Stephen Toulmin a good realistic argument consists of six elements and he described them as Data, Claim, Warrants, Qualifiers, Rebuttals and Backing and each element according to Toulmin model [58] explained as:

• Claim: The statement which is being argued.

• Data: The evidence or facts which is needed to prove the argument.

• Warrants: The links between claim and data which are implicit logical statements. • Qualifiers: The statement that propose the conditions under which the argument is true. • Rebuttals: The statements which indicates the circumstances when the general arguments are

not true and it can be consider as counter arguments. • Backing: The statements that supports the warrant.

Figure 2. Toulmin model representation [58]

However, the three major functionalities need to be fulfilled by an argumentation language in developing a safety case are making explicit claims, providing supporting evidence and making set of arguments that links claims to evidence. From the past few years many argumentation languages are evolved, and each language is different to the other with some added features and in this thesis, we are considering mainly graphics-based argumentation languages, they are:

2.1 Structured Assurance Case Metamodel

The Object Management Group (OMG) [57] published a standardized modelling language for describing the safety case called Structured Assurance Case Metamodel (SACM) [24], which is a potential application to software assurance and it provides a common framework for the best practices in the safety, security, and reliability domains in the assurance of a software. It unifies broadly used graphical notations for documenting safety cases.

The main purpose of software assurance is to depict that the system is built as per our design. The system fulfills all its functions and to make sure that it is free from defects. The model is very useful throughout the software development lifecycle, which provides assurance based on claims, arguments and evidence. This model and framework provide machine-readable repository for assurance case artifacts, such as claims, arguments, and evidence.

Generally, the SACM provides the abstract representation of safety case and in which the concrete representation can be rendered through GSN or CAE. The representation of SACM is like a direct graph, the nodes are claims, arguments and evidence elements. The arguments in SACM represent the claims and citation of artifacts like evidence or artifact reference and the links between the elements how one or more claims are asserted to another claim, or how artifacts are asserted as

represent counter-arguments, counter-evidence and the Assertedcontext. The example representation of metamodel structure can be seen in Figure 3.

Figure 3. Example of SACM Representation [24]

The semantics of SACM elements can be explained as follows:

• Claim: A statement asserting some characteristic, property or behavior of the software or system.

• Argument: Arguments support claims through reasoning or logic that links evidence to claim.

• Evidence: Evidence is objective, reproducible, repeatable, non-disputable information used to support a claim.

•

or negative

statement.

The claims can be evaluated, demonstrable and is supported by arguments based on objective evidence. The Arguments defines the relationships by directly linking the claim and evidence. Evidence is very essential to make assurance case trustworthy and to substantiate claim.

2.2 Goal Structuring Notation

The Goal Structuring Notation [2] is a widely used argumentation notation to represent the safety case graphically. In which we can represents the elements of safety arguments as elegant graphical elements such as goal, strategy, solution, etc., and the elements are linked using directed lines. The main purpose of goal structure is to show the decomposition of goals into sub-goals until claims supported by evidence.

The elements and their semantics involved in this GSN are shown in Table 1.

Table 1. Elements of GSN

Name of the element

Semantics Representation

Goal This node represents the claim about a property of the system

Strategy The strategy is a description of the reasoning about how and why a goal can be

decomposed into sub-goals

Context It gives the definition of boundary where the goal is given.

Solution It shows the actual evidence that satisfies the goal to which it is connected.

Justification It gives further rationale for a selected GSN entity.

Assumption The statement can be taken as true without further explanation.

Supported by It is an evidential relationship that shows the link between goal and evidence. It shows the connection between goal to goal, goal to solution and goal to strategy.

In-Context of It is used in contextual relationship, it gives linkage between goal to assumption, goal to context, goal to justification, strategy to context, strategy to assumption.

Undeveloped Goal

The undeveloped goal means that the goal needs to be further developed.

Un-instantiated Triangles

The un-instantiated triangles are attached to the nodes that contain un-instantiated parametrized expressions

The GSN argumentation has been using in many industrial sectors because of its easy access and clear understandable structures, we can see an example of how safety case represented through GSN model which was made using Astah GSN [27] tool and it is shown in the Figure 4.

Figure 4. Example of GSN Representation

2.3 Claims Arguments and Evidence Notation

Claims Arguments and Evidence (CAE) is a simple but efficient notation [20] for structuring the safety cases, which contains three main elements. The semantics and representation of elements are shown in Table 2.

Table 2. Elements of CAE

Name of the element Semantics Representation

Claim A statement in the argument which is assessed to be true or false.

Argument It is description of an argument approach that support and satisfy parent claim.

Evidence It is the reference to the evidence presented in support of claim or argument

Sub-claim Each claim is supported by number of sub-claims.

Each claim is supported by number of sub claims, arguments or evidence, the approach in this is straightforward and it is permissible to simply link directly from supporting claims. The example representation of CAE argumentation shown in Figure 5.

Figure 5. Example of CAE Representation [20]

2.4 NOR-STA Argumentation

NOR-STA is a platform of software services which supports development, maintenance and assessment of structured arguments based on evidence argument management methodology TRUST-IT [13]. As per [21] NOR-STA is an efficient and complete automation argumentation

used efficiently in editing complex safety cases and manage user requirements. The basic elements involved in NOR-STA Argumentation are shown in Table 3.

Table 3. Elements of NOR-STA

Name of the element

Semantics Representation

Claim Claim represents a

premise to be further justified by its own premises.

Argumentation Strategy

The argumentation strategy shows how claims are supported by its elements

.

Counter Argumentation strategy

It shows how claims to be refuted based on elements linked to it.

Rationale It is the justification of validity of inference linking the premises & conclusion of argumentation strategy. Assumption It is used to represent some assumed

property and represents context dependent constraints.

Fact It represents a premise which is supported by

some evidence.

Reference It is used to integrate external evidence like webpage, document or any element.

Information It is used to integrate the

argumentation part with external contextual data.

STA is mainly focused on logical structure of the argument, the argument structure in NOR-STA is presented as a tree-like structure with root element on left and supporting elements expanding down and to right. In which each type of element is indicated by icon, as shown in Table 3. We can see the representation of simple NOR-STA argument in Figure 6.

Figure 6. Example of NOR-STA Argument Representation [21]

2.5 Resolute

The resolute [3] is considered as both language and tool, it is framework for automatically generating assurance cases based on system model specified in an architectural design language and resolute allows user to define set of claim functions and connect them with AADL model. Thus, in this argumentation, safety cases depend on the AADL (Architectural Analysis and Design Language) model and the changes in AADL model may reflect the changes in the safety cases. Resolute is an Open Source under BSD License and available in online [22]. The Open Source AADL Tool Environment (OSATE) is used to implement the Resolute. The Resolute annex clauses placed directly in AADL package and Resolute command can be invoked on a component implementation. With the Resolute tool, user defines [23] claim functions, computational functions in Resolute annex libraries. The obtained verification results are then displayed in a view labeled

Assurance Case.

• Claim Functions: The claim functions [23] represent requirements, verification actions, and assumptions. The claim function expression is assumed to be a logical expression (and, or, and then, or else, implies ((=>)) or quantified expressions (for all, exists) to represent a predicate. The users define claim functions in Resolute annex libraries, we can declare resolute annex library directly in package through annex Resolute {** **}; statement. The syntax of claim function shown in Figure 7.

Figure 7. Syntax of Claim Function [23]

The claim functions are invoked on a component by specifying a prove statement in a Resolute annex subclause of the component implementation, after the Resolute command is invoked the claim function executes.

•

these are used to calculate value of any type and the result can be Boolean, numeric, model elements

. The syntax of computational functions shown in

.

Figure 8. Syntax of Computational Function [23]

The computation functions are invoked in claim functions that represents verification actions or assumptions.

• Common Resolute Function Library

Binding Related Functions: The Binding related functions of common resolute function

library are shown in Figure 9.

• Connection Related Functions: The connection related functions of common resolute function library are shown in Figure 10.

Figure 10. Connection Related Functions [23]

We can see the example of a successful safety case which uses the resolute is shown in Figure 11.

The dynamic safety cases (DSCs) are described as [9] “a novel operationalization of the concept of through-life safety assurance, whose goal is to enable proactive safety management”. In the framework for dynamic safety cases the safety management system concept extends to provide continuous, through life cycle updates and management of respective safety substantiation artifacts. In the dynamic safety cases arguments are defined as a labeled tree with structural restrictions and conditions. The argument elements are represented as nodes of tree, labels give the description of nodes, types and metadata. The DSCs for a system made up of:

• Assurance Variable: The system, environment and assurance artifacts as a collection of assurance variables (AVs) constitutes all relevant safety artifacts.

• Argument structure: The metadata relating its nodes to AVs and nodes of a confidence structure constitutes an argument structure.

• Confidence structure: In this node relate to argument fragments and monitor output. • Monitor Types: Monitors perform data analysis, they generally triggered when a particular

event occurs, and in DSCs events are considered as AVs. The collection of monitors examines AVs and return either discrete or continuous output.

• Update Rules: In this the rule, condition C is a formula over confidence structure and AVs. Where Q is a query in the AdvoCATE query language to determine the parts of argument that requires changes.

Dynamic Safety Case Framework: The dynamic safety case framework supports three main principles they are

➢ Provides mixed automation framework

➢ Provides an increased level of formality in the safety infrastructure.

➢ Proactively compute the confidence in and update the reasoning about the safety of ongoing operations.

Lifecycle: The suggested lifecycle in dynamic safety cases comprises four continuous activities

they are

• Identify: The sources of assurance deficits in safety cases can weaken confidence in safety so identifying it.

• Monitor: In this monitoring is done by periodically interrogating the arguments and its links to external data by argument querying.

• Analyze: To update the confidence in the associated claims and to understand the impact on safety reasoning, the analyzation of operational data is done to examine whether threshold defined for assurance deficits are met.

• Respond: The proactive response is enabled by operational events that affect safety assurance which is at the heart of DSCs, depends on a combination of factors the appropriate response is decided.

Figure 12. Lifecycle Overview of Dynamic Safety Case [48]

The framework for dynamic safety cases extends the safety management concept to provide continuous through-life safety updates to the relevant systems and management of corresponding safety substantiation artifacts.

3 Related work

From the past two decades lot of research and studies have been done on safety cases, safety case tools and safety case development from which many papers have been evolved and published by researchers. Gleirscher et al. [34] done the survey of design and argument patterns with a focus on the last two decades. The authors present the state-of-the art of patterns in safety critical system design and assurance argumentation. They presented survey of reusable concepts for specification, design, and assurance of safety critical systems. The research in the paper has been carried out on reusable design and argumentation concepts for the assurance of system safety. Finally, we understood that this paper mainly concentrates on system safety, safety case and patterns. But in this paper, we found very little information regarding tool support.

Maksimov et al. [12] present a systematic review of progress made in the development of tools for assurance cases from the past two decades. In this paper [12] authors discovered broad varieties of tools and provided their information. They also done the evaluation of capabilities of individual tools and their support in the creation, maintenance, assessment, collaboration, reporting and integration with respect to assurance cases. Thus, this paper mainly focused on safety case and its tools. However, this paper [12] did not provide the information of tool support in the understandability, consistency, risk mitigation, relevant evidence demonstration of safety case. This paper [12] does not focus or present information on domain where these safety case tools have been using and which standards have been using these tools.

John [59] presents the interpretation and evaluation of assurance cases. Particularly the author concentrated on the standard DO-178C. In this paper [59] they evaluate the assurance cases and they found two main issues, one is how to assess the impact of overall argument and the other is fallibility of human judgement in performing assessments. The author’s largest concern is the degree of independent review that can be applied to a bespoke assurance case. They advocate hybrid approaches and recommend future revisions to guidelines regarding DO-178C should be constructed as assurance case templates rather than fully bespoke developments.

Doss et al. [33] present the challenges and opportunities in agile development in safety critical systems by conducting survey to draw responses related to incremental and iterative nature of safety requirements development, hazard analysis and safety case development. They target the experienced practitioners from the safety critical systems development and agile development methods. They obtained positive feedback from the integration of agile and safety methods. Thus, this paper mainly focuses on agile development, safety critical systems, hazard analysis, safety requirements and safety cases.

The related work shown that many papers have proposed different argumentation approaches and their supporting tools. But from the best of our knowledge and research, we consider: no paper has been came into existence that presented collection of different argumentation notations (graphics-based notations) and supporting tools. None of the published paper presented the benefits of using different argumentation notations with respect to safety case development. Therefore, in this thesis we have done broad research and presented description of different argumentation notations and their advantages and disadvantages with respect to safety case development. We also discovered that there is no paper yet presented the safety cases with respect to different domains, standards

analysis of argumentation languages with respect to different criteria in the context of safety case development.

4 Problem Formulation and Resolution

4.1 Problem Formulation

As we discussed in Section 1.1 the selection of appropriate safety argumentation notation is essential for ensuring safety of a system or application. In general, for any stakeholder, researcher or a new user to demonstrate a safety case, the person must be aware of argumentation language. Since, there are variety of argumentation notations in practice. Therefore, selecting one among the available argumentation notations is the challenging task for the person because from our research we understand that there is a lack of proper guide which presents the evaluation of different argumentation languages. The paper [32] presented some preliminary ideas to people about text-based notations for representing safety cases. But no paper has yet presented the idea about

different graphics-based notations in the single publication for ease of verification, hence our

thesis aims to present the detailed concepts of some graphical-based notations and addition to that we made their comparative analysis in the context of safety case development. We provided the merits and demerits of using various argumentation language in the context of safety case development. This thesis may provide support to academia or industrial sectors, such as for safety case practitioner or a researcher.

From our study we came to know that there is also inadequacy of information regarding the

domains where these argumentation languages are used and specification of standards, which

supports those argumentation languages. Therefore, we tried our best to explore and present the information of different domains where the mentioned argumentation languages have been adopted and list of standards which used those argumentation languages. This information may help people in choosing specific argumentation language when working with certain domain or standard. Finally, we trace out the types of stakeholders who gets more benefits with respect to different argumentation languages, this information provides the stakeholders some idea in selecting specific argumentation language which is more suitable to their purpose.

4.2 Problem Resolution

The main objectives of our thesis are to present a clear and vast information of distinct

graphics-based argumentation languages. The assets and flaws of using those languages with respect to

certain aspects like understandability, reusability standardization, consistency, maintenance, traceability and assessment. The discovery of tools to utilize those languages. The other objective is presenting the comparison of those languages by adopting some comparison criteria in the

context of safety case development. The demonstration of distinct domains where those

languages can be used, and the list of standards which used those languages. The classification and presentation of stakeholders who gains benefits from those languages.

We formulated our thesis into three main goals as following research questions and provided answers to these questions in further work.

RQ1: How did research in graphics-based argumentation language develop over time?

RQ2.1: What are the available argumentation languages that support in the modelling of safety

cases?

RQ2.2: What are the tools available for utility of different types of argumentation languages? RQ2.3: Which argumentation languages have been used by different standards?

RQ2.4: Which argumentation language supports in the context of understandability?

RQ2.5: Which argumentation language provides consistency between system design and

developing safety cases?

RQ2.6: Which argumentation language provides automatic updates in the maintenance of

safety cases?

RQ3: What is the potential practical impact of the studies?

The RQ3 refined into following research questions:

RQ3.1: Which stakeholders are more beneficial with respect to different argumentation

languages?

RQ3.2: Which argumentation languages have been using in different application areas

(domains)?

RQ3.3: What are the advantages and dis-advantages of using various argumentation

languages with respect to different aspects like consistency, standardization, maintenance, traceability and assessment in the context of safety case development?

We elaborated, analyzed and presented each research question into following sub-sections:

4.2.1 Historical Development of Argumentation Languages (RQ1)

From the RQ1, this thesis aims to study distinct research papers related to graphics-based argumentation languages published during the period 1990–2019. In this process we identified different venues where the papers related to graphics-based argumentation languages have been published. We studied how the development of graphics-based argumentation languages has been carried out during the past two decades and how new languages evolved with some additional features.

In the process of our study, we identified different types of graphics-based argumentation notations they are a standardized modelling language called SACM which is an abstract syntax for the graphical notations GSN and CAE, the GSN and CAE are the popular graphics-based notations,

The NOR-STA which is a software platform that supports graphics-based notations, Resolute which is a language and tool that provides graphics-based notations, one new unique safety case called Dynamic Safety case which supports graphics-based notations like GSN. After

identifying the various types of argumentation languages, we further extracted and studied different papers published about each type of notations individually. To study the individual languages and their features, to know the process of their development in the context of safety case development. Although all the notations are graphics-based notations, the representation, the elements and the features are all different in each type of notations. The knowledge of the study made in graphics-based argumentation languages provided us a base in making comparative analysis of argumentation languages in the context of safety case development.

Thus, the RQ1 provides the knowledge of the types of existed graphics-based notations and their historical evolution which has been carried out during the past few years that is between 1990 – 2019 (present).

4.2.2

Analysis of Languages, Tools and Standards (RQ2

From the RQ2, this thesis aims to study and analyze the various graphics-based argumentation languages and their support in modelling safety cases, and some of their supporting features in the context of understandability, consistency and maintenance. We analyze various tools to implement the identified languages. This thesis also aims to analyze the standards that have been used different argumentation notations in each study. We split the RQ2 into following sub-questions for the ease of understanding, lets briefly study them:

• Available languages in the modelling of safety cases (RQ2.1): In general, there are many types of argumentation languages in practice to document the safety cases. In our thesis we aim to trace out and present the available graphics-based argumentation languages. The modelling of safety case using graphics-based notations can be accustomed with the supporting tools. However, some type of argumentation languages no need of additional tool support, they itself act as tool. Hence, we aim to present the list of available graphics-based argumentation languages and their way of support in modelling safety case.

• List of tools for the utility of distinct argumentation languages (RQ2.2): In the modelling of safety cases we use argumentation languages. To implement argumentation languages, we need different tools or techniques. Therefore, here we aim to trace out the tools that are available to utilize the argumentation languages. There are some tools which may support more than one argumentation language, we therefore try to analyze features of each discovered tool from the study of published research papers.

• Supporting feature in terms of understandability (RQ2.3): Whatever may be the language, the main thing for the user or researcher is the necessity of understandability, to access the safety case easily. Therefore, in our thesis we aim to trace out the understandability feature of argumentation languages.

• Standards used in the study (RQ2.4): As we know that there are so many standards like ISO 26262 for automotive, EN 50128 for railway, etc. From our study of different argumentation languages, we understand that there are certain standards that have been using the graphics-based argumentation languages. Hence, we figure out those standards and presented it in Chapter 6. This study helpful when working with certain standard in finding supporting argumentation language.

• Study of consistency feature (RQ2.5): Consistency between system design and the development of safety case makes the safety case efficient. Hence, it is one of the features need to be check with respect to different languages. Therefore, we aim to study the consistency feature of identified argumentation languages.

• Providing automatic updates in maintenance (RQ2.6): The maintenance of safety case is an important feature in the process of developing safety case. The facility of providing automatic updates is very beneficial to the user in the maintenance of safety case. Hence, in our thesis we aim to study different argumentation languages and their support in providing automatic updates in maintenance.

4.2.3 Feasible findings from our study (RQ3)

From the study of tremendous published papers related to argumentation languages, we found efficient information of merits and demerits of using different argumentation languages with respect to different aspects, we discovered domains where the studied argumentation languages have been using, and the types of stakeholders who gaining benefits from different argumentation languages. Therefore, we split our research question into following sub-questions to present detailed information.

• Beneficiary Stakeholders (RQ3.1): The study of different argumentation languages gave us information about the types of stakeholders who gaining benefits from different argumentation languages. Hence, we presented the detailed list of beneficiary stakeholders from our findings in the Chapter 6.

• Domain specification (RQ3.2): In general, there are broad varieties of domains. From our study of different argumentation languages and their usability in different domains we discovered and presented the list of distinct domains where these argumentation languages have been using and their detailed description is given in Chapter 6.

• Advantages and Dis-advantages (RQ3.3): From the study of the selected argumentation languages we discovered some major advantages at the same time, some drawbacks in using different argumentation languages with respect to different aspects like consistency, standardization, maintenance, traceability and assessment in the context of safety case development. Hence, we presented the merits and de-merits of using different notations with complete details in the Chapter 6.

5 Method

In this chapter we present the methods that have been applied in this overall thesis work, to extract the relevant information needed to solve the research problem mentioned in Section 4.1. To find out the required answers for the research questions mentioned in Section 4.2. Primarily, in this Section we are going to explain the method of research done in this thesis step by step. The detailed description of each research method and their adoption in this thesis explained in the Section 5.2 and the complete review of selected papers shown in Section 5.3.

5.1 Research Measures

As the foremost objective of this thesis is to present the detailed information of the graphics-based argumentation notations. Hence, to identify the list of different argumentation notations that are in practice in different sectors can be achieved by searching different sources. To do searching we adopted certain research methodologies and by implementing certain keywords, we collected data needed to solve the research problem. Then by applying inclusion and exclusion criteria we identified some final useful papers for our thesis study and used them throughout the thesis evolution, steps involve in our complete research is shown in Figure 13.

Figure 13.Steps in Research

The foremost steps involved in the research process of our thesis is shown in the Figure 13. Now we go into deep of our research study by identifying keywords to use in the search strategy are explained in the Section 5.1.1.

5.1.1 Search Strategy

In our research, search strategy aims to trace out data from the sources which are relevant to our research problem and research questions. Hence, we designed our search strategy to discover and extract every relevant publication. In our electronic search we searched some prominent databases which are mostly used in software engineering and computer science as mentioned in [35]. They are the IEEE Explore Digital Library, ACM Digital Library, SpringerLink, Google Scholar, ScienceDirect. These databases also provide the efficient information of bibliography. The search terms we used in our research are split into two major dimension and are shown in Table 4.

Table 4. Dimensions and keywords

Dimensions

Types of Argumentation Languages

safety case argumentation languages, safety case argumentation notations, documentation of safety cases, list of argumentation languages

Tools to implement argumentation notations

safety case tools,

assurance case tools, Safety case editors, automatic safety cases

In order to gain the data from the source we consider different alternative words (e.g., argumentation languages/notations, argumentation/safety case tools), synonyms (e.g., languages, notations). We also used Boolean operator “OR” to join the alternate words and synonyms, and the Boolean operator “AND” to make the search string adequate to extract the data.

5.1.2 Inclusion and Exclusion Criteria

We searched the papers that have been published between the period 1990-2019, which are related to safety critical systems, and among that we identified the papers which are more relevant to our research by applying the inclusion and exclusion criteria, the factors we followed in inclusion and exclusion criteria is shown in Table 5.

Table 5. Including factors Type Factors

Include i. The study reports that possess related information about argumentation languages or safety case documentation. ii. The study reports that derived from acceptable sources as

per our search method such as from peer-reviewed scientific journals, conference or workshops.

iii. The reports that are related to safety case development process.

iv. The reports that presents the tools for the utility of argumentation languages.

Table 6. Excluding Factors Type Factors

Exclude i. The study does not contain related information of argumentation languages or safety cases

ii. The study doesn't relate to peer-review process such as journals or conference paper.

iii. The study reports which are not clear and understandable. iv. The study that contains information of non-graphic based

argumentation notations.

Thus, in our research primarily by using search strings mentioned in Table 4 and implementing the research methodologies mentioned in Section 5.2, we gone through tremendous scientific journals, conference and workshop papers and then applied inclusion/exclusion criteria to choose some papers which were more related to our thesis.

5.2 Research Methods

The research methods we adopted in this thesis to search data are of mainly three types they are

Manual Search and Establishing the QGS, Web-Based Search and in the final search process,

we perform Snow Balling Process. Each search method elaborated and explained in detail in the Sections 5.2.1, 5.2.2 and 5.2.3, respectively.

5.2.1 Manual Search and Establishing the QGS

In this approach to find as many primary studies as possible that are relevant to research questions, we establish quasi-gold standard (QGS). It is a set of known studies from relevant publication venues on a research topic [35]. In the manual search we searched some popular venues which consists of different proceedings of conferences specialized in safety critical systems and some efficient journals where the community often publish the research. The results of the manual search used to establish QGS. After gaining the knowledge of domain specific venues we check the published papers by checking the title-abstract-keywords. Then we get to know the paper and by applying the inclusion and exclusion criteria we decided the paper to include or exclude in the thesis studies.

The identified conferences, workshops and journals from which we used our thesis mostly obtained the papers are shown in Table 7 and Table 8.

Table 7. Identified Conferences

No Name of the Conference/workshops

1. International Conference on Computer Safety, Reliability, and Security - 1999

2. Workshop on Assurance Cases: Best Practices, Possible Obstacles and Future opportunities

3. Conference on High Integrity Language Technology

4. International Conference on Computer Safety, Reliability, and Security- 2015

5. Engineering and Technology International Conference on System Safety 6. IEEE/IFIP International Conference on Dependable Systems and

Networks

7. IEEE International Conference on Software Engineering

8. International Conference on Model-Driven Engineering and Software Development

9. ERCIM-DECOS Workshop

10. International Conference on Quality of Information and Communications

11. 10th International Conference on Intelligent Computer Communication and Processing (ICCP)

12. IEEE 7th International Workshop on Requirements Engineering and Law

13. Aerospace conference

Table 8. Identified Journals

No Name of the Journal

1. Automated Software Engineering 2. _{Systems Architecture}

3. _{Journal of Research of the National Institute of Standards and Technology} 4. _{Safety and Reliability}

5. _{Computer Standards and Interfaces} 6. _{Computing Research Repository}

We identified the workshops, conference and journals details for our thesis then we started to explore the databases where these venues are available mostly, as from our research, we found the following databases which are valid as per QGS to search through due to their popularity in Safety Critical Systems research. The list of databases where we found the papers most are shown in the Table 9.

Table 9. Searched databases

No Name of the database

1. IEEE Xplore Digital Library 2. ACM Digital Library 3. ScienceDirect 4. Springer Link 5. Google Scholar

5.2.2

We performed web-based search [36] to gain some additional data for our thesis work. We used some search strings containing the words like “safety arguments”, “argumentation notations”, “SACM”, “Goal Structuring Notation”, “Claims Arguments Evidence”, “NOR-STA notations”, “Resolute argumentation”, “safety case” AND “tools” in the search engine Google to discover some more information. Then we identified and presented in Table 10, some of the safety argumentation related citations that helps in discovering information about some argumentation notations and about some safety case tools.

Table 10. Results of Web-Based Search

No Gained information about Citation

1. SACM https://www.omg.org/spec/SACM/About-SACM/ 2. CAE https://www.adelard.com/asce/choosing-asce/cae.html 3. NOR-STA www.argevide.com 4. Resolute https://github.com/smaccm/smaccm

5. Resolute User Guide https://usermanual.wiki/Pdf/userguide.717489719/help 6. OpenCert https://www.polarsys.org/projects/polarsys.opencert 7. AMASS https://www.amass-ecsel.eu/public-tags/opencert

8. Astah http://astah.net/

9. D-Case Editor http://www.jst.go.jp/crest/crest-os/tech/D-CaseEditor/index-e.html

10. CertWare https://nasa.github.io/CertWare/

11. ISCaDE http://www.iscade.co.uk/

Thus, from the web-based search we gain in depth information about some argumentation notations and gain more information related to different safety case tools used in argumentation to

5.2.3 Snow Balling Process

We perform Snow Balling Process [37] that is manually scanning and analyzing the references and citations of the selected papers to ensure that our study also covered relevant follow-up works that might exist. That is after finding a paper we read the abstract first and other parts and definitive decision will be taken to include or exclude the paper.

In this process we identified and included some papers:

• A paper about new type of safety case that is Dynamic safety case mentioned in Section 2.6. • Some papers which explains about distinct argumentation languages and their supports in the

context of safety case development.

• Some survey papers in the context of safety case development. • Some types of papers that presents data of distinct safety case tools. In this approach we excluded some papers due to the following reasons: • The papers which are related to non-graphical notations.

• The papers which are not associated with any of our research question.

• The paper which presents different safety case patterns which is not needed for this thesis. • The papers which is not a peer-reviewed such as journals, workshop or conference paper.

We have done the snow-balling process continuously until no new papers are found then we finished the snow-balling process. The snow-balling process additionally added 9 papers to our research. Thus, our research methods Manual research establishing QGS, Web-Based research and Snow-balling process which have been used in this thesis are explained.

5.3 Review of Papers

In our thesis, we started our searching process using the research methods mentioned in Section 5.2, from January (2019) to March (2019). We searched the papers published between the period 1990-2019. We illustrate the search process and the number of primary studies identified at each stage in Figure 14. The initial search of review process started with our primary studies of different papers collected from manual and web-based search in which first we found 64 studies through different data sources and websites. In which we first identified relevant studies by scanning the title, keywords and abstract. we filtered the studies by applying the inclusion/exclusion criteria, and removed the studies which are not relevant, for instance, the paper does not belong to peer-review process like journals, workshops or conference papers and which are duplicates. From that we obtained 11 studies. Then we started applying snow-balling process and inclusion-exclusion criteria then we obtained 9 studies. Finally, from thorough analysis we finalized 20 studies which are most relevant to answer our research questions.

Figure 14. Illustration of search process

6 Comparative Analysis

The main contribution of our thesis is presented in this chapter. All the extracted 20 studies which were mentioned in Section 5.3 were analyzed in-depth and relevant data were extracted from these studies. In the data extraction step, we used excel sheets to record and correlate the extracted information. Here we have done the comparative analysis of graphics-based argumentation languages which can be obtained by answering all our research questions mentioned in Section 4.2.

6.1 Historical Development of Argumentation Languages (1990-2019)

The safety case can be documented in textual and graphical notations. In this thesis we concentrated on graphical notations and have done study of different research papers which provide data regarding graphical notations. From our study we found the existence of numerous research papers, which present the details of individual graphical notations and their features. But we have not found any research paper which shows a collection of graphical notations in a single research paper. However, for the ease of understanding we segregated our study with respect to individual graphical notations according to the year of their initial development. Hence, here we are comparing and presenting each type of graphical notation and their evolution found in our thesis studies published between the period 1990 – 2019 which gives the answer to the research question

RQ1 which is mentioned in section 4.2.

The Goal Structuring Notation is one of the prominent types of graphical notation [1] developed

at the University of York during 1990 initially by Tim Kelly and his colleagues, it has evolved over time and gained popularity in the year 2012. It has been standardized for graphical documentation of safety cases in the year 2014. The latest version of GSN standard [42] is version

2 which is released in January 2018. The GSN standard is maintained by assurance case working

group which is organized by safety critical systems club and consists of GSN practitioners.

The Claims Arguments Evidence (CAE) is an effective and simple graphical notation developed

by Adelard LLP [43] which is associated with City University in London, UK. The early work on structured assurance cases started by Adelard in 1998. Particularly the formulation of safety case in terms of claims arguments and evidence [20] was developed there and, over the years, this has been formalized and systematized, it is referred as CAE (the name for Adelard’s specific methodology and notation is Adelard Safety Case Development, or ASCAD [41]. ASCE software developed by Adelard simplifies the creation of safety and assurance case, CAE [20] is one of the prominent notations available in ASCE. In December 2017 the latest version [49] of ASCE 4.2 Feature Pack 3 is released by Adelard.

The SACM is developed by OMG [57]. The SACM is an OMG standard that provides a common framework for assurance case development and information exchange. The Argumentation metamodel and Evidence Metamodel combined to form SACM. The initial work on SACM started in 2008 and the SACM specification version 1.0 was published in January 2013. The latest version 2.1 beta [24] published in March 2019. SACM has been using in United Kingdom and United States for developing safe and secure systems [14].

The NOR-STA [21] is not actually a graphics-oriented tool but it focuses on logical structure of argument. We can also generate diagrams in reports using NOR-STA. The NOR-STA research and development project [44] started in January 2010 and it lasts till March 2014, and their actual

services started from March 2014. The development of NOR-STA software, the sharing and support for users is provided by Argevide [21]. The Argevide released different version of NOR-STA in the years 2015 - 2017 and the very latest version was released in April 2018. This latest version with some changes providing some additional features.

The Resolute language developed by Rockwell Collins in the SMACCM research project [46] initially their work on developing assurance systems updated in 2012 as per the SMACCM project website (year obtained from SMACCM website [46]). In SMACCM project they developed high assurance systems along with Rockwell Collins and their partners. The Resolute can be implemented by OSATE, which is available in online [22]. The latest version of OSATE is Version 2.5.1 released in July 2019.

The new type of safety case called Dynamic safety cases described by Ewen Denny and Ganesh Pai and they published a paper on dynamic safety cases in May,2015. They introduced it to enable proactive safety management. Based on the paper published by the authors on dynamic safety cases, Ibrahim Habli, University of York also presented a paper on Dynamic Reasoning for Safety Assurance [48] which we found from Google search. An Open seminar also given by Ibrahim Habli, University of York at MDH, Sweden [47] on Dynamic Reasoning for Safety Assurance in 2015. The period of evolutionary study of graphics-based languages over time can be seen in Table 11.

Table 11. Initial Evolution of argumentation notations(1990-2019) Name of the Argumentation notation Initial evolution of argumentation notation started in the year Latest update related to the argumentation notation

GSN 1990 GSN standard latest version

2 released in January 2018.

CAE 1998 ASCE tool which is used to

implement CAE released its latest version 4.2 Feature Pack 3 in 2017.

SACM 2008 (Initial works published in 2010 approved version 1.1 released in 2015)

SACM specification latest version 2.1 beta published in March 2019.

NOR-STA 2010 NOR-STA latest version

released in April 2018.

Resolute 2012 OSATE which is used to

released its latest version 2.5.1 in July 2019.

Thus, from the evolutionary study of different argumentation notations we have found that the initial studies of graphical notations have started in the year 1990, and different languages evolved in different period. But still till today, all the mentioned graphical notations are in use in different sectors. However, the evolution and development of argumentation notations carried out between the period 1990 – 2019 (present).

6.2 Comparison of Argumentation notations

In our research, we studied numerous publications related to safety critical systems. We found large number of papers, which present the information of different argumentation notations and their supporting features, we extracted all the information and presented it in a clear and understandable way.

From our research in evolutionary study of graphics-based argumentation languages, in the Section 6.1, we found different types of prominent notations. Where the most popular languages are GSN and CAE, where the NOR-STA and Resolute are considered as both language and tool. We found one standardized modelling language called SACM which is an abstract syntax for GSN and CAE and we also found special type of safety case called Dynamic safety cases. The basic information of these languages is presented in the Chapter 2. Here in this Section we are going to compare the mentioned argumentation notations under the characteristics of Modelling of safety case, tools to

implement the notations, notations used by different standards, understandability of safety case, consistency between system design and developing safety case, automatic updates in maintenance of safety case, beneficial stakeholders, applicability to domains, advantages and dis-advantages of using the notations with respect to different aspects. This comparison will give

answers to the research questions RQ2 and RQ3 mentioned in Section 4.2.

6.2.1 Modelling of Safety Case

i. SACM: The SACM is an abstract syntax for GSN and CAE. The SACM [24] is an

independent metamodel which creates safety case using its own argumentation component. It is used to model structured arguments, also to model evidence. The SACM model also supports in creating safety case with controlled vocabulary. The SACM can document the safety case with the support of certain types of tools. The SACM supports some additional features like re-use of assurance artifacts, automated pattern, instantiations etc., From the past few years many automated tools have been developed and came into existence for assisting people. These tools support in the creation of safety case, the tools that we discovered from our research study, which supports in the modelling of SACM safety case are OpenCert, AdvoCATE, and Astah GSN.

ii. GSN: The safety case can be modelled and documented in the graphical format using the

GSN. The GSN [2] is a simple notation to structure the arguments but has broader applications. There is very much requirement of safety case in certification or regulations of various industries. From our study of various publications, we found that the safety case can be modelled using GSN with the support of certain types of tools. There are some tools