Test Process Assessment of Industrial Control Systems via Safety Standards

Course Code: DVA503

Master Thesis in Intelligent Embedded Systems

TEST PROCESS ASSESSMENT OF

INDUSTRIAL CONTROL SYSTEMS VIA

SAFETY STANDARDS

Ladan Pourvatan

lpn19016@student.mdh.se

Examiner: Thomas Nolte

Mälardalen University, Västerås, Sweden

Supervisors: Wasif Afzal, Eduard Paul Enoiu

Mälardalen University, Västerås, Sweden

Abstract

Context: As more systems are becoming embedded hardware-based, challenges regarding soft-ware safety and considerable consequences of their failure arise. Various safety standards as-sure certain safety aspects of systems, addressing areas including testing. The safety standards chosen for this thesis are ISO/IEC/IEEE 29119-2 & 3, IEC 61508-1 & 3, ISO 13849-1 & 2, and ISO/IEC/IEEE 12207:2017. Objective: This thesis tackles the problem of compliance with safety standards by utilising a lightweight assessment method, leading to recommendations for improving the test process of an industrial control system. Method: A case study is performed on an auto-mation company to achieve the objectives of this thesis. The method used for the qualitative data analysis results in recommendations regarding the compliance of the company’s test process with selected safety standards. As the final step, the execution of a focus group research leads to the industrial evaluation of the recommendations and assessment results. Results: The company’s development process fully complies with 22% and fails to comply with 58% of the extracted require-ments from the selected safety standards. Furthermore, the thesis results in recommendations for improving the test process of an industrial control system. As a result of performing the case study, a method for a lightweight assessment of the development process of industrial control systems is achieved. The generic method follows five steps, firstly tabulating the data to attain assessment criteria and items, used by the assessment step to get a compliance degree per requirement. The analysis step comes next to shed light on areas of strength and weakness, leading to recommenda-tions. The final step evaluates and refines the recommendations according to the results of a focus group. Conclusion: Further development of the method used in this thesis can lead to a generic method for assessing development processes, concerning safety standards, using limited resources. The results of this generic method can lead to recommendations for test process improvements of control systems via safety standards.

Acknowledgements

I would like to thank my supervisors Eduard Paul Enoiu, Wasif Afzal, and my industry supervisor for their advice throughout the process of writing this thesis. They never failed to answer my endless questions and kept me going despite the challenges. This thesis would have been impossible without the help of all three supervisors, and I am greatly appreciative of them all for their time and contribution.

Contents

1 Introduction 1 2 Background 2 2.1 V-Model Process . . . 2 2.2 Safety Standards . . . 3 2.2.1 IEC 61508 Standard . . . 4 2.2.2 ISO/IEC/IEEE 12207 Standard . . . 5 2.2.3 ISO/IEC 29119 Standard . . . 6 2.2.4 ISO 13849 Standard . . . 7 2.2.5 ISO/IEC 33063 Standard . . . 8 3 Related Works 8 4 Problem formulation 10 4.1 Scope Definition . . . 10 4.2 Limitations . . . 11 5 Method 11 6 Ethical and Societal Considerations 14 7 Case Study Design 15 7.1 Data Collection . . . 16

7.1.1 Document Review . . . 16

7.1.2 Interview . . . 26

7.2 Data Analysis . . . 27

7.2.1 Tabulating the Data . . . 28

7.2.2 Assessment . . . 29 7.2.3 Analysis . . . 30 7.2.4 Recommendation . . . 30 7.2.5 Industrial Evaluation . . . 30 7.3 Threats to Validity . . . 31 8 Results 32 8.1 Tabulation of Data . . . 33 8.2 Assessment . . . 38 8.3 Analysis . . . 42 8.3.1 Use Cases . . . 43 8.3.2 Safety Standards . . . 44 8.3.3 V-Model Categorisation . . . 46 8.3.4 Keyword Classification . . . 48 8.3.5 Analysis Resolutions . . . 50 8.4 Recommendations . . . 51 8.4.1 Test Plan . . . 51 8.4.2 Test Strategy . . . 51

8.4.3 Test Status Report . . . 52

8.4.4 Test Completion . . . 52

8.4.5 Test Design Specifications . . . 52

8.4.6 Test Environment Set-up and Monitoring . . . 53

8.4.7 Test Execution . . . 53

8.5 Industrial Evaluation . . . 53

8.5.1 Refined Recommendations . . . 56

11 Conclusions 60

References i

Appendices iv

A. Appendix A - Safety Standard Requirement Extraction . . . iv

B. Appendix B - Interview Questions . . . xxi

C. Appendix C - Data Sources and Types . . . xxiv

D. Appendix D - Condensed Assessment Results . . . xxv

E. Appendix E - Focus Group Procedure . . . xxxvi

F. Appendix F - Recommendations . . . xxxvii G. Appendix G - Refined Recommendations . . . xlii

1

Introduction

In the modern-day world, many systems depend on the correct operation of machines and autonom-ous systems. As many industries encounter the development of safety-critical systems, one of the main concerns that affect our everyday lives, is the considerable consequences of their failure. These consequences may involve loss of life, property damage, or environmental damage [1]. As electron-ics technology advances, more and more systems are becoming embedded hardware-based, leading to the challenges of software safety and system performance quality [2]. The quality of a product consists of many aspects, one of which is reliability and completeness of testing [3]. Throughout the years, different safety standards have been generated to assure certain safety aspects of control systems in different domains, including testing and validation. Optimal test processes can assure the completeness of testing according to testing standards. Test processes are knowingly costly, as well as resource-consuming [4].

This thesis intends to tackle the problem of compliance of safety-related systems’ test processes with relevant standards. The assessment of the compliance of a process with certain criteria entails reviewing the documentation produced in the standard and verifying they adhere to the instructions in the standard [5]. The fundamental question directing this research addresses recommendations entailed in criteria for a test process, which will assist the industrial control systems to achieve higher conformity with specific standards. One step for reaching the main goal of the research is gathering an assembly of relevant instructions concerning test processes and validation of safety-related control systems. Using this collection, a case study shall be performed to solidify the compliance of a specific case to safety standards. The case selected for this case study consists of the development process documentation for two use cases, provided by an automation company in Sweden. The results achieved from studying the compliance of the process in place at the company can be transferable to other similar cases. A collection of instructions assisting the test process of an example of a control system to comply with safety-related standards has been the answer to the first research question. Answers to the research questions have led to the utilisation of a lightweight method for generating recommendations for improving the test processes of industrial control systems.

This thesis aims to investigate relevant safety standards concerning control systems in regards to their application in safety and validation in the test processes of industrial control systems. Furthermore, the compliance of a test process with relative safety standards is investigated. This study continues to conduct a thorough case study at a large automation company to assess the degree to which the existing processes comply with the safety standards. The extent of compliance of a company with relevant safety standards is decided through reviewing the documentation provided by the company. The review has been done through multiple stages, applicable to other companies. Moreover, the review will lead to ensuring the thoroughness of the justifications of the conformity of the company’s process with the relevant standards.

The steps performed for answering the research questions in this thesis include tabulating the data, assessment, analysis, recommendation, and finally industrial evaluation. Data collection that takes place prior to and in parallel to these steps, determines the development process that is to be assessed. In addition, the data collection results in the assessment criteria, which is a selection of requirements from the safety standards. Careful consideration of the desired safety standards leads to assessment criteria. Collecting documentation on the development process of a company leads to evidence for compliance of the process with said standards. Tabulation of the data sources and the assessment criteria into the V-Model Categorisation and Keyword-Classification schemes further enlightens the areas in need of improvement. This tabulation can be customised to a development process, as it uses keywords and qualitative analysis of texts. Once tabulated, the evidence and criteria are matched and assessed. The results of assessing the example industrial control system have resulted in 58% failure of compliance with the selected safety standards. Furthermore, the analysis of these results revealed the areas of growth to be within integration testing, traceability of the documentation, and performance of analyses for achieving test cases. The recommendation step uses the results of the assessment and analysis phases to cluster and summarises a series of suggestions to improve the test process in place at a company. The recommendations generated in this specific case were focused on Test Planning activities. Furthermore, the recommendations, assessment, and analysis results are to be evaluated by those affected. The case under study utilised

the recommendations by prioritising and further clustering them. The focus group also revealed the usability and the value of the results achieved through the assessment and analysis stages.

This thesis has put forth a set of steps for lightweight assessment of test processes of industrial control systems, as a result of its exploratory case study. The steps in this method require further refinement and evaluation to be applied generally and can contribute to industry and academia by limiting the resources needed for identifying areas of improvement in the test process of industrial control systems. The thesis mostly focuses on the results of the assessment of an example of an industrial control system’s development process with a focus on testing and recommendations made for the test process in question.

2

Background

The thesis aims to make recommendations for improving test processes, taking into account the assessment of development processes. The steps taken in this thesis for assessing the development processes are applied to a large automation company in Sweden. For achieving the goals of the thesis, working closely with a large automation company in Sweden, various safety standards will be taken into consideration and analysed. These standards are regulated systems and compliance with them is usually either required or advised [6].

This study will also focus on the analysis and improvement of test processes, namely the test process in place at the company. The software development process used at the company is the V-Model. As testing is a vital and inconvenient part of every organisation, appropriate and mature test processes are needed [7]. Test process improvement approaches guide through the improvement of different software testing processes [4]. In this thesis, the assessment of the processes is done by reviewing documentation provided by the company and ensuring their compliance with specific standards.

As test process improvement is of great importance, it must be dependent on its compliance with appropriate safety standards. The compliance of a test process with safety standards is determined through the review of documentation provided. The documentation provided by the company involves their development process, following the V-Model. How the following of the recommended guidelines is justified in the documentation, is what leads to compliance. Compliance with a standard is particularly of high importance in safety-critical systems since the test processes recommended in these standards ensure the safety of human lives as well as avoiding the loss of equipment and harming the environment.

2.1

V-Model Process

The software development life cycle model used by the company is the V-Model. The book “ Software Testing: An ISTQB-BCS Certified Tester Foundation Guide” is used as a reference for the definition of the V-Model [8]. The book [8] states that a software development life cycle is basically a set of processes containing tasks and activities, which leads to the entire cycle from the retrieval of requirements to the release of the system. One of the simple, traditional development life cycle models is the Waterfall model. The V-Model is an extension of the waterfall model and has many different variants [8].

The Waterfall model sequentially takes steps from the beginning being requirement specifica-tion, all the way to the last step, which is testing. The steps in the waterfall model are as follows: requirement specification, functional specification, technical specification, program specification, coding, and finally testing [8]. Testing acts as a final quality check in the Waterfall model, and the checks done throughout the life cycle are known as verification and validation. Verification includes the reassurance of the system meeting the requirements and the correct development of a system. Validation, on the other hand, is the set of tasks that ensure the conformity of the system with customer requirements, meaning reassurance that the correct system is being developed. V-Model is known as an extension of the Waterfall model. Figure 1 shows a variant of the V-Model for software development.

Figure 1: A variant of the sequential V-Model presented by [8]. The figure demonstrates a model for a software development life cycle, stating different phases or activities which must be achieved. The order to follow the sequential V-Model of software development is also demonstrated.

As can be seen in Figure1, the V-Model contains all the steps that the Waterfall model does. However, in the V-Model, the planning for testing is done at the start [8]. The first step of the V-Model is requirement specifications, in which the needs of the user or customer are captured. Plans for acceptance testing must be made at this stage since acceptance testing includes testing the system against requirement specifications. The next step is functional specifications, during which the system design is done. This step is when the functions aimed to fulfil the requirements are defined. The functional specification stage is parallel to system test planning since the system tests are designed based on the functional specifications. The third stage is technical specifications, during which the previously defined functions are designed at a high level. The technical design of the functions helps clearly define the connections between the internal entities and other systems or the environment, which provides the necessary information for designing the integration tests. Once the technical specifications and the integration planning are done, the program specification phase begins. Program specification includes a low-level design or an internal design of all the modules; this is the phase that makes the system ready for unit test planning.

Once all the specifications are completed, and with them, the test plans are done, the coding is performed [8]. The right side of the incremental V-Model focuses on testing. Keep in mind that at the point of coding, all the test plans are ready. The tests are done from the bottom up. First, the unit tests are executed, making sure that each module functions correctly. Next, the relationship between the modules, and the technical specifications are tested via integration testing. Once the technical aspects of the modules are confirmed to work properly, the system is tested against the functional specifications of the system; this is called system testing. The final step of testing according to the V-Model is acceptance testing, which aims to test the system with respect to the requirements presented by users [8].

The V-Model is an extended model of the Waterfall model and is used as a model for software development life cycle [8]. The V-Model has specifications about planning the tests during the specification of each phase of the system and executing them later. The planning of the tests helps with better specifications, and faster execution of tests once the code is done. The development life cycle model used at the company is a variation of the V-Model. A simplified variant of the V-Model is utilised at the company, which allows the safety requirements to be satisfied.

2.2

Safety Standards

Safety standards can be categorised into two groups: those which focus on the practical guidance on methods to be used during development for achieving the system objectives, and those which apply constraints on the development process without practical guidance [9]. The standards taken into consideration for this thesis will be among those considered as highly relevant in the context of process assessment of safety-critical control systems, and will mostly belong to the first group

Safety standards are applied to the development processes of safety-critical systems in order to reduce risks to an endurable level. Generally, safety standards in industrial control systems tend to specify safety requirements in addition to a guideline to their implementation through safety functions. [10]. The criteria for choosing the standards used during this case study include their involvement in functional safety of machinery, software testing and development process assessments, and those specified to be mandatory by the company. The standards chosen are to be concerned with safety, clearly address test process assessment and safety of machinery in control systems, and be highly regarded in their respective domains.

2.2.1 IEC 61508 Standard

One of the standards used in this work is IEC 61508. This is a standard for the functional safety of electrical/electronic/programmable electronic safety-related systems. What brings this standard to attention for this thesis is its applicability to control systems. This standard is likely concerned with the safety of systems. Safety in this context is referred to as the absence of failures that may cause harm to persons, environments, or have economic implications [11]. Furthermore, Nevalaian et al. has identified this standard as being useful when it comes to using methods and techniques as evidence for the evaluation of the achievement of safety [12].

IEC 61508 is an international standard addressing system performing safety functions having electrical, electronic, or programmable electronic (E/E/PE) elements [11]. The standard contains recommendations and instructions for all safety lifecycle activities for the systems mentioned. The standard covers a specific scope, namely the characteristics that E/E/PE systems must have if they carry out safety functions. As IEC 61508 is generic, it is therefore applicable to all E/E/PE safety-related systems, regardless of their application, with the exception of medical equipment in compliance with the IEC 60601 series. Compliance with this standard entails the satisfaction of all the relevant requirements with their respective criteria and meeting the objective of each specified clause [11].

The standard IEC 61508 has many clauses, of which the software safety lifecycle requirements in part 3 are of interest. However, it is useful to briefly look at the other clauses related to software requirements. Part one of the standard focuses mainly - other than the scope, documentation, and management of functional safety - on the overall safety lifecycle requirements in systems under interest. The overall safety lifecycle aims to systematically deal with the necessary activities for having the required safety integrity in a system. The lifecycle has sixteen main phases in addition to Verification, Management of Functional Safety, and Functional Safety Assessment. The final three are relevant to all the main sixteen phases. For each phase, the standard includes the sub-sequent sub-clauses, adhering to which, leads to the compliance of the phase with the standard: the objectives, scope, sub-clauses with requirements, required inputs and required outputs. All these aspects are documented in detail in IEC 61508-1:2010. The phases in the overall safety lifecycle are: Overall scope definition, Hazard and risk analysis, Overall safety requirements allocation, Overall operation and maintenance planning, Overall safety validation planning, Overall installation and commissioning planning, E/E/PE system safety requirements specification, E/E/PE safety-related systems – realisation, Other risk reduction measures – specification and realisation, Overall install-ation and commissioning, Overall safety validinstall-ation, Overall operinstall-ation, maintenance and repair, Overall modification and retrofit, Decommissioning or disposal, and Verification.

Even though the standard recommends overall safety lifecycle requirements, the main point of interest for this thesis is the software safety lifecycle requirements presented in IEC 61508-3:2010 [13]. The document includes nine general requirements for the software safety lifecycle, which lead to the software development process being structured into defined phases and activities. Once the general requirements are stated, the standard moves on to the requirements and objectives of software safety requirements specifications. The sub-clause that follows includes guidelines for the validation plan for software aspects of system safety. The requirements in this sub-clause include the specific technical and procedural steps to be carried out for planning so that the safety requirements are satisfied. More precisely, the standard continues with the aspects of system safety that must be considered in this phase. The sub-clause includes the required information for the technical strategy that has been chosen for validation, as well as criteria for accomplishing software

validation [13].

After the three phases explained above (General requirements, Software safety requirements specification, and Validation plan for software aspects of system safety), comes the stage of Soft-ware design and development. The objectives that must be achieved for the softSoft-ware design and development phase to be compliant with the standard are stated clearly along with some general requirements. These general requirements precede specific criteria for software architecture design, provisions for support tools, including programming languages, conditions for detailed software design and development, specifications for code implementation, requirements for software module testing, and last but not least necessities for software integration testing. All the sub-clauses in the software design and development phase include their objectives and requirements, which will be employed for deciding on the compliance of the company’s documentation in regards to the two use-cases with instructions provided by instructions IE 61508.

2.2.2 ISO/IEC/IEEE 12207 Standard

ISO/IEC/IEEE 12207, titled Systems and software engineering — Software life cycle processes, is a standard used by the company under study. The compliance of the software development process at the company with the standard will be thoroughly assessed. Although ISO/IEC/IEEE 12207 is not a safety standard per se, it is concerned with critical quality characteristics. The quality characteristics subsume aspects related to health, safety, security assurance, reliability, availability and supportability [5]. Safety in this standard is referred to generally as an expectation of not endangering human life, health, property, or the environment. Therefore, the standard ISO/IEC/IEEE 12207 is utilised to assess the processes of the software development for the use cases under interest, due to its safety measures in addition to its usage by the company.

The document for the standard ISO/IEC/IEEE 12207 includes instructions and requirements for various processes [5]. The standard acknowledges its vastness and states that all the provided processes may not be needed for a particular project. Therefore, the document provides instructions for full conformance and tailored conformance, meaning the readers will have ways to assess a life cycle’s compliance with this standard tailored to the needs of the system under interest. The full conformance is a union of conforming to the outcomes and the tasks in a process. There exists a prescribed tailoring process in the standard, which assists with modifying the outcomes and tasks to the needs of a specific system under interest. One may claim tailored conformance to the standard by demonstrating that the outcomes, activities, and tasks, as modified, have been achieved [5].

The documentation for the standard ISO/IEC/IEEE 12207 uses some essential concepts as its basis which are thoroughly described [5]. Generally, the concepts can be divided into four groups: software system concepts, organisation and project concepts, lifecycle concepts, and process con-cepts. Software system concepts involve explanations about a software system, its structure, its lifecycle processes, as well as enabling systems. Organisation and project concepts explain that the execution of processes by which organisations or parts of organisations is not of concern of this document. Lifecycle concepts have explanations for software lifecycle stages and lifecycle models for software systems. It is presented that, lifecycle stages portray the major progress and achieve-ment milestones of the software system through its life cycle. A lifecycle is additionally explained as an abstract functional model. The functional model must embody the need for a system, the system’s realisation, utilisation, evolution, and disposal [5].

The descriptions of the concepts for the processes in the documentation of the standard are of most interest to this piece of research. The process concepts model three basic principles for determining what is identified as a life cycle process. According to ISO/IEC/IEEE 12207, there is a strong relationship between the tasks, outcomes, and activities in a life cycle process [5]. Each process is to have the ability to be executed by a single organisation in the life cycle. Also, the processes are to have as minimised dependencies amongst them as feasible. Besides fulfilling the basic three criteria mentioned, a process in the documentation of ISO/IEC/IEEE 12207 has the following attributes: a title, purpose, outcome, activities, and tasks. The title of a process demonstrates the domain and scope of the process. A process’s purpose is a description of the goals that are to be achieved from performing the process. The outcomes of a process are referred to as the expected results that can be perceived once the process is successfully executed. The set

actions, requirements, or recommendations that assist in achieving the desired results from the process [5].

Now that the concepts have been explained to the extent needed to understand the work, the software life cycle processes are taken into consideration. The processes are classified into four groups: Agreement Processes, Organizational Project-Enabling Processes, Technical Management Processes, and Technical Processes. Furthermore, there are requirements for the processes in each of the groups. For each of the processes, requirements and instructions are provided for their purpose, tasks, activities, and outcomes. The actual requirements will not be mentioned in this text but may be accessed through the actual text in the documentation for the ISO/IEC/IEEE 12207 standard [5].

The Agreements Processes group includes the acquisition process and the supply process. Or-ganizational Project-Enabling processes consist of six processes, each with their recommendations regarding their attributes. The processes in this group are the Life Cycle Model Management process, Infrastructure Management process, Portfolio Management process, Human Resource Management process, Quality Management process, and Knowledge Management process. The following eight processes are grouped into the Technical Management processes: Project Planning process, Project Assessment and Control process, Decision Management process, Risk Management process, Configuration Management process, Information Management process, Measurement pro-cess, and Quality Assurance process. Finally, the Technical processes consist of Business or Mission Analysis process, Stakeholder Needs and Requirements Definition process, System/Software Re-quirements Definition process, Architecture Definition process, Design Definition process, System Analysis process, Implementation process, Integration process, Verification process, Transition process, Validation process, Operation process, Maintenance process, and Disposal process.

To sum up, the document for ISO/IEC/IEEE 12207 standard used by the company, comprises requirements, definitions and recommendations in regards to software life cycle processes and is used as a reference for software life cycle processes. The total software life cycle is divided into groups of processes and each group has multiple processes. The processes have basic attributes, requirements about which are provided in the standard documentation to assure quality and safety for the software-under-interest.

2.2.3 ISO/IEC 29119 Standard

ISO/IEC/IEEE 29119, titled Software and systems engineering – Software testing, is a standard with four parts, each containing requirements about different aspects of software testing [14]. Part 1 of ISO/IEC/IEEE 29119, titled Concepts and definitions, specifies the concepts and definitions in software testing which are vital for understanding the other parts of the series. ISO/IEC/IEEE 29119-2, Test processes, is the part that describes the test processes that can be utilised for the implementation, governance, and management of testing software, and works independently of which software development life cycle model is utilised. Part 3 of the series is titled Test documentation, and it comprises templates for various software test documentation, specified as outputs of processes described in part 2. Part 4 includes descriptions and definitions for test design techniques that may be used for the processes in ISO/IEC/IEEE 29119-2. The thesis at hand will focus mainly on parts 2 and 3, using them for assessment of the process in place at the company, as well as making safety-related recommendations for a control system’s test process.

ISO/IEC/IEEE 29119 has classified the testing activities in a software life cycle into three layers: Organisational Test Process, Test Management Processes, and Dynamic Test Processes [15]. The standard continues to further describe the processes in each group in terms of three attributes. The attributes used to describe each process include the purpose, activities and tasks, as well as the desired outcome of the process. The document explains which activities and tasks must be performed per process, and defines the purpose of the processes clearly. To conform to this standard, the desired outcomes of each process are to be assessed as well. The activities, tasks, and desired outcomes may be modified based on the needs of the system. When modifications are made, they must be justified and recorded, and include the applicable risks of the tailoring. The justifications of the tailoring decisions may lead to tailored conformance of test processes with the ISO/IEC/IEEE 29119 standard [14].

The organisational test process layer aims to define processes for creating and maintenance of organisational test specifications [15]. The organisational test specifications may include test policies, strategies, procedures, and so on at the organisational level. The layer involves a single process with an expected information item as a result, desired outcomes, its purpose, and three basic activities, each with their tasks. The details of each attribute are not mentioned in this thesis but can be read in the documentation of the standard [14]. The second layer of the test process model used in ISO/IEC/IEEE 29119 is the Test Management Processes layer. There are three processes each having its own aforementioned attributes. The test management processes are generically applied at different levels and test phases in a development process. The three processes are the test planning process, test monitoring and control, and the test completion process. The final layer, namely the Dynamic Test Processes layer, involves four processes. Again, the processes each have a purpose, a set of activities with tasks, the desired outcome, and an information item that is produced as a result of the execution of the process. The attributes involve the details of the requirements and recommendations given by the standard. The processes in the final layer consist of the Test Design and Implementation Process, Test Environment Set-Up and Maintenance Process, Test Execution Process, and finally the Test Incident Reporting Process. The details of each of these layers and their respective processes assist in having a safe and secure overall test process [15].

As previously mentioned, ISO/IEC/IEEE 29119-2 has an information item per process. These information items are documents resulting from the successful execution of each process at each level. Part 3 of the ISO/IEC/IEEE 29119 series provides templates for these documents [16]. Following these templates, or justifying the specific changes made to tailor them to the needs of the system-of-interest, will lead to compliance with ISO/IEC/IEEE 29119-3. The details of these templates are not given in this section but are used to decide on conformity of the test process at the company, as well as making relevant recommendations for a safe test process.

2.2.4 ISO 13849 Standard

The safety standard of central interest in this thesis is ISO 13849, both parts 1 and 2. This standard is used by the company for their software development process along with ISO/IEC/IEEE 12207. As this research aims to assess the test process of industrial control systems, the main part considered from ISO 13849 is the software safety requirements clause.

Software safety requirements in ISO 13849 seek to achieve software with high readability and understandability, as well as maintainability, and most of all testability [17]. The standard hopes to achieve this objective by stating that the faults introduced in the software life cycle should be avoided in the life cycle activities of safety-related application or embedded software. The requirements in the standard are divided into those for safety-related application software and those for safety-related embedded software [17].

Fundamental measures are to be taken for safety-related embedded and application software as recommended by ISO 13849. The first measure includes the software safety life cycle with veri-fication and validation activities. Secondly, design and speciveri-fication documentation is required. Modular and structured design and coding is the third measure to be taken. Control of systematic failures and functional testing are among the steps recommended by the standard. Finally, appro-priate software safety life cycle activities, proceeding modifications of the system, are required as expected actions for safety-related software.

In addition to these instructions, safety-related embedded systems containing components with performance levels with a high probability of a rate of dangerous failure shall take more measures. This high-risk safety-related software should pay more attention to project, quality, and configur-ation management and quality management systems along with more structured code and testing criteria among other measures.

The safety-related application software should apply the previously mentioned fundamental measures, together with more actions for components with higher risk rate [17]. The safety stand-ard makes recommendations in ten different areas. These areas consist of reviewing the safety-related software specifications, selection of tools, libraries and languages, features mandatory for the software design, and recommendations for where both safety-related and not safety-related ap-plication software are combined in one component. The clauses also include instructions regarding

ment, and finally modifications of the software [18]. The recommendations and instructions given for each of these areas are explained further in the document for ISO 13849-2:2012 [18].

2.2.5 ISO/IEC 33063 Standard

ISO/IEC 33063 is a standard titled “Information technology — Process assessment — Process assessment model for software testing”. This standard has been suggested as a process assessment model for software testing by Garcia et al. [19]. ISO/IEC 33063 will help with the assessment of the actual process in place at the company, and given its assessment criteria assist with future recommendations regarding a test process compliant with safety standards. This process also uses the process reference model in ISO/IEC/IEEE 29119-2 as its basis [20][15].

The standard ISO/IEC 33063 has as its fourth clause an overview of the process assessment model. The structure of the process assessment model is explained in the standard, through the outlining of the assessment criteria for different processes [20]. The standard is strictly used as a reference guideline for process assessment as opposed to the other standards, which are used for their safety-related requirements. The document continues to discuss process dimensions and process performance indicators. ISO/IEC 33063 includes assessment guidelines for the compliance and conformity of the process assessment model [20]. This standard will be studied in the future phases of the thesis and used as a reference assessment model.

3

Related Works

The thesis at hand focuses on test process assessment of industrial control systems via safety standards. The objective of the thesis can be partitioned into viewing how the test process as-sessment is conducted in general. The asas-sessment of test processes in the industry is performed in various ways, and Toroi et al. [21] have done interesting work in identifying areas of improvement in test processes. In general, industrial control systems have an approach to their test processes to assure safety; this is discussed in Nunns et al. [22]. Finally, the use of safety standards in assessing the test processes is considered by researchers. Safety standards, in general, help with the assessment of various phenomena. The way standards are applied to test processes is discussed by Panesar-Walawege et al. in two of their works [23] [24].

Many methods are used for assessing and improving test processes. The standard ISO/IEC 15504 gives guidelines for the software testing life cycle processes, and CMMI determines how verification and validation should be performed [21]. The research done by Toroi et al. states that test improvement models are found to be too difficult by software companies, producing the need for a lightweight approach to test process improvement [21]. This research used the LAPPI (A Lightweight Technique to Practical Process Modeling and Improvement Target Identification) technique for modelling the test processes studied and proceeded to identify improvement targets in the test processes. The case study was performed in three case organisations. All cases had issues with inadequate unit testing, handling code modifications, and a lack of exit criteria for testing. The study helps this thesis have more of a focus on these areas as well. Another issue found by Toroi et al. was the integration of automated and manual test processes. The study suggests that there is a need for test process standardisation and confirms this by identifying the common and the test related problems that reappear in the software industry [21].

In order to assess the safety of a system, an approach needs to be consistent. Many different methods are being used for the assurance of safety among industry, which may lead to confusion [22]. Nunns et al. identify one of the approaches helping the industry face various challenges to be the use of safety standards. One of the safety standards used for safety assessment is IEC 1508, an International Standard by the International Electrotechnical Commission, which addresses safety-related systems [22]. IEC 1508 contains general documentation of standards used in different sectors of industry. The seven different parts of IEC 1508 undertake various instructions regarding requirements, definitions, and the application of the instructions. The standard consists of seven parts: Part 1 General Requirements, Part 2 Requirements for electrical/ electronic programmable electronic systems, Part 3 Software Requirements, Part 4 Definitions, Part 5 Guidelines on the application of Part 1, Part 6 Guidelines on the application of Parts 2 and 3, and Part 7 Bibliography

of techniques. Nunns et al. demonstrate how IEC 1508 helps with the assessment of safety in different sectors of industry by involving four key concepts. The concepts are The Safety Life Cycle, Safety Management, Competencies, and Design of safety-related control and protective systems. The guidelines for the structured approach to the safety life cycle assist with the assessment of processes [22].

Panesar-Walawege et al. [23] describes A Model-Driven Engineering Approach to Support the Verification of Compliance to Safety Standards. The methods used to ensure compliance with safety standards help this thesis’s approach to the assessment of the test process via safety standards. The work by Panesar-Walawege et al. addresses the complications due to the company’s risk of lacking essential recorded details during development [23]. The work also takes into account the loss of productivity of an assessor leading to delays due to incomplete poorly structured evidence and documentation provided by the company. To face these challenges, Panesar-Walawege et al. use various Model-Driven Engineering technologies to propose an approach to system designers for linking what the standards require as evidence to the concepts of their application domain. The challenge of making such relations threatens the progress of this thesis, therefore, the methods used in the research done by Panesar-Walawege et al. are of great help. The thesis at hand also faces the challenges of understanding the documentation provided by the company, as well as justification of the collected evidence being complete with attention to the standard’s requirements. The work by Panesar-Walawege et al. helps to face these challenges by using UML profiles and developing a profile of the system according to a standard’s conceptual model. This modelling helps with linking the safety standard’s concepts to the application’s domain in a systematic way, leading to a clear route for showing the relationship between the company’s documentation and the standard [23].

Panesar-Walawege et al. use Model-Driven Engineering techniques to verify a system’s com-pliance with safety standards in later works [24]. As previously mentioned, the presentation of evidence by the suppliers of safety-critical systems is essential for the demonstration of their com-pliance with certain safety standards. Clear-cut interpretation of the standards is needed to avoid missing crucial details during development [24]. The work by Panesar-Walawege et al. states the importance of a systematic approach to assessment, which is also supported by automation. Panesar-Walawege et al. suggest a method for linking the concepts of an application domain to the requirements of relevant standards. Just like their previous work, Panesar-Walawege et al. use MDE technologies. To create a profile for the domain model of the system, UML is used, which is then augmented with constraints and transformed into the Object Constraint Language [24] [23]. Panesar-Walawege et al. performed a case study resulting in how their approach of using general modelling tools can assist in the production of adequate evidence for verifying the compli-ance of a system with standards. The approach proposed by Panesar-Walawege et al. establishes a relationship between the domain model of a system and the model of a safety standard. The constraints put forth by the standards and the profile are automatically verified using the existing OCL constraint engines [24].

The studied works suggest the need for a systematic and disciplined approach to the collection and analysis of data in this thesis. The related works are about the test process assessment of industrial control systems and the role safety standards play in this assessment. The pieces of research inspire a clear path for assessing the development processes in industry, when a company aims to follow specific safety standards. The works help with setting a path for creating a systematic view of the evidence provided by the company, and the requirements put forth by the standards. Another outcome of studying the related works is how the improvement targets in test processes of industrial control systems are identified and which areas need more focus.

4

Problem formulation

All safety-critical systems are required to meet specific safety standards. Safety standards ensure that the appropriate validation mechanisms for a control system are in place and the system meets safety. By conducting an exploratory case study, one can assess the test process of an industrial control system with regard to safety standards and develop the steps to do so. This thesis will investigate various standard documentation concerned with safety and process assessment, and by applying them, evaluate the compliance of an existing development process at a large automation company in Sweden.

The thesis will investigate four different safety standards. The selection of safety standards aims to include ones that regard test processes, safety, and activities within the development process that concern test processes. The safety standards are explored in the context of safety-critical control systems’ test processes. Furthermore, the standards were reviewed and chosen according to their relevance to the functional safety of machinery and testing standards.

The problem addressed is the compliance of a development process, with attention to testing, with the noted safety and process standards. A lightweight method is used for thoroughly studying and investigating development processes with respect to the chosen standard documentation. The use of a lightweight method is beneficial as opposed to a full assessment, as it utilises fewer resources and identifies areas with a need for more effort faster [25]. The method is performed on the development process in place at an automation company. Furthermore, recommendations made for improving the process will aid in the efficiency and compliance of the testing process with selected safety standards, as they bring forth the relevant requirements in one place. This will contribute to the world of research in exploring new methods for making beneficial recommendations for test processes and the assessment of development processes that aim to follow specific safety standards. The exploratory aspect of the case study performed in this thesis leads to unfolding areas that need consideration in test process assessment and improvement.

This study aims to answer the following research questions:

RQ1. What recommendations further improve safety compliance for the test process of an industrial control system?

RQ2. To what extent does the current development process in place at an automation company comply with specific safety standards?

The first research question is the main problem addressed in this study. The thesis answers the research questionRQ1by answering the second research questionRQ2. The main research question results in a collection of recommendations that, if followed, will lead to a test process compliant with a selection of safety standards. The method used to assess the development process is generic and may be applied to other cases. To accomplish the desired set of recommendations for the test process, initially, the suggestions made by relevant safety standards are to be gathered. Secondly, the assessment of the development process, with attention to testing, leads to valuable information regarding the areas of improvement in the development process leading to a more compliant test process. Hence, answering the second research question gathers a concise collection of instructions from preferred safety standards, using them for more distinct assessment criteria for compliance. The exploratory nature of this study has led to a lightweight method used for the assessment of the development process of industrial control systems and is applied to an automation company to answer the research questions. This methodology for a lightweight assessment of a test process is a by-product of answering these research questions.

4.1

Scope Definition

This thesis focuses on standards concerned with safety and test processes in control systems. It is paramount to assign boundaries to such a broad problem. The scope of this study is limited to safety-related recommendations for the test process of industrial control systems. The recommend-ations are specific to the test process within the software development process of control systems. To bound the domain of the standards, they are narrowed to those related to the functional safety of machinery, specific to software development processes, testing, and validation. Therefore, while

studying the standards, only elements concerning safety-related parts of control systems are con-sidered. In addition, the requirements extracted from the safety standards are those that somehow affect the test process of an industrial control system. The scope of the thesis does not include in-structions from the standards that address areas unrelated to testing. The test process researched will be specific to two safety functions in the form of use cases provided by the company, which will conclude transferrable results.

4.2

Limitations

One remarkable limitation conflicting the world at the moment is the COVID-19 pandemic. This pandemic has caused restrictions concerning access to on-site resources of the company and delays in information gathering.

The issue of confidentiality is considered as the thesis includes data from an automation com-pany in Sweden. Initially, the confidentiality issue was dealt with by signing a Non-Disclosure Agreement (NDA) by the student. Under the NDA, access is granted to the needed information for performing the thesis work. Furthermore, the company has provided an excerpt from the pro-prietary software which is allowed to be included in the public thesis report. Another approach for addressing the confidentiality issue has been to avoid sharing any sensitive data on any cloud platform and using local and not cloud-based backup methods for the work.

5

Method

This thesis is performed as a case study by validating industry needs concerning test process assessment. As the thesis hopes to contribute to both academia and the needs of the industry partner, the method chosen is a case study executed at an example company, the results of which will be transferable to other cases with a similar development process. The methods for data collection in this case study can be used for other cases, as the gathering of archival data and interview are applicable for other companies. Furthermore, the data analysis steps are transferable as they are clearly defined in the following sections and can be applied to other companies with similar development processes. Multiple sources of assessment criteria are utilised on two different use cases, and data triangulation is considered by using observation, documentation, and interview as data collection methods [26]. The case study will ensure the compliance of the test process of safety-critical control systems with relevant safety standards. The case study method will start with a planning stage and follow to iterate through the phases of design, preparation, data collection, analysis, and sharing [27][28].

Once planning and time management was thoroughly complete, the case study design stage led to the definition of objectives and a more reliable overall plan. Subsequently, the prepara-tions for data collection solidified the descripprepara-tions for procedures and protocols for collecting the data [28]. The execution of data collection on the development process at the company followed the preparation stage. Once the needed data was collected, it was investigated by qualitative data analysis methods, with attention to constructing validity, internal validity, and external validity of the study [29]. The thesis performed analysis by studying the compliance of the test process with specific standards. Reporting or sharing the results of the analysis is the final stage. In this stage, recommendations were made to the test process at the company. The general flow of the case study procedure, inspired by Runeson et al. and Baškarada is demonstrated in Figure2[28].

Figure 2: This diagram is inspired by the recommendations of Runeson et al. [28] and Baškarada[27] in regards to case study guidelines. The diagram demonstrates the flow of different stages of a case study, where case definition and case study protocol are sub-steps to be taken for the design phase.

The first stage of performing a case study is the case study design and planning. The design for a case study involves case and subject selection. This step includes precisely defining the objective, the case, the theory, and the research questions. The case in this context is defined as what is being studied. The theory is the frame of reference for the case study, as in the field of software engineering, theories are not commonly used. Once the design of a case study is complete, the preparation is performed. The preparation part of the case study design stage involves the arrangements for the data collection procedures [28]. The methods for collecting the data at the design level for this study are in the independent and the direct categories, namely document analysis and interview, respectively. The selection strategy is seeking data from two specific safety functions, in the form of use cases and the development process in place at the company, with definite attention to testing. The relevant documents regarding the development process of the company, specifically for the two safety functions, was provided by the company. Likewise, an interview conducted with an R&D engineer at the company contributed supplemental information regarding the parts of the development process at the company related to testing.

Figure 3: Figure to show the steps taken for the data collection phase. In step (1), the documents to be assessed were requested and acquired from the company. Step (2) involved choosing the safety standards according to their relevance to the scope of the thesis. During step (3), all the acquired safety standards were thoroughly read to achieve sufficient familiarity with them. Furthermore, in step (4), the generic exclusion and inclusion criteria for the instruction and requirements were produced and then applied to the requirements in step (5) to find those relevant to the scope of the thesis. Finally, The selected requirements were evaluated and confirmed in step (6), with results shown in Tables1,2,3,4,5,6, and7.

The data collection stage in this thesis was performed through document review and inter-view. The various steps of the data collection are shown in Figure3. The document review was executed by the acquisition of documentation concerning the company’s development process and familiarizing oneself with the documents(1)through carefully studying them. Another aspect of the document review stage of the data collection phase is(2)the acquisition and selection of the safety standards. Once the safety standards were read carefully(3), generic exclusion and inclusion criteria for the instructions and requirements were determined(4), which lead to their extraction (5)from the safety standards. Finally, the selected requirements were confirmed by the supervisors (6), leading to a set of instructions and requirements within the scope of the thesis, from relevant safety standards, as well as documentation from the company. The process of performing the independent document review in the data collection stage is shown in Figure3.

After independent collection of data through document review, the first two steps of analysis - explained further in the following paragraph - were performed. Following the first two steps, the data collection was continued through an interview. An interview is a direct method for data collection used in this thesis [26]. A semi-structured interview was performed, with questions about the general instructions of the safety standards. The questions also referred to the sub-clauses of safety standards’ instructions which had a failing compliance degree by the development process. The interview was executed in two stages, one with broad questions about the development process and the other with respect to the requirements failed by the provided documentation. Subsequently, the results of the interview were put through the first two steps of the analysis, and then the analysis was continued. A more detailed description of the steps for the data collection stage may be found in Section7.1.

Another stage of the case study is data analysis. Data analysis is performed by initially (1) tabulating the collected data from the safety standards and company documents and grouping them based on two grouping methods: their application to the V-Model, and abstract keywording. Once all the data was tabulated, and each requirement and the data source were assigned to each group, the development process at the company was assessed. The assessment(2)was performed through their documentation based on the extracted and tabulated requirements. The results of the assessment were then presented in tables assigning a compliance degree to each requirement and referring to a justification. Next, the results of the assessment were analysed via their compliance degree, based on the different safety standards, V-model categories, keyword classification groups, and the two use cases to find patterns in their evaluation(3). The patterns were then assembled (3)and produced conclusions. Following the determination of the conclusions, recommendations were made(4)concerning the test process of the control system. The recommendations were then presented to the company and evaluated in the form of a focus group(5). Finally, all the steps

the company’s development process, as well as a list of instructions from safety standards that are within the scope of test process assessment in safety-related control systems. A more detailed description of the data analysis, along with visualisation of the parallelism with the case study method in Figure5, may be found in Section7.2.

The final stage of a case study, according to Runeson et al. [28], is reporting the procedure. The report stage was done iteratively throughout the entire process of the case study. According to guidelines proposed by Runeson et al. [28] and Baškarada [27], the report structure and char-acteristics considered for this thesis were determined. Section 7 (Case Study Design) follows the propositions of Runesn et al. [28], including descriptions for the selection of case and subject and procedures for data collection, data analysis, and validity of the study. The collected data and res-ults of the analysis procedure are presented in Section8(Results). Section9(Discussions) presents the speculations made from the results of each stage of analysis, and finally, Section11 (Conclu-sions) concludes the thesis by stating a summary of the thesis and presenting the connections between the results and the research statement.

6

Ethical and Societal Considerations

As this thesis addresses safety-critical systems and is performed as a case study, there are ethical considerations on two fronts. Firstly, the assessment that is carried out in this thesis must be precise, as its results may affect the way the development process at the company is viewed by the company. If the assessment is treated too leniently, it may put future users of the systems at risk. Furthermore, the case study is performed on a very real subject which must remain confidential to contain the integrity of the company and avoid possible negative consequences [26]. The collection of data in the forms of document review, interview, and focus groups each come with their ethical issues.

This research involves sensitive data from a large automation company, leading to the need for keeping the confidentiality [29]. The thesis assesses and analyses the company’s development process which leads to results that may lead to negative consequences in the public domain. The consequences are avoided by keeping everything confidential, beyond not naming the company or its members [26]. An effort has been made to redact any words or small piece of information that may lead to exposure of the company’s identity or its specific domain [28]. As the thesis has scientific value, the company is motivated to reveal the information to the student, considering unknown future risks [29].

To avoid further ethical issues, the execution of the case study was performed with the informed consent of the company. Furthermore, the company has been continuously informed of the progress of the thesis via weekly meetings and has access to all the results, data gathered, and analysis performed to avoid possible deception of participants [26]. Data collection in the forms of document review and interview may lead to ethical issues concerning consent, board approval, confidentiality, inducement, and feedback [28]. Board approval and informed consent regarding data collection have been acquired before the start of the thesis activities. Furthermore, the documents are referred to by codes and types as to not reveal any specific information. The interview results are redacted from the public report, and the information is only referenced. Another aspect of ethical considerations in this thesis is performing the(5)Industrial Evaluation step of data analysis in the form of a focus group. Dealing with ethical issues in executing a focus group research is similar to that of data collection through interview or archival documents [30]. To conquer popular ethical issues, the interview and the focus group session were both performed online in a safe environment to put the participants at ease [31]. The participants were also assured of confidentiality and the recording of the session. Once transcribed, the recordings were deleted to assure no further distribution. The focus group session was kept as unbiased as possible, and the questions leading the session were kept simple, short, and distributed in such a manner to avoid confusion [31]. In conclusion, with the actions taken to face the ethical considerations, the risks of this thesis outweigh its benefits as they are minimal, and it will help improve the safety of control systems for the company, and perhaps using the methods presented further the safety of other systems as well [26].

7

Case Study Design

The first stage of performing a case study is the case study design and planning. Following the guidelines provided by Runeson et al. [28], the design stage includes a clear definition of the case and its units of analysis. This stage also clarifies the problem formulation, author’s intentions, existence of data or method triangulation, and the rationale behind the case selection. The design phase also addresses the construct validity of the study, as well as taking into account the integrity of the company under study.

Figure 4: The alignment of the case study procedure with the procedures in this thesis

The objective of this case study is of an exploratory and improving nature [28]. The expected result is a set of recommendations leading to a test process that complies with specific safety standards. The case studied for achieving this goal is the development process carried out at a big company for two specific use cases. The frame of reference for the case study is the testing process and safety; meaning that the process is viewed from a safety perspective, and the main focus is on the testing process. The questions answered as a result of this case study are found in detail in Section4Problem Formulation. In short, the questions involve recommendations for a test process having compliance with safety standards, and the assessment of the current development process at an automation company. The design of this case study utilises data triangulation, as well as method triangulation.

Multiple sources are considered for executing the case study. The assessment is performed on two different use cases, as well as four different safety standards. As for data triangulation, other than a literature review, an interview is conducted, increasing the precision of the study, by the means of it collecting the data on different occasions [28]. The rationale behind choosing the said standards is selecting those which are concerned with safety, clearly address test process assessment and safety of machinery in control systems, and are highly regarded in their respective domains. According to Runeson et al. [28], construct validity is to be addressed in the design phase. Construct validity is the determination of the valid relevance of the specified case to the research questions. Research questionsRQ1and RQ2address the selected case specifically. The selection criteria for the safety standards makes them relevant to the scope of this research, as explained in detail in Section2.2. The relevance of the two use cases chosen from the company to RQ2is determined by the company itself. The main research question,RQ1, uses the results from the other research question to generate a set of suggestions for a test process compliant with the selected safety standards. The generalisation of the issues in place at the company under study may be done in further research by applying the same process of generating recommendations on other industrial control systems which use the same development process model. The company’s integrity is taken into account by removing their specific data and name, avoiding their identification in the public domain.

The case study design explains the process of preparing for, and performing the data collection stage [29]. This thesis acquires the relevant data to analyse through independent document review and an interview. The document review is executed in six steps, shown in Figure3. Furthermore, after the analysis of the data attained by document review, conducting an interview confirms the results of the analysis and assessment, leading to the second form of data collection. The data collected from the interview is analysed once more, going through the steps as explained in Section 7.2. Each of these data collection methods is further described in this section.

7.1.1 Document Review

The data collection stage, in this thesis, involves obtaining the appropriate documentation from the company, selecting the standards, and extracting the relevant requirements from the specified safety standards. A visualisation of the different steps of data collection in this thesis is shown in Figure3 Firstly, the relevant documentation is obtained from the company(1). This step was performed in the beginning by receiving the relevant documentation about the two specific use cases from the company, and further on iteratively throughout the process as the need for more documentation was established. The acquisition of company documentation(1)is followed by selecting the desired safety standards(2)with specific criteria, described in more detail in Section7.1.1.2. After reading and becoming familiarised with the selected safety standards(3), generic criteria for including and excluding clauses and sub-clauses from the safety standards are produced(4). Using the generic inclusion and exclusion criteria, each clause and sub-clause of the safety standards is determined to be included or excluded (5) for data assessment, with specific justification per clause or sub-clause, and a reference to the generic inclusion or exclusion criteria. Once the extraction of relevant requirements from each safety standard was finished, these requirements were further evaluated(6) and refined. The guidelines proposed by Runeston et al. [28] require the data collection to include the specifications of triangulation as well as measurement definitions. The guidelines also specify that the gathered evidence must enable further analysis, identify sensitive data, be traceable, and address the research questions.

To assure the clarity, transferability and reliability of the study data triangulation has been performed [28]. The selection of multiple data sources for this study leads to data triangulation. The data include various documentation provided by the company, which concern two different use cases. The data also involves four different standards concerned with safety and software lifecycles. The selection of data sources regarding two use cases rather than one, and the inclusion of multiple safety standards for process assessment at the company, gives a larger sample for the case under study, which is the development process in place at the company. Having multiple sources leads to a broader application of the data, which also assists with the transferability of the study. The recorded data also allows further analysis to ensure the performance of the case study. As some of the data is sensitive to the organisation, the data is labelled vaguely in the public reports and the name of the organisation is not mentioned. The text snippets extracted from the documentation provided by the company are ensured as to not contain information allowing an onlooker to identify the company. Finally, the collected data provides the ability to address the research questions in various ways.RQ1is answered by the outcomes ofRQ2. AsRQ2refers to the assessment of the development process at the company, certain assessment criteria must be specified. Recommendations of a selection of safety standards lead to the assessment criteria, therefore, the principle under which the standards are chosen is important in the data collection phase. The data provided by the company, with its relation to the requirements of the standards, help fulfil the answers toRQ2.

The measurement definitions for the data are the criteria under which the standards and the company documents have been chosen. The measurement procedures for performing the analysis must be specified in the data collection phase [28]. The analysis of the collected evidence will be based on the conformance of the provided documentation with the selected instructions and requirements from the specified standards. This conformance is measured in three levels of Pass, Fail, and Partial, otherwise named a compliance degree throughout the thesis. The extent to which the company conforms with the standards is measured based on what is specified in approved documentation. If a document has not been approved based on the company system and is a work

in progress, a Partial grade will be granted with comments as to what is missing. A Partial score is also granted when some of the sub-clauses of a requirement clause are met and some are not. Other than the two instances mentioned, the passing or failing of the requirements and instructions are done in a binary fashion.

As previously mentioned, the measurement definitions for the data include the criteria under which the standards and the company documents have been chosen. The standards have been selected based on their addressing of software testing and development process assessments and the functional safety of machinery. The selection also takes into account the mandatory standards identified by the company, and that the standards be highly regarded in their respective domains. The documentation is provided by the company, initially according to those deemed appropriate to the development process of the two use cases. As the process carried on, there were documents added as more were needed for the assessment of the company’s standards, concerning the chosen requirements and instructions from the selected standards.

7.1.1.1 Selection of the Safety Standards

Four standards have been chosen for this thesis as more would broaden the scope of the thesis to an extent that would cause timing issues, and less would not be enough to gather sufficient assess-ment criteria. IEC 61508:2010 and ISO/IEC/IEEE 29119:2013 are standards for the functional safety of electrical/electronic/programmable electronic safety-related systems and software and systems engineering - software testing respectively. These two standards are both considered to be among the most popular approaches used in the improvement of software test processes [32][4][19]. As the study at hand focuses on assessing test processes, these two standards were deemed ap-propriate. The other two standards used are ISO 13849:2016, parts 1 and 2, and ISO/IEC/IEEE 12207:2017. The two have recommendations concerning the safety of machinery and software li-fecycle processes. ISO 13849 and ISO/IEC/IEEE 12207 are used by the company and are also among the most prevalent standards used by industry to ensure the safety of industrial control systems [33].

7.1.1.2 Selection of Requirements and Instructions

In addition to the selection of the actual standards, the relevant instructions and requirements from the specified standards must also be extracted. As the scope of this thesis focuses on test processes, and assessment of the company’s overall process in terms of verification and validation, not all the requirements from all the standards have been chosen for this assessment. The list below gives an overall view of the limitations considered for removing certain clauses of the standards from the assessment criteria used in this thesis.

Generic Exclusion Criteria

1. Parts of standards must be considered while performing the assessment; however, they are not selected as requirements for assessment. For example, the Conformance clauses are considered in detail, as they are vital for understanding how the compliance of a process with the requirements of a standard shall be assessed; however, such clauses are marked as "not included" in the tables. Conformance clauses explain how to read the standards and do not include instructions or requirements.

2. As the thesis is concerned with specifically the two use-cases, and not the entire organisation, requirements for certain processes do not apply to the scope of this thesis.

3. All stakeholder interactions are ignored in this thesis, since (according to the company) customer requirements are at a very high level, and the company writes the software re-quirements to support the development process. Communications with stakeholders, and between members of the organisation, are not in the scope of this thesis and therefore are not considered in the assessment.