Improving Validation & Verification data management by deploying a life cycle management tool

(1)

Improving Validation & Verification data management by deploying a life cycle management tool

BASTIEN FRANCOMME

KTH ROYAL INSTITUTE OF TECHNOLOGY

SCHOOL OF ELECTRICAL ENGINEERING AND COMPUTER SCIENCE

(2)

(3)

Verification data management by deploying a life cycle

management tool

BASTIEN FRANCOMME

Master in Embedded Systems Date: October 12, 2020 Supervisor: Håkan Lane Examiner: Elena Troubitsyna

School of Electrical Engineering and Computer Science Swedish title: Effektivisering av verifierings- och

valideringsprocessen vid införandet av en applikation för livscykelhantering

(4)

(5)

Abstract

Managing data can be a challenge for large scale projects involving dozens of collaborators with different expertise. Data could be requirements, test proce- dures or product breakdown structures (PBS) for example. The deployment of an Application Life cycle Management (ALM) platform marks the beginning of the transition from the document-centric approach to new object-oriented methods such as Model-Based Systems Engineering (MBSE) to store data.

This thesis evaluates the deployment of Polarion ALM tool to improve validation & verification (V&V ) processes’ time and cost.

An action research methodology has been implemented on the case study of aeronautical projects. A data management prototype has been developed.

It consisted in (a) designing a database structure, (b) designing workflows and processes to handle data, (c) designing reporting views to compute, sort and display information, and (d) developing import/export templates depending on the pre-defined formats.

An anonymous survey addressed to the prototype users showed that the respondents are eager to work with such a tool and predict time and cost saving on the long term for V&V processes. There is indeed no answer with a mean or median lower than 3 over 5 on the satisfaction scale, meaning that the par- ticipants globally agree that the new approach is at least equal or better than the previous methodologies. However, configuration management has been a challenging feature to implement and does not meet all users’ expectations.

Using a more recent version of Polarion ALM tool could enable those needs to be fulfilled.

The assessment has also shown that a successful ALM tool deployment relies on both the implemented data structure and the management of change (also called change management). Including the future users during tool de- velopment facilitates the adoption of the ALM platform. Thus, it necessitates strong interactions with the company collaborators through defined communication channels and a proper training program.

(6)

Sammanfattning

Hantering av data är en utmaning för storskaliga ingenjörsprojekt projekt som involverar dussintals medarbetare med olika expertis och arbetsuppgifter. Ex- empel på data kan vara kravspecifikationer, testförfaranden eller ett projekts uppdelning och struktur. Genom att implementera en Applikation för Livscy- kel Hantering ALH påbörjas en övergång från dokumentbaserad till objekto- rienterad metod för datalagring i modellbaserade ingenjörsprojekt.

I denna avhandling bedöms införandet av Polarions ALH-verktyg för att minska tid och kostnad vid validering & verifiering (V&V ).

Metoden har aktivt testats och utvärderats på ett projekt i flygindustrin. En prototyp för datahantering har utvecklats med hjälp utav Polarions ramverk.

Arbetet med prototypen delades upp i fyra delar, (a) utforma en databasstruk- tur, (b) bestämma arbetsflöden och processer för att samla och hantera data, (c) att utforma rapport-gränssnitt för beräkning, sortering och förmedling utav information och slutligen (d) skapa import-/exportmallar beroende på föregå- ende definierade format.

En anonym undersökning hos prototypanvändarna visade en önskan att arbeta med ett sådant datahanteringssystem och få en prognos över tid och kostnadsbesparingar på lång sikt för V&V-processer. Vid utvärderingen över hur tillfredsställda användarna var med verktyget så var samtliga median och medelvärden minst 3 på en skala av 5, där 5 motsvarar ”högsta” nöjdhetsgrad med verktyget. Testet gjordes för användare globalt och visar på att deltagarna minst tyckte verktyget var likvärdigt med systemet som fanns tidigare men of- tast bättre. Hanteringen av olika konfigurationer har dock varit en utmanande funktionalitet att implementera och uppfyller inte alla användares förväntning- ar. Troligt skulle dessa krav uppfyllas genom att använda en nyare version av Polarion.

Bedömningen har också visat att ett framgångsrikt införande av ALH är beroende av både den implementerade datastrukturen och hanteringen av för- ändringar. Att inkludera framtida användare under verktygsutveckling under- lättar antagandet av ALH-plattformen. Därför krävs det starka interaktioner med företagets medarbetare genom definierade kommunikationskanaler och ett ordentligt träningsprogram.

(7)

List of Figures

2.1 Waterfall Model . . . 7

2.2 V-model from structured systems design [19] . . . 9

2.3 Reviewed Spiral Model by Vogel [18] . . . 10

3.1 Life cycle model implemented in the case study company . . . 27

3.2 Stakeholders of the data management system . . . 29

3.3 Extract from the prototype data structure . . . 33

3.4 Different layers of the data structure . . . 34

3.5 LiveDoc display of customer requirements . . . 35

3.6 Database display of customer requirements . . . 35

3.7 Extract from the System Requirement Workflow . . . 36

3.8 Extract from a Bottom Up Validation Matrix . . . 37

4.1 Status of the reported User Stories . . . 40

4.2 Status of the reported Change Requests . . . 41

4.3 Status of the reported Issues . . . 42

4.4 General Questions Results . . . 45

4.5 Illustration of a top-down and bottom-up tables showing the same information . . . 50

5.1 Example of three activities running simultaneously using three collections . . . 57

viii

(11)

List of Tables

2.1 Waterfall Model with feedbacks Advantages and Drawbacks

[15, 18] . . . 7

2.2 Sashimi Model Advantages and Drawbacks in comparison to Waterfall model [15, 18] . . . 8

2.3 V-Model Advantages and Drawbacks [20] . . . 9

2.4 Iterative Models Advantages and Drawbacks . . . 10

2.5 Agile Models Advantages and Drawbacks [23] . . . 11

2.6 DAL categories [28] . . . 21

4.1 Extract of the most common categories for user stories . . . . 40

4.2 Extract of the most common source for change requests . . . . 41

4.3 Domain expertise data . . . 42

4.4 Answers Value . . . 46

4.5 General questions statistics with all answers . . . 46

4.6 General questions statistics excluding the marginal answer . . 47

4.7 Traceability assessment statistics . . . 48

4.8 V&V assessment statistics . . . 49

4.9 Project monitoring assessment statistics . . . 50

4.10 Configuration Management assessment statistics . . . 52

4.11 Results of the question (Q1): I was able to share my needs and feedback to the development team. . . 52

4.12 Results of the question (Q2): The development team reacted quickly when I needed information. . . 52

4.13 Results of the question (Q3): I’m informed of the updates and new functionalities. . . 53

4.14 Preferred communication channels statistics . . . 53

4.15 Summary of Q1, Q2 and Q3 . . . 54

ix

(12)

(13)

Chapter 1 Introduction

Systems in the aeronautic field are becoming more and more complex, while the need for safety is increasing. Complex aeronautical systems are often composed of software, hardware and physical subsystems. The complexity comes from both the implementation of new technologies and the multiplication of actors in development and maintenance [1].

1.1 The Need of Validation & Verification

In 1994, the Standish Group released a study showing that only 9% of big companies’ software projects were successful¹ that year. The other 91% of projects had cost overruns, time overruns, or were aborted. According to the IT executive managers of those projects, 21% of the failures directly came from the requirements (unclear statements or unrealistic expectations) [2]. Those statistics highlight the need for proper Validation & Verification (V&V) of the requirements.

Barry Boehm defined in 1981 [3]:

• the Validation process as answering the question “Are we building the right product?”;

• the Verification process as answering “Are we building the product right?”.

The two recent Boeing 737 Max crashes (October 2018 and March 2019) caused by an embedded equipment malfunction also revealed that the rising

1In this context, a successful project is a project which implemented all its objectives on time without cost overruns.

1

(14)

complexity in the aeronautic domain challenges the established V&V processes [4]. Aerospace is indeed a field where the safety is essential and the development costs represent tremendous amounts of money. Communication between the different life cycle activities (system specification, V&V, quality, management, production, maintenance, etc.) is essential to minimize errors and development time.

1.2 Appearance of ALM

On large scale projects, the different activities have tend to be separated and highly specialized in order to achieve better efficiency [5]. However, this methodology shows its limits, for example, when changes or reworks occur [6].

The Application Life cycle Management (ALM) approach copes with the separation of the activities. On the contrary, it aims at keeping a strong interaction between all life cycle steps. ALM tools have emerged to provide companies ways of integrating all activities [5]. The tools supply the digital platform to organize, manipulate and share all project data (client data, system specification, V&V data, management of change data (or change management data), etc.).

1.3 Research Question

This master degree thesis aims at proposing a verification data management using Polarion, an Application Life cycle Management Tool (ALM tool), that would lead to improvements during the Validation & Verification (V&V ) pro- cesses. The improvements should be seen as a reduction of the development duration, meaning also a cost reduction.

The hypothesis is that the deployment of an ALM tool enables more clarity, error discovery and automation which leads to improvements in V&V processes’ time and cost. The hypothesis will be confronted with the results of an ALM tool prototype implementation on pilot projects.

• Would the deployment of an ALM tool lead to improvements in V&V processes’ time and cost in the context of the case study?

The case study considers a complex aeronautical critical equipment project.

It is composed of mechanical elements and software components implemented

(15)

on hardware structures. In the aeronautical development context, "critical"

means that equipment failure could lead to the crash of the airplane.

Note that the choice of an ALM tool is not in the scope of this thesis work.

However, section 2.3 introduces some criteria that can be taken into account while selecting an ALM tool. Some alternatives of Polarion are referred to section 2.5 entitled "Previous Work".

1.4 Contribution

The primary contributions of this thesis work to the field of systems engineering are:

• Gather the basic knowledge needed to customize an ALM tool in the aeronautical industrial context (see Chapter 2 )

• Show an example of an ALM tool deployment in a specific industry domain where other data management methods are used

• Expose some difficulties encountered while deploying an ALM tool

1.5 Outline

The thesis is structured as follows. Chapter 2 provides context and the theo- retical background, introducing some relevant aspects of System Engineering.

The method used to answer the question of this thesis is described in Chapter 3.

Chapter 4 presents the results. Finally, Chapter 5 discusses the main findings of this study and relates to the possible impacts in the industrial fields.

(16)

Chapter 2 Background

The background section will provide knowledge on:

• Systems Engineering and examples of development life cycle models,

• some activities performed during the development of an aeronautical product,

• ALM/PLM tools,

• the aeronautical context and its constraints.

This knowledge is necessary to develop the new data management system in the context of this thesis’ case study.

2.1 Systems Engineering

Systems in the aeronautic field are becoming more and more complex while the need for safety is increasing [7]. The complexity comes from both the implementation of new technologies and the multiplication of components or actors [1]. For example, if all components of a given system work indepen- dently, it doesn’t mean that this system will work as a whole. Moreover, the optimum of a system is rarely reached by concatenating the optimum of the subparts.

This increasing complexity has led to the creation of systems engineering division in companies. Schlager K defines in 1956 Systems Engineering (SE) as a combination of management and technical science. He distinguished five steps in systems engineering, which some relates to management decision and others to analysis work [8]:

4

(17)

• Planning: investigate client needs

• Analysis: study the different possible approaches to address the problem

• Optimization: optimizing the chosen approach according to criteria

• Integration: design the solution,

• Evaluation: determine if the solution is satisfying

The INternational Council On Systems Engineering (INCOSE) was cre- ated in 1990 to develop and promote international coordination around Sys- tems Engineering [9]. They define Systems Engineering as: “a transdisci- plinary and integrative approach to enable the successful realization, use, and retirement of engineered systems, using systems principles and concepts, and scientific, technological, and management methods.” [9].

2.1.1 Systems Thinking

Systems thinking philosophy assumes that the world is systemic. It means that everything can be considered as a system with its own structure, behavior and relations at its boundaries. Systems thinking can be defined as a way of grasping complex systems¹. It involves among others: adopting a holistic perspective, defining and questioning system boundaries, its structure and the relations with its environment [11, 12].

System thinking also advocates the need to resist the desire to provide a solution rapidly. It means taking time before delivering a solution in order to spot every detail and future opportunity of the situation [13, 14].

2.1.2 Life Cycle Models

Each system has a life cycle. It is the period over which the product is being designed, developed, implemented, verified, deployed, maintained and retired.

Market retirement concludes this life cycle. Systems Engineering takes into account the whole period in order for the product to be optimized. For instance, the maintenance will be a concern from the very start of the project [15].

Application Life cycle Management (ALM) tools aim at better organizing how each life cycle activity is performed and also how information is shared

1The term “complex” is to be understood in the etymological sense “complexus” (Latin:

what is woven together) [10]. The complexity here is linked to the number of interactions within the system and with the environment.

(18)

between the different activities. It is then essential to understand how the activities relate to each other in order to deploy an ALM tool.

Systems Engineering life cycle models have emerged over the years. Stephens [15] categorized them into different categories²:

• Predictive: All needed features are described at the start of the project.

Then the design arises from the requirements. Finally, the product is developed and delivered.

• Iterative: All features have a temporary description at the start of the project. They are partially designed, implemented and delivered at the first iteration. Then they are refined at each iteration in order to reach full implementation for all functionalities at the last iteration.

• Incremental: At each iteration, only one feature is described, designed, developed and delivered.

• Agile: Few functionalities are described, partially designed, developed and delivered at the first iteration. At each iteration, other functionalities are added and old ones are refined.

The pilot projects considered during the project degree are the development of fuel control systems. The systems are composed of tanks, fuel pumps, a fuel refueling control panel, valves, pipeline and a fuel control calculator [16]. This means that the systems contain mechanical, hardware and software components. Thus, the systems engineering management system studied should comply with the mechanical, hardware and software fields. Moreover, in the aeronautic field, any delivered critical equipment (such as fuel control system) must perfectly comply with client needs, certification and safety requirements [17]. Thus, the models with several iterations do not fit the industrial need of the case study³.

Selecting a life cycle model is a topic under serious debate for many companies as no model is perfect. The objective is then to select the most appropriate one and manage deviations [18]. This project degree doesn’t entail selecting a SE model for the company. However, it is important to analyze the different models to understand the project workflow of the company. More- over, although models other than predictive are discarded, there is still some

2In this context, one iteration is a “mini-project” with a full life cycle.

3Note that if iterative methods do not fit the global project development, they could suit the development of subprojects such as the software design of the fuel control calculator.

(19)

interest in studying them. Interesting features of the non-predictive models could indeed lead to deviations from the predictive model chosen by the company.

The sections below introduce some models with their respective advantages and drawbacks.

Waterfall [Predictive]

The Waterfall model illustrates the ideal situation where each step of the life cycle is performed correctly before starting the next one (see Figure 2.1).

Figure 2.1: Waterfall Model

The Waterfall with feedback model enables to go back to previous stages in case something must be reworked. Then, the waterfall needs to be run through again. Table 2.1 shows some advantages and drawbacks of the Waterfall with feedbacks model.

Advantages Disadvantages

• Simple and intuitive

• Optimal when no rework needed

• Suits to experienced teams

• Easier documentation

• Control points clearly set

• Only very small changes permitted

• Only fits low-risk projects

• Effort must to made to clearly define the requirements

• Testing starts late in the life cycle Table 2.1: Waterfall Model with feedbacks Advantages and Drawbacks [15,

18]

(20)

Sashimi Waterfall [Predictive]

The Sashimi Waterfall model has the same structure as the Waterfall model.

However, the life cycle steps can overlap. For instance, the software design team can work on one portion of requirements that have been defined while the definition of other requirements is still in progress [15]. Table 2.2 shows some advantages and drawbacks of the Sashimi model in comparison to the Waterfall model.

• Time spared as one team can start working before another team has finished the previous life cycle step

• Design-before-implementation principle is still present

• Lack of control: the teams could lose focus as the project is at different stages at the same time

• Configuration management

Table 2.2: Sashimi Model Advantages and Drawbacks in comparison to Waterfall model [15, 18]

V-model [Predictive]

The V-model is an extension of the Waterfall model. The model has a V shape as shown in Figure 2.2 [19]. The left leg is the decomposition leg. It consists in dividing the wanted product into different parts. Those elements will then be decomposed again. The result is a detailed design composed of or- ganized small elements. The right leg is the verification leg. It consists in verifying the product at different stages. First, the verification is performed on the small elements (unit testing). Then the elements are tested as a group.

Finally, the product is tested in its final environment. There is a strong rela- tionship between the development phase and the verification phase as, to each decomposition task, a verification task is associated [15, 20].

(21)

Figure 2.2: V-model from structured systems design [19]

• Simple and intuitive

• Optimal when no rework needed

• The system design can be reviewed before the implementation

• Easier traceability

• Only very small changes permitted

• Only fits low-risk projects

• Effort must to made to clearly define the requirements

Table 2.3: V-Model Advantages and Drawbacks [20]

Spiral Model, DevOps, Rapid Prototyping [Iterative]

The Spiral Model, DevOps and Rapid prototyping are software-oriented mod- els. They suit well in situations where the requirements are not entirely known at start. It is then advantageous to develop a prototype, get feedback on the prototype, refine the requirements and start developing a new prototype. Those models cannot be considered for the case study regarding the drawbacks pre- sented in Table 2.4. However, [18] suggests a Spiral Model with deviations for medical software development without the notion of prototyping. The output of the rotation could be a document. [18] also renamed the quarters: Risk As- sessment & Management, Develop, V&V Test and Plan. The concept where the four activities are performed in a spiral model is interesting for the case study.

(22)

• Clients requirements can change at each iteration

• Full specification is not needed at start

• Fits high-risk projects

• Prototyping is costly for non- software products

• Safety analysis must be done at each iteration

• Certification process cannot start before all functionalities are implemented

• Configuration management is to be taken into account

Table 2.4: Iterative Models Advantages and Drawbacks

Risk Assessment & Management

Development Planning

V&V Testing

Figure 2.3: Reviewed Spiral Model by Vogel [18]

Extreme Programming, Scrum [Agile]

Agile regroups a set of change- and people-oriented methodologies shuch as Extreme Programming or Scrum. It encourages constant change at any stage of the life cycle.

As a result, Agile enables to reduce cost of later found errors [21]. The Chaos Report of 2015, issued by the Standish Group, reveals that the suc- cessrate⁴ of Agile large software projects is about 18% against only 3% for

4In this context, a successful project is a project which implemented all its objectives on time without cost overruns.

(23)

traditional Waterfall large software projects [22].

• Change well supported

• Better communication / Human- oriented methodology

• Overall better project management (cost and delay)

• Lack of documentation

• Not fitting to very large team scat- tered across the world

• Interchangeability hard to achieve when the project involves many fields of studies (SE, V&V, Safety, QA, etc.)

• Configuration management

• Traceability more challenging Table 2.5: Agile Models Advantages and Drawbacks [23]

Despite being born from software considerations, Agile development has spread to the development of physical products. In a study carried in 2017, Ag- ile hardware development has led to benefits rather in "soft factors" (commu- nication, commitment) more than "hard factors" (time, resource spent, quality) [24].

Several key aspects of some Agile process frameworks are not compatible with the case study: little documentation vs. full documentation for certification, small team vs. large team, interchangeability vs. specialization and independence, team predominantly collocated vs. spread across the world, ...

(see Table 2.5). However, the Manifesto for Agile Software Development will be kept in mind in order to extrapolate its principles to the physical aeronautical equipment [25]:

“ Individuals and interactions over processes and tools Working software over comprehensive documentation

Customer collaboration over contract negotiation Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.”

(24)

2.1.3 Object-Oriented Systems Engineering OOSE and Model-Based Systems Engineering MBSE

Object-Oriented Systems Engineering (OOSE) consists in handling any piece of information as an object with attributes. For example, a client requirement could be seen as an object with an ID, a description, a status (covered or not covered), a severity (insignificant or important), etc.

In the traditional system, the information is written in various documents.

The latter are manually reviewed, shared and stored after every update. This process is time-consuming. Moreover, this document-centric approach doesn’t provide any connection between the pieces of information stored in those documents. It is then unclear what the impacts are of an item modification (requirement or design decision change for instance). OOSE addresses this issue by taking into account the relations between objects. It is then easier to capture the influence of any change. When needed, the information can be exported in a document to be delivered to the client or the suppliers [9, 26].

Model-Based Systems Engineering (MBSE) is a way of formalizing sys- tems engineering information based on OOSE. Some decades ago, all the information used to be stored on paper documents (texts, drawings, graphs, etc.).

With the spread of digitalization, those documents are now digitized (spreadsheets or text files). A recent trend aims at moving to digital models using modeling languages such as SysML (Systems Modeling Language) or UML (Unified Modeling Language). Those languages provide standardized graphi- cal notations, semantics or syntaxes.

The advantages of the Model-Based and Object-Oriented approaches:

Cost The maintenance of the centralized database is easier than the mainte- nance of numerous traditional documents.

Consistency A change can immediately be propagated through the central- ized database. There is no need to review each document in search of a potential modification impact.

Communication These approaches involve standardization which makes com- munication between teams easier.

(25)

2.2 Life Cycle Activities

This section introduces the life cycle activities whose data will be considered during this project degree. Various organizations provide guidelines for those activities: EUROCAE and RTCA (DO-178C/DO-254), Information Systems Audit and Control Association ISACA (Capability Maturity Model Integration CMII), etc.

2.2.1 Planning

Planning not only consists in scheduling the activities but also organizing the entire frame of the project. It aims at defining all processes and how information will be shared. Planning is a crucial step. Its outputs will be audited by the certification agencies.

One aspect of planning is defining input and output data structure for each process. Planning can take the form of several documents such as: "Verifica- tion Plan", "Development Plan", "Configuration Management Plan", ... [27, 28].

2.2.2 Requirement Analysis

Requirement Engineering is a field of study which is part of Systems Engineer- ing. It consists of collecting, specifying and decomposing requirements from a given problematic. The output of this phase should be the specifications for the product [3, 29].

One procedure to tackle complex problem is to decompose the main question into numerous smaller questions which are easier to analyze. However, one should make sure that the subquestions cover the whole scope of the main one [10].

(26)

2.2.3 Validation & Verification

The Validation & Verification process (V&V ) consists in validating and then verifying the requirements previously introduced. Barry Boehm defined in 1981 [3]:

• the Validation process as answering the question “Are we building the right product?”;

• the Verification process as answering “Are we building the product right?”.

This process is primordial to achieve the safety objectives of the aeronautical field. Stephens [15] studied the development of a commercial software application in which 85% of the errors were introduced during the analysis and system design phases. However, only 28% of those errors were detected before coding and testing. Moreover, the additional cost and time delays necessary to fix those errors grow exponentially with time for traditional life cycle models. It is therefore in companies’ best interest to invest resources in those early phases as finding errors sooner often results in cost saving [21, 29].

Validating and Verifying covers more than testing, as testing cannot prove the absence of errors. Different methods can be used during the V&V process [26, 30]. There are often referred as IADT (Inspection Analysis Demonstration and Test):

• Inspection: examination using basic senses (sight, touch, smell, hearing, etc.)

• Analysis: examination using calculation, model or simulation

• Demonstration: basic manipulation of the intended functionalities of the system/product (without any specific measurement equipment)

• Test: precise examination with measurement equipment and predefined inputs to collect data which will be analyzed

As Sharma [20] wrote, the V&V process isn’t the final step of a project. It is implemented through the development phase too. It is indeed important to Validate the system specification before implementing it. Verification is also taken into account in the early phases of the project. For instance, a verification mean (ie, a mean to prove that the requirement has been implemented) has to be planned for every requirement during the requirement analysis.

(27)

Validation

Validation is the step where the product specifications are compared to the client and safety requirements. This process should be performed as close as possible to the requirement analysis and before the actual implementation of those product specifications. Competitive projects⁵ are often associated with proper writing of the system specifications. Criteria for an appropriate requirement formulation can be summarized in the SMART acronym: Specific Measurable Achievable Relevant Time-bound. Other criteria could be added to this mnemonic such as Verifiable or Consistency with other requirements [29].

Verification

Verification is the step where the designed product is compared to the requirements. The criteria for the verification are set from the beginning of the project during the Requirement Analysis phase through the verification plan. The verification is performed from the lowest design level up to the final product in its real environment. This step should generate readable data to prove that any requirement is satisfied with an acceptable level of incertitude. A requirement that cannot be verified is not suitable [31].

Each requirement is assigned one or several verification procedures. De- pending on the criticality of the DAL, the activities could have to be performed by a different team from the one which designed the equipment. The traceability between the requirement, the procedures and the related results should be explicit.

The norms require a rigorous verification process which takes much time and money. The automatization of the testing is therefore an asset as it is more reliable than manual testing and cost-efficient [32]. The concept of automated testing is also applicable for physical equipment through test benches.

2.2.4 Configuration Management

Modifications can happen at any stage of the product development (customer requirement change, system architecture change, component change, line code change, ...). Configuration Management is a fundamental process that has the objective of ordering any information. It implies [26–28, 31]:

• Identifying the items which need to be tracked

5in terms of quality, cost and delay

(28)

• managing traceability between the design and the requirements

• managing change: any change, at any stage of the development, is controlled, documented and stored

• tracking of the implications of a change: technical, cost or schedule impacts

• identifying baseline⁶: it is impossible to modify a baseline once it has been frozen. Changes must be performed on the current version which will be frozen into a baseline later.

• identifying, tracking and reporting problems

• archiving all data

Bad configuration management could lead to tragic accidents such as the explosion of ARIANE 5’s flight N°501. A software feature had been reused from the ARIANE 4 program. It was casting a 64-bit floating value in a 16-bit signed integer. It was working perfectly on ARIANE 4. However, ARIANE 5 had more powerful thrusters. Thus, this variable was bigger and could not be cast into a 16-bit signed integer. The change had not been traced up to the casting function. This resulted in inaccurate flight data which lead to the explosion of the rocket, a 1-year delay and a loss of approximately 0.36 Billion US $ [33].

2.3 ALM/PLM Tools for Systems Engineering

Life Cycle Management tools (LCM tools) aim at improving and automating engineering processes to achieve better efficiency and quality throughout the entire life cycle of a product (from the first need statement to the product retirement). A distinction is often shaped between the hardware-oriented Product Life cycle Management (PLM) and its software-oriented counterpart Appli- cation Life cycle Management (ALM). PLM/ALM tools should not be seen as pure products but as a combination of processes, people and tools [34, 35].

The objectives of PLM/ALM tools are [34, 36]:

• Organize engineering data management

• Share engineering data efficiently across teams

6A baseline is a frozen version of a set of items.

(29)

• Automate non-value-added tasks

• Facilitate re-use across projects

• Comply with the domains/companies norms and protocols

Although Systems Engineering should be systematic on every project, the use of SE tools is not mandatory [29]. In this case study, the number of stakeholders and the amount of data generated during the product development make the use of PLM/ALM tools very promising. Over the last decades, several PLM/ALM environments have emerged from either hardware and software backgrounds: Enovia from Dassault, Polarion from Siemens, PTC In- tegrity from 4CAD Group, IBM Rational Rhapsody from IBM, PREEvision from Vector, ...

Although the choice of an ALM tool is not part of this degree, it is interesting to study how tools can be assessed in practice. [37] enumerates some aspects of comparison:

• "Actual costs": individual or floating licenses cost

• "Costs of savings": installing costs, migration costs, formation costs, server maintenance costs, personalization costs, ...

• "Savings by usability and maintenance": benefits while using the tool (decrease of the team’s workload, better communication, etc.)

• "Indirect benefits": benefits that would come from potential future improvements of the tool

• "Importance of traceability": features regarding traceability, connection to other tools in order to implement full traceability from client requirements to lines of code or hardware components

• "Ground for refusal": compliance with non-negotiable company’s rules (for instance, no cloud-based application for security reason)

In this case study, the company has chosen Polarion ALM developed by Siemens. The following section introduces some of Polarion’s features.

(30)

2.3.1 Polarion

Polarion Software was a company founded in 2004 and acquired by Siemens PLM Software in 2016. In 2005, the first version of Polarion ALM was released. It is a 100% browser-based platform for life cycle management with over 2.5 million users [38].

The tool is divided into projects. Each project contains:

Work Items A Work Item is an object with attributes such as a status or a de- scription. It can correspond to any pre-defined type of objects (requirement, test-case, test result, analysis, problem report, etc.). Work Items can be linked with each other through any pre-defined link. For exam- ple, each requirement is a Work Item which can have a status (Working, In Review or Verified) and be linked to a test-case.⁷

Pages A Page corresponds to a pre-defined HTML page where information can be displayed (project information, statistics, graphs, etc.) or/and actions can be done (creation/suppression of Work Items, linking of Work Item, etc.)

Polarion comes with pre-made templates where a template contains pre- defined types of Work Items, links or pages. However, it is possible to customize its own template and therefore create its own data management system based on what Polarion ALM provides. A Software Development Kit (SDK ) is available to provide documentation for Polarion’s APIs (Application Program- ming Interface) and the embedded database schema. The APIs enable in-depth customization with for example the automatization of tasks using scripts or the exportation and layout of data to spreadsheets. The potential improvements studied in this project degree will not come directly from Polarion but from the V&V data management system implemented thanks to Polarion template customizing possibilities.

Some of Polarion’s features that interest us for V&V data management are:

access rights management, centralized database, document generation and vi- sualization, data check-out/check-in, document round-trip, configuration management, data states and workflows [36, 39].

7Polarion is based on an OOSE.

(31)

2.4 Aeronautical Norms

In this section, some norms constraining the development in the aeronautical domain are introduced. They define the framework to which the verification data management should comply to.

2.4.1 Appearance of Regulation Organizations

When the first pioneers such as the Wright Brothers started developing planes, few workers were needed to design the whole machine. For example, the first successful plane, the Flyer, was composed of very basic elements with few suppliers (only for the engine and some instruments) [40]. However nowadays aircraft manufacturers deal with many different challenges involving very different skills and knowledge. Thus, they are helped by hundreds of suppliers for each airplane program [41]. For example, the airbus A380 contains roughly 4 million spare parts supplied by 1500 different companies [42].

During the 20th century, organizations were created to organize the aviation industry through standards and norms. The principal goals are to make the aviation industry competitive and prevent malfunctions from happening.

Among those organizations, this section will focus especially on two organizations:

• EUROCAE, The European Organization for Civil Aviation Equipment, is a non-profit organization created in 1963. It provides guidance for aeronautical equipment development and maintenance [43]. Its Amer- ican counterpart is the RTCA, Radio Technical Commission for Aero- nautics.

• EASA, European Union Aviation Safety Agency, is a European agency created in 2002. It is responsible for certification, regulation, and standardization. It mainly focuses on safety and environment protection [44]. The certification of the EASA is mandatory for every newly designed aircraft or airline company in order to enter the European skies [45]. It uses some EUROCAE documents as standards or means of com- pliance with technical standards. Its American counterpart is the FAA, Federal Aviation Administration⁸.

8In this section, only European and American organizations have been introduced. How- ever, there are counterparts for other regions of the world.

(32)

The Certification Specifications (CS) depend on the size of the aircraft in development (Sailplane, Rotorcraft, Large Aeroplanes, Hot Air Balloons, etc.) [44]. The case study of this master thesis will focus on Large Aeroplanes, especially embedded equipment of a Large Aeroplane. Thus, the CS to which, the product development needs to comply with is the CS-25 (JAR-25 for the FAA) [45]. The article 1309 named “Equipment, systems and installations”

concerns especially the system providers. It states, among other requirements, that failures should be prevented according to their hazardousness [17].

2.4.2 Aeronautical Guidance Documents

The industrial context applied to this master thesis is both hardware/mechanical and software. There are numerous safety and quality requirements applying for development processes, software development, hardware development, both software/hardware development and many other fields. Those norms are usu- ally accompanied by Acceptable Means of Compliance (AMC). AMC define rules and processes which, if performed well, would be sufficient to state that the norm has been enforced [45].

One objective of this degree project is to automate the generation of documents which serve as proofs that the AMC has been implemented. Among those AMC, this section will enlighten one general Guidance document (ED- 79/ARP4754), one dealing with software development (ED-12C/DO-178C) and one dealing with hardware development (ED-80/DO-254).

2.4.3 ED-79/ARP4754, Guidelines for Development of Civil Aircraft and Systems

This document is issued by SAE International, previously known as Society of Automotive Engineers, jointly with EUROCAE. It provides guidelines on the aircraft development cycle (including requirement analysis and V&V) [46].

This document has a global perspective while the other documents described in this section are focused on either software or hardware development.

The system requirements arise from the system operational requirement (for example, the system requirement “the plane shall have fuel tanks“ is de- rived from the operational requirement “the plane shall have a mean of provid- ing fuel to the engine throughout the flight”). In parallel, safety requirements are developed from the system safety assurance process. For each of those requirements, a verification mean should be stated. The objective of the verification process is to prove that the system will perform its functions under

(33)

any foreseeable operation condition [47].

A Design Assurance Level (DAL) or Item Development Assurance Level (IDAL), is assigned to each functionality representing its potential contribution to aircraft system failures. The V&V process is highly dependent on this level of criticality. The more critical the level is, the more complex the V&V process is. This DAL is decomposed into 5 categories (see Table 2.6) [28, 47].

Level Category Description

A Catastrophic Failure may cause multiple casualties (or a crash), a loss of critical functions

B Hazardous Failure may cause serious or fatal injuries to a rela- tively small number of occupants, a significant reduction of safety margins, a significant increase of flight crew workload or physical distress for some occupants C Major Failure may cause injuries to some of the occupants, a significant reduction of safety margins, a significant increase of flight crew workload or physical distress for some occupants

D Minor Failure may cause discomfort, a slight reduction of safety margins or a slight increase in flight crew workload

E No Effect Failure that would have no effect on safety margins, nor on flight crew workload

Table 2.6: DAL categories [28]

For example, the dysfunction of a passenger TV screen may be character- ized as a DAL D or E failure as it could cause a slight increase in flight crew workload. However, a dysfunction of the navigation system may be character- ized as a DAL A or B failure as it will reduce the capability of the aircraft to fly.

2.4.4 ED-12C/DO-178C, Software Consideration in Air- borne Systems and Equipment Certification

The purpose of the document is to provide guidance to satisfy certification requirements of aeronautical software equipment. It has been recognized by the

(34)

EASA and the FAA as an acceptable mean of compliance to the airworthiness regulations.

This document describes some objectives for software life cycle processes, activities that satisfy those objectives and pieces of evidence which indicate that the objectives have been met.

The DO-178C associates severe constraints on DAL from A to D. The document lists a set of tasks. To each DAL is assigned a portion of those tasks.

The higher the DAL is, the more numerous the assigned tasks are. Moreover, the higher the DAL is, the more independent those V&V processes must be. It means that for high DAL, the V&V process must be independent of the development task. It often results in the V&V process being performed by someone that has not participated in the designing task. The guide also states data that should be accessible concerning, for example, test cases, test procedures, test results or traceability matrices [28].

2.4.5 ED-80/DO-254, Design Assurance Guidance for Airborne Electronic Hardware

This guide applies to hardware equipment. It is similar to its software counterpart, the DO-178. Hardware life cycle, Planning, Design and V&V processes are described. Following this guidance would ensure the system supplier that its product meets the airworthiness requirements [27, 48].

The example of those AMC enlightens that the V&V processes are implemented to guarantee safety through complying with international aeronautical norms⁹.

2.5 Previous Work

Ebert [34] introduced the general principles of a successful PLM/ALM tool integration based on the case study of an automotive company using Vector’s PREEvision PLM solution. He found improvements in quality, time, flexi- bility, communication and skills management. However, he concluded that

9The point of view of a system supplier will be taken during this project degree. Thus, the clients (the aircraft manufacturers) can also impose their own internal norms in addition to the one previously introduced.

(35)

PLM/ALM tools are not a magical solution that could codify every methodology. Moreover, they do not replace the usual personal contact as a way of sharing knowledge.

Segonds, Mantelet, Nelson, and Gaillard [36] have studied the use of PLM tools from the textile industry perspective. They used interviews and surveys to identify the data that needs to be managed. Then, they contributed to a personalized PLM tool prototype. Their study showed that 96% of the test users would adhere to such a tool. They concluded that it is interesting to implement a PLM tool solution in the textile context based on user feedback.

Kääriäinen and Välimäki [49] report the results of an ALM methodology deployment on an automation industry case study. The adoption of an ALM solution has shown several improvements in communication between team members, traceability of system artefacts and reporting. Their evaluation was based on questionnaires and semi-structured interviews.

Taylor and Panella [50] studied the "parameterization of requirements" using IBM Rational Doors, a feature of the IBM Jazz environment. The objective is to prevent conflicting parameter definitions and facilitate the re-use of previous requirements definition, validation and verification. Their case study is a flap actuation system that is part of the aircraft’s secondary flight control systems. In this study, the use of a PLM tool has enabled parameterization which led to benefits in schedule and costs, requirements robustness, change management and information consistency.

Deuter, Otte, Ebert, and Possel-Dölken [51] used an industrial case study to map the different life cycle activities to either an ALM tool (Polarion ALM) or a PLM tool (Teamcenter). The general system, software, hardware and mechanical requirements are handled with the ALM tool. Only the software design and implementation are implemented in the ALM tool. The hardware/mechanical design and implementation are completed using the PLM tool. Finally, the V&V is entirely performed using the ALM tool. This task allocation to the ALM tool is actually what the company in this case study intends to do.

Tüzün, Tekinerdogan, Macit, and İnce [5] have conducted a 7-year action research study on the impacts of transitioning to ALM platform for large companies. They provide guidelines for selecting an ALM tool and also discussed critical success factors to adopt the new platform. Their empirical data shows

(36)

that the adoption of an ALM approach has led to better quality management and the reduction of development time and cost. They conclude that transitioning to an ALM approach is a challenging process relying on many factors.

Jandl [39] carried a comparison in 2012 between three ALM tools: Polar- ion ALM 2012, HP ALM 11 and IBM Rational Concert Enterprise 3.0. The comparison was based on features such as requirements management, change management, test management, task management, build management or re- lease management. The author seems¹⁰ to conclude that IBM Rational has better results, followed by Polarion and finally HP ALM.

This comparison only took into account the features currently implemented in 2012. The tools have been updated between 2012 and 2020. Thus, it could be interesting to refresh this comparison. Moreover, the choice of an ALM tool is also influenced by other criteria (see Section 2.3).

2.6 Reference Criticism

The references presented in the background chapter come either from books, articles, organization’s official websites or scientific websites. Although each source has been chosen for its accuracy and objectiveness, the sections bellow introduces possible reference criticisms.

2.6.1 Software References

Despite Systems Engineering having appeared before the spread of software development, it emerged with the need of software methodologies. This lead to the creating of the term "Software Engineering". This explains why the majority of the references relate to software development [52].

Agile is for example a concept born from software development that is not easily applicable to hardware development. However, as Graves [53] wrote, it is possible to extrapolate some Agile concepts to hardware development.

Striving to apply the Agile Manifesto [25] through this project is an example of extrapolation.

2.6.2 Project Success Figures

Each year, the Chaos Report issued by the Standish Group is widely cited by scientists and media. It often depicts a catastrophic situation for the software

10Translation algorithms have been used to translate the document from Czech to English.

(37)

project’s success rate. This report has been referred to in this section of the project degree to compare the traditional and Agile methods. However, those figures bear strong criticism.

Eveleens and Verhoef [54] wrote that the Chaos Report data are "mislead- ing, one-sided, pervert the estimation practice, and result in meaningless figures". Their definition of success is indeed questionable. According to the Standish Group [2], the success of a project depends on how well they fit the initial features, cost and time estimations. Thus, the success rate measures deviations from estimations. It doesn’t take into account user satisfaction or the context for instance. The background plays an important role as for some project a 2% cost overrun is insignificant. Whereas for others it would greatly harm the project success. Moreover, the initial estimations are biased as some companies set larger safety margins than others [54].

Those figures should then not be seen as reflecting the authentic reality; but as illustrating a global trend in software development. This trend can also be seen in the study of Ambler [55] (independent study from the Chaos Report).

2.6.3 PLM/ALM Research

In 2016 Deuter and Rizzo [35] highlighted the differences between ALM and PLM life cycle models. Those two life cycles should converge to implement an efficient PLM/ALM tool. They pointed out the mismatch between academia research and industrial application. They indeed conducted a literature study in four scientific databases (IEEE Xplore Digital Library, ACM Digital Li- brary, Springer Digital Library and ScienceDirect). Only 11 publications were considered as relevant according to their criteria which would indicate that ALM/PLM convergence is hardly covered by academic research. Therefore, they encourage academic research activities in these fields and better collaboration between companies and research teams.

The ALM/PLM convergence concerns the case study as the fuel control systems are composed of hardware/mechanical elements controlled by an embedded computer unit. However, as the scope of the degree is limited to the V&V data management (no technical schemas or blueprints, no production data for example), the LCM tool could be considered as an ALM tool.

(38)

Chapter 3 Methods

The problematic of improving V&V data management will be answered through the development of a data management tool prototype in the context of an Ac- tion Research methodology.

3.1 Action Research

Action Research is an empirical research methodology focusing on problem- solving and action-taking. This term was introduced in 1944 by Kurt Lewin. It has been widely used in social science, organization development research and lately in software engineering research. Action Research consists in analyzing a situation, then applying changes and finally observing and reflecting on the outputs [56].

[5] decomposed Action Research in five steps which could be iterated if necessary:

1. diagnosis - identify the problems and the context (see Section 3.2) 2. action planning - plan actions and expected results (see Section 3.3) 3. action taking - execute actions (see Section 3.4)

4. evaluating - compare the results and the expected results (see Sections 4.1 and 4.2)

5. specify the learning - reflect on the results and possible future actions (see Section 5)

26

(39)

3.2 Initial Diagnosis

The considered case study is the development of critical aeronautical equipment. One project could involve up to 100 persons who perform various dis- tinct activities but have a strong need for synchronization between themselves.

The company’s motivation for an ALM tool is to reduce time and cost for product development while keeping a high level of safety.

3.2.1 Process Perspective

The V-model is the life cycle model that is the most comparable to what the case study company has implemented (see Section 2.1.2). Each activity is performed and checked before starting the next one. Section 2.2 lists some activities that concern this case study. However, due to the size of the projects, it is possible that some portions of the project progress faster than others. Thus, some activities can overlap as illustrated in the Sashimi Waterfall model (see Section 2.1.2). Figure 3.1 illustrates the development life cycle implemented by the company. It is then essential to track progress for all the different parts of the project. The danger of this situation is to re-do tasks that have already been done which leads to loss of time and money.

Validation Customer needs

Capture System Specification

Architecture design

Subsystems specification

Implementation

Subsystems testing

System integration testing

System testing

Verification

Figure 3.1: Life cycle model implemented in the case study company

(40)

Figure 3.1 also depicts how the V&V activity interacts with the specifications activities (left branch of the V-model). Each specification activity is validated within its scope and also in regard to the previous activity to ensure consistency. The specification can also simulated to validate whether it will satisfy the customer needs or not. Finally, each element of the implemented product is verified and compared to the specification sections it relates to.

3.2.2 Tool Perspective

Each team has its own specific tools to create, manage, trace, modify and share data. Information can be seen as a set of system artefacts where a system artefact is a specific object containing information. For instance, it could be any type of requirements, analyses, reports or feedback. Most of the system artefacts are stored in documents: spreadsheets, text documents or notes.

Tools other than word processing or spreadsheets software programs are handled. However, they are specialized for one activity. Thus, they cannot be spread and used across teams. No matter how information is stored, the preferred format to share and exchange data within the company or with clients is word document or spreadsheet.

3.2.3 Stakeholders

Different stakeholders have been identified (see Figure 3.2). They interact with the V&V data management system in various ways:

Customer, Supplier and Certification organizations

• Provide specifications

• Receive project information Requirement specification teams

• View project information and system artefacts

• Write/Rework system artefacts

• Trace system artefacts

• Receive feedback

(41)

Data Management System Customer,

Supplier and Certification Organizations

Requirement Specification

teams

Safety teams Projet Managers

V&V teams

Figure 3.2: Stakeholders of the data management system

V&V teams

• Check system artefact on several criteria

• Validate/Invalidate system artefacts

• Report feedback Safety teams

• Write/Rework system artefacts

• Check system artefact on several criteria

• Report feedback Project Managers

• Manage the data management system

(42)

3.3 Prototype

The case study company has chosen to regroup some activities in one ALM tool. A compromise has been done between having several highly specialized tools for each activity and using a more general tool that would cover the whole life cycle of the project.

The conclusions of a previous ALM tool deployment advice a strong interaction with some key team members while developing the major functionalities. If they experience improvements using the prototype on pilot projects, they could become ambassadors and substantially help in later change management to adopt the tool [5]. Thus, the study will focus on a tool prototype development and its evaluation by some key team collaborators.

The selected ALM tool for this study is Polarion ALM 18.3 running on a Windows 2012 server (see Section 2.3.1). The web-based tool has been accessed via Mozilla Firefox, Microsoft Edge and Chrome. Polarion ALM 19.2 was also accessible on another server to test the migration of data between tool versions. The selection results of a tools comparison previously done by the case study company. The criteria included features implementation, server installation, license cost, security and 30-year support. An example of ALM tools comparison has been published in [39] (see Section 2.5).

3.3.1 Prototype objectives

[5] identified three factors that could be influenced by the adoption of an ALM approach: traceability, visibility and process automation. Those elements represent the principal levers for action to achieve the case study objectives.

The basic features of a tool are often not sufficient for company needs.

Templates, new functionalities and processes must be established based on tool basic features and API. Thus, the prototype consists in the customization of the ALM tool to fit V&V data management needs in the case study specific context. This involves:

• implementing a database structure for the system artefacts

• implementing system artefact workflows

• designing views to visualize information within the tool

• designing import/export templates to share information

(43)

3.3.2 Prototype Deployment method

Literature advocates strong interaction with prototype users to get frequent feedback [5]. Thus, the chosen deployment method is Agile (see Table ??

in Section 2.1.2). It follows a scrum-like method where sprints are delivered monthly. Each sprint implements new features and refines features previously implemented. Three sprints are planned for this study. The first one focuses on the implementation of the data structure and the validation process. The sec- ond sprint focuses on the verification processes. Finally, the last one focuses on improving already existing features and writing prototype documentation.

Transitioning from one tool to another induces major changes in the company’s ways of working. Thus, involving team members throughout the prototype development is essential as people are more likely to accept change if they are involved [5]. The communication channels are:

• On-site presence to enable direct communication

• Weekly meetings with the different teams

• A user support platform where prototype users can report any issue, need for improvements on implemented features or needs for new features It is also important to open a dialog with the tool provider to discuss the indirect benefits which will come with future tool updates [37].

3.3.3 Prototype Evaluation method

The Action Research Method evaluation aims at comparing the initial state and the final state after applying changes. The comparison of projects’ Key Performance Indicators (KPIs) before and after the changes is an efficient way of evaluating a change. In our context, the KPIs could be:

• time and cost to perform the validation and verification processes

• number of V&V loops¹to validate and verify the requirement

• number of artefacts validated and verified at the first loop

• time needed to perform a V&V loop

1A loop is a cycle composed of the writing/correction of a system artefact and its validation/verification. For example, if an artefact is written, invalidated, reworked and then validated, two loops were needed to validate the artefact.

(44)

• traceability related criteria success rate

In section 2.6.2, the relevance of project data comparison has been associated with the comparison of the project context and company’s ways of working. Thus, the project data comparison should be limited within the case study company. However, the KPIs listed above necessitate the full development of a project using the prototype. This was not achievable within the time constraints of this study. The comparison of quantitative KPIs data is then shelved.

Hence the prototype evaluation is based on a qualitative comparison through a survey addressed to the company’s key team members. Similar previous studies also used this evaluation method for ALM tool deployments in the textile and automation industry context. The questions aimed at comparing their former methodology to the newly adopted ALM approach [36, 49]. Using such an ALM tool does not directly generate income for the company. In- stead, if correctly integrated with teams’ ways of working, it facilitates some processes which lead to better clarity, error discovery or automation. Then, this results in cost and time savings during the development phase. Thus, the assessment focuses on user experience to evaluate the improvements in some functionalities.

The survey respondents should represent at least the requirement specification and V&V teams, which are the two main categories of the V&V data management stakeholders (see Section 3.2.3). The survey emphasizes how they perceived this change and how do they expect the quantitative KPIs to evolve. The accuracy of the data is supported by the anonymity of the survey.

The user support statistics could also reflect some aspects of the prototype evaluation. It indeed contains a part of the company teams’ expectations (user stories and change requests) and bug reports (issues). The user support platform also tracks the progress of the user stories implementation or the issues resolution.

3.4 Prototype Implementation

In practice, the tool customization consists in implementing several elements.

They are listed in the sections below.

Improving Validation & Verification data management by deploying a life cycle management tool