TOOL DEVELOPMENT FOR TEST OPTIMIZATION PURPOSES

Västerås, Sweden

Thesis for the Degree of Master of Science (60 credits) in Computer

Science with Specialization in Software Engineering 30.0 hp DVA424

TOOL DEVELOPMENT FOR TEST

OPTIMIZATION PURPOSES

Gezim Cako

Examiner: Wasif Afzal

Mälardalen University, Västerås, Sweden

Supervisors: Sahar Tahvili, Leo Hatvani

Mälardalen University, Västerås, Sweden

Company supervisor: Ola Sellin

Alstom Company

which pays off in detecting flaws and defects, alternatively leading to high-quality products. Generally, software testing is performed manually by a human operator or automatically. While many test cases are written and executed, the testing process checks if all the re-quirements are covered, and the system exhibits the expected behavior. A great portion of the cost and time of the software development is spent on testing; therefore, considering the type of the software, test optimization is needed and presented as a solution in cost efficiency and time-saving.

Aim: This thesis aims to propose and evaluate the improved sOrTES+ tool for test opti-mization purposes, consisting of selection, prioritization, and scheduling of the test cases integrated into a dynamic user interface.

Method: In this thesis, test optimization is addressed in two aspects, low-level

require-ments and high-level requirerequire-ments. Our solution analyzes these requirerequire-ments to detect the dependencies between test cases. Thus, we propose sOrTES+_{, a tool that uses three}

different scheduling techniques: Greedy, Greedy DO(direct output), and Greedy TO(total output) for test optimization. The mentioned techniques are integrated into a dynamic user interface that allows testers to manage their projects, see useful information about test cases and requirements, store the executed test cases while scheduling the remaining ones for execution, and also switch between the mentioned scheduling techniques regarding the project requirements. Finally, we demonstrated its applicability and compared our tool with existing testing techniques used by our industrial partner, Alstom company, evaluating the efficiency in terms of requirement coverage and troubleshooting time.

Results: Our comparison shows that our solution improves the requirement coverage,

increasing it by26.4% while decreasing the troubleshooting time by 6%.

Conclusion: Based on our results, we conclude that our proposed tool, sOrTES+, can be used for test optimization and it performs more efficiently than the existing methods used by industrial partner Alstom company.

utveckling, som lönar sig vid upptäckt av brister och defekter, alternativt leder till hög-kvalitativa produkter. Generellt utförs programvarutestning manuellt eller automatiskt. Medan många testfall skrivs och utförs kontrollerar denna process om alla krav täcks och systemet visar det förväntade beteendet. Med hänsyn till olika faktorer som produkttyp och komplexitet behövs testoptimering och presenteras som en lösning för kostnads- och tidsbesparing.

Mål: Den här rapporten syftar till att utvärdera det förbättrade sOrTES +verktyge för testoptimeringsändamål, bestående av urval, prioritering och schemaläggning av testfall integrerade i ett dynamiskt användargränssnitt.

Metod: I den här rapporten behandlar vi problemet med testoptimering i två aspekter,

lågnivåkrav och högnivåkrav. Vår lösning består av att analysera dessa krav för att upptäcka beroendet mellan testfall. Således föreslår vi sOrTES+, ett verktyg som använder tre olika tekniker: girig, girig DO och girig TO för testoptimering. De nämnda teknikerna är integrerade i ett dynamiskt användargränssnitt som gör det möjligt för testare att ladda upp nya projekt, lagra det utförda testfallet medan de ordnar om de återstående, välj också mellan de nämnda teknikerna med avseende på kraven som krävs för att tillgodose. Slutligen demonstrerade vi dess användbarhet och jämförde vårt verktyg med befintliga testtekniker som användes av vår industripartner och utvärderade effektiviteten när det gäller kravtäckning och felsökningstid.

Resultat: Jämförelsen visar att vår lösning förbättrar kravtäckningen och ökar den med

26.4 % samtidigt som felsökningstiden minskas med 6%.

Slutsats: Baserat på vårt arbete drar vi slutsatsen att vår lösning, det föreslagna verktyget

sOrTES+ kan användas för testoptimering, och det fungerar mer effektivt än de befintliga metoderna som används i vår industripartner Alstom företag.

of Athanasios Stratis, Henrik Jonsson, Ola Sellin by the Alstom company.

I would like to thank my supervisors Sahar Tahvili, Leo Hatvani, and my team partner Matijs Daemen for their invaluable advice, continuous support. Their immense knowledge and abundant experience have encouraged me in all the time of my academic research.

Finally, I would like to express my gratitude to my family for their encouragement, understanding, and financial support during my studies.

1 Introduction . . . 1

1.1 Problem Formulation . . . 2

1.2 Thesis Overview . . . 3

2 Background and Motivations . . . 5

2.1 Software Development Life Cycle . . . 5

2.1.1 Planning Phase . . . 5 2.1.2 Analysing Phase . . . 5 2.1.3 Design Phase . . . 6 2.1.4 Implementation Phase . . . 6 2.1.5 Testing Phase . . . 6 2.1.6 Deployment Phase . . . 6 2.1.7 Maintenance Phase . . . 7

2.2 Current Testing Methods . . . 7

2.3 sOrTES . . . 7

2.4 Test Optimization . . . 8

2.4.1 Test Case Selection . . . 8

2.4.2 Test Case Prioritization . . . 9

2.4.3 Test Case Scheduling . . . 9

2.4.4 Multiple-Criteria Decision-Making . . . 9

2.4.5 Requirement Coverage . . . 9

2.5 Integration Testing . . . 10

2.5.1 V-Model . . . 10

2.6 Test Case Dependencies . . . 11

2.6.1 Functional Dependency . . . 11

2.6.2 Abstract Dependency . . . 11

2.6.3 Temporal Dependency . . . 11

2.6.4 Causal Dependency . . . 12

3 Related Work . . . 13

3.1 sOrTES and Our Work sOrTES+ _{. . . 16}

4 Method . . . 17

4.1 Research Process . . . 17

5 Extraction Details . . . 19

5.1 Proposed Solution . . . 19

5.2 Requirement Dependency . . . 21

5.3 Test Case Dependency . . . 21

5.4 Requirement Extraction . . . 22

5.5 Test Case Extraction . . . 22

5.6 Mapping Test Cases and Requirements . . . 22

5.6.1 Requirement Dependency Detection . . . 23

5.7 Requirement Coverage Measurement . . . 24 5.8 Dependency Degree . . . 25 6 Visualization . . . 26 6.1 Front-end Project . . . 26 6.1.1 Project Structure . . . 26 6.1.2 Main Components . . . 28 6.2 Scheduling Techniques . . . 35 6.3 Back-end Project . . . 37 6.3.1 Load Information . . . 37 6.3.2 Create Project . . . 38

6.3.3 Execute Test Case . . . 38

6.4 Interactions with the Supervisors . . . 38

7 Result . . . 40

7.1 Test Case . . . 40

7.2 Requirements . . . 40

7.3 Visualization Chart . . . 41

7.4 Empirical Evaluation . . . 42

7.4.1 Unit of Analysis and Procedure . . . 42

8 Performance Comparison Between sOrTES+_{and The Current Test execution} Order at Alstom . . . 43

8.1 Performance Comparison . . . 43

8.2 Comparison between 1000 Random Cycles . . . 45

9 Validity Threats . . . 47

10 Ethical, legal, and social considerations . . . 49

11 Discussion . . . 50

11.1 Back-end and Scalability Concern . . . 50

11.2 Project Management . . . 50

11.3 Scheduling and Further Optimizations . . . 50

2.1 Software Development Life-cycle [18]. . . 5

2.2 The structure of sOrTES workflow (see [3]). . . 8

2.3 The V-model presentation of software development steps [6] . . . 10

4.1 The research process followed for conducting this master thesis. . . 17

5.1 The low-level and high-level requirements, software application, and the corresponding test cases in the Singapore R151 project. . . 19

5.2 The high-level requirements dependency. . . 20

5.3 The dependency between requirements. . . 21

5.4 The dependency between test cases. . . 22

6.1 React project structure. . . 27

6.2 React project src folder. . . 27

6.3 Project manager component. . . 28

6.4 General information component. . . 29

6.5 General information component - graph tab. . . 30

6.6 General information component - test cases statistics. . . 31

6.7 General information component - requirements statistics. . . 31

6.8 Test cases table component - all test cases tab. . . 32

6.9 Expanding test case rows . . . 33

6.10 A test case with not inputs or outputs. . . 33

6.11 A dependent test case with dependent test cases and outputs. . . 33

6.12 Test case table component-executed test cases tab. . . 34

6.13 Graph representation of a test case. . . 35

6.14 Switching between scheduling algorithms. . . 35

6.15 Test cases scheduled based on coverage . . . 36

6.16 Scheduling based on coverage and total outputs. . . 37

7.1 Test cases, requirements, and their connection visualization graph. . . 41

7.2 Single requirement and test case captured from the main graph. . . 42

8.1 The cumulative requirements coverage comparison between the Greedy scheduling in sOrTES+ _{and Alstom for execution cycle B. . . 45}

8.2 The cumulative requirements coverage comparison between the Greedy scheduling provided by sOrTES+ _{and 1000 random schedules for a set of} 691 test cases from Singapore R151 project. . . 46

5.1 The "Requires" and "Provides" links of two SyRS. . . 21 5.2 The SyRS and test cases extracted by the traceability matrix. . . 21 8.1 The test case ranking regarding the particular ordering provided by sOrTES+

1

Introduction

In the recent decade, software testing has gained increasing attention in the process of software development. The main goal of testing is to reduce the number of possible faults in the software. The testing process can be implemented in the different stages of software development, such as planning, designing, building, testing, deployment, and maintenance; therefore, it is able to ensure the quality of both the development process and the finished product [1]. In order to have an efficient and productive software testing process, developers are required to take many contributing factors into consideration. Selecting the best element or criterion to have the lowest cost value is called test optimization [2],[3]. The most crucial factor in test optimization is to estimate the total cost of testing a software product which justifies its inclusion. A careful analysis has revealed that the cost of software testing is usually from 15% to 25% of the entire project budget [4], [5]. Many companies estimate the amount of fee allocated to measure and test the software’s quality, corrections, and re-work, which is also fundamental and a risk worth taking. According to the actual scenarios, developers can apply various testing approaches such as test automation, test kit minimization, trial random selection, priority, and test scheduling to solve the raised issues [6]. However, these approaches cannot be applied to all testing facilities within the industry [3]. The cost of testing software is classified into two categories, fixed costs, and variable costs [1]. The fixed costs may include testing tools automation, payroll testing, and testing environments [4]. On the other hand, the variable costs may consist of the efforts to resolve possible issues and re-execution of failed tasks [3]. Fixed testing costs may be managed by optimizing organization’s approach towards development process while minimizing variable test costs heavily relies on increasing software testing efficiency.

Moreover, a poorly designed software testing can result in inefficient use of resources like redundant executions due to unnecessarily generated test cases (TCs) [3]. As long as there is evidence to support that testing reduces the faults and maintenance costs of the development, it is reasonable to implement software testing [4]. Also, it is possible to manage the cost of the testing process by optimizing its efficiency.

Testing is a significant step in the software development life-cycle (SDLC). This process may be applied manually, semi-automatically, or fully automatically [1]. The essential techniques in this regard are test case selections, prioritization and scheduling, and also test automation [3]. The test optimization process can be performed via employing different algorithms, which follow a number of necessary steps based on the software requirements. In this thesis, we optimize the testing process by customizing the aforementioned methods while considering our own requirements and limitations [2] [7] [8].

Suppose a developer is considering the possibility of upgrading the testing process from manual to automated; in that case, he/she should first understand that sometimes this might not be the most optimal solution to detect defects [9] and also to decrease the total cost [10]. Some researches show that manual testing is efficient in testing safety-critical systems [3]. However, the dependencies (e.g. functional, temporal, abstract, and causal) between test cases are usually more complicated at the integration testing level [3]. In order to optimize a software testing, the mentioned dependencies should be considered in the process [3], [5].

This thesis utilizes and measures the requirement coverage of the test cases in order to optimize the testing process using various scheduling techniques. The objective is to implement and evaluate sOrTES+_{, a tool that improves the performance of testing}

stages. The motivation behind developing sOrTES+ _{and applying it to other projects}

the data and the dependencies between test cases more clearly, allows users to manage their projects, and understandably shows test execution rankings for the tester to execute them accordingly. The successfully executed test cases are stored, and the rest are rescheduled. Furthermore, by reducing troubleshooting efforts and increasing the requirement coverage, sOrTES+ _{is able to decreases the costs, optimizes the}

ex-ecution time and use of resources, and identifies systems bugs early on in the testing process. The following contributions are made in this thesis:

1. Reviewing state-of-the-art in the domain of software testing, natural language pro-cessing, and artificial intelligence;

2. Conducting an industrial case study at Alstom company by finding the dependencies between requirements at higher-level (SyRS) and lower-level (SRS) and test cases in an ongoing project named Singapore R151.

3. Creating and building a dynamic user interface which enables the user to interact with test cases (e.g. executing or scheduling them according to project requirements) and see the results in real-time.

4. sOrTES+ _{integrates dynamic scheduling techniques by saving the execution state}

and storing the executed test cases.

5. Compared to the current test execution method at Alstom company, sOrTES+

achieves better performance in terms of reducing the troubleshooting by 6%, and increasing the requirement coverage by 26.4%.

This project is conducted in three major phases. The first phase, data extraction, is done in collaboration with Mathijs Daemen and the data is used throughout both studies. The second phase of sOrTES+ _{visualizes the data in a dynamic user-interface and}

incorporates two of the scheduling algorithms designed and developed by Mathijs Daemen, respectively Greedy DO and Greedy TO. The third phase of this project, evaluation, is conducted independently by Gezim Cako and does not rely on Greedy DO and Greedy TO scheduling algorithms.

1.1 Problem Formulation

Testing is an indispensable process and lately has attracted the attention of researchers concentrating on proposing different measures, models, and tools to optimize the testing process [1],[11]. Performing manual testing requires more time and is usually more inefficient in terms of fault detection. Considering this fact, several authors have researched test optimization domains such as test case specification, scheduling, and prioritizing test cases. Optimizing in terms of applying new algorithms for the aforementioned concept might lead the companies to better results and reduce the current testing costs [3]. Furthermore, automated testing is introduced to increase the efficiency of the testing process by using software tools that execute several test cases in a short interval by a scheduled plan. Moreover, data selection, prioritization, and scheduling are conducted by the tool while human intervention is not demanded, at least during the dependency detection phase. Literature and sOrTES tool, previously proposed by Tahvili et al [3], showed that automated testing not only contributed to increasing the effectiveness of software testing, but also raised the rate of execution and reduced the manual testing error percentage. Considering the advantages of automated testing and its benefits in terms of cost and efficiency, there

are some restrictions that suggest automated testing tools are unreliable to use, especially in a high-risk critical issues when the human judgment is more reasonable [3].

Researchers have identified several dependencies such as functional dependency, tem-poral dependency, abstract dependency, and causal dependency [3], [12]–[16]. Several researchers have proposed different methods for detecting dependencies to avoid testing complexity. However, different systems have various dependencies that need to be detected and considered in order for the system to operate smoothly. There are cases where some dependencies are complex to test and cause problems for the system testing in terms of costs and delays. The testing cannot be performed in an effective process if we do not analyze the dependency issues. In this regard, the best alternative is detecting the dependencies and scheduling them to satisfy the requirement coverage in a short interval of time.

The aim of this thesis is to address the dependencies between test cases in lower and higher requirement level. This task is achieved by finding dependencies between the test cases and ranking them regarding the new scheduling method in a dynamic user-friendly interface. Our proposal consists of improving and developing a python-based tool currently being used and has shown valuable results in Bombardier Transportation (BT). The main idea is to customize the tool using Project Singapore R151 while being adaptable to other ongoing projects at the Alstom company. Our work consist of the following research goals:

• RG1: Improving and further developing the sOrTES tool.

RG1.1: Building a dynamic user interface for displaying and ranking test cases

based on their requirement coverage.

• RG2: Conducting performance comparison between our tool and the existing testing process at the Alstom company.

RG2.1: Conducting an industrial test case study with the principal aim of

detecting and investigating the dependencies within test cases in an ongoing project at the Alstom company (Singapore R151).

To conduct the case study we have determined the following research questions: 1. How to improve the sOrTES tool and adapt to ongoing projects at the Alstom

company?

2. How to build a dynamic user interface for uploading and ranking test cases based on the selected scheduling techniques?

3. How to evaluate the troubleshooting and requirement coverage of sOrTES+_{, our}

proposed tool, and compare it with the current testing process at the Alstom company? 4. How to analyze the data provided by the DOORs database and detect the

dependen-cies among test cases in the Singapore R151 project?

1.2 Thesis Overview

The thesis is organized in the following sections:

Section1 introduces the context, problem formulation, the main goals, research questions

of the thesis also the motivation behind and its theoretical and practical implications;

Section 2 introduces the main concept of test optimization, also a research overview

selection, prioritization, and scheduling;

Section 3 extends the research overview on test optimization;

Section 4 describes methodology followed for conducting the thesis;

Section 5 describes the proposed solution for solving the initial problem and presents

implementation details about the data extraction process;

Section6describes the visualization details regarding the implementation and its features;

Section 7 presents the obtained results and the units of the analyse;

Section 8 presents a performance comparison between sOrTES+, the current testing

process at the Alstom company, and random scheduling;

Section 9 includes validity threats;

Section 10presents the ethical, legal and social considerations;

Section 11includes discussions and future direction points;

2

Background and Motivations

Software testing is a crucial step during Software Development Life-cycle (SDLC). The primary purpose of testing is finding critical system defects to design high-quality software. Finding system defect quickly will facilitate improving software quality. Plus, the develop-ment time of the project will decrease; therefore, the developdevelop-ment will be finished earlier. Test optimization has received attention in different approaches [3]. Nowadays, developing or generating test cases at a low cost is a concern.

2.1 Software Development Life Cycle

Developers can design and develop software applications with a higher quality reaching through all SDLC phases. The life cycle covers software inception stages according to specified requirements through deployment and maintenance [17], [18]. Steps followed in this regard can be unique in different models such as Waterfall, Iterative, Agile, and Rapid Application Development [19]. The SDLC generally consists of seven stages, which help the developer remove pitfalls from their projects since the beginning of the development, in other words, the redundant work of fixing defects in the future.

Figure 2.1: Software Development Life-cycle [18].

2.1.1 Planning Phase

The planning stage enables the developers to plan or represent a new way of treating a problem to accomplish business needs and improve the working process. Throughout this phase, all the components are added, such as technical and financial ones. The project plan is constructed during this phase where all project costs, needs, the technical in-house used technology, requirements and also the time interval for each following stage of SDLC are determined and thoroughly specified [19].

2.1.2 Analysing Phase

The Analysing stage facilitates planning for developers by creating a comprehensible representation of the upcoming product. This stage stems from a creative idea or identifying a requirement and/or an opportunity. In the business field, software developers collaborate

with other team members to conduct a product feasibility study in technical, financial, and operational areas. End-users can also provide valuable inputs since they are the ones who will have the most interaction with the final product. In this stage, unpredictable risks can be minimized by integrating the quality assurance and risk management plan for the product [19]. As a result, the project team creates a project plan which defines the project costs as well.

2.1.3 Design Phase

The design stage enables the developers to design an application. It affects the efficiency and speed of which the user interacts with the application [18], [20]. In the business field, the technical design phase collects solutions in the information systems, modeling production, financial situations. Meanwhile, the detailed design phase aims at developing and refining the system, adjusting the structure, creating documents, i.e, operating instructions, etc. Requirements are analyzed at this stage, allowing the developers to collaborate with business leaders and gather data on what business process to automate. The customers can have the opportunity to approve the product requirements specified in the Software Requirement Specification document. The project team also compiles the Design Document Specification [19].

2.1.4 Implementation Phase

The implementation stage is where the developers implement the designed and planned requirements described in previous phases. In order to develop an actual software, the company must define the coding standards and the developing tools related to their in-house programming resources. The developers can record all the functions provided at a technical level in the Functional Specification Document [19].

2.1.5 Testing Phase

The testing stage enables the developers to test their products to deliver quality software to their users. This stage can easily be integrated into a different part of all stages. Developers can use diverse tools to automate their testing phase and perform code quality testing, functional testing, integration testing, vulnerability testing, and performance testing. Developers, users, and testers can be part of the testing process. Developers are in charge of inspecting the software in order to find vulnerabilities. The functionality of the routine tasks can also be evaluated by the end-users. Meanwhile, the security, architecture, and software integration with other systems can only be evaluated by the quality assurance experts [18], [19].

2.1.6 Deployment Phase

The deployment stage enables the developers to decisively deploy the finished product that is able to accomplish all the gathered automated requirements. This stage can be an automated when the software application is deployed instantly or it can be a manual in case the applications cannot be deployed automatically. In both cases, the developers verify the execution of the all test cases so that the software execution, comprehensiveness, and correctness are successful [19]. The Deployment Plan, together with the installation, administration, and user guide should be well defined and validated. When the software development is finalized, the developers should create the Contingency Plans, which defines the course of action the project team should take in an event of a system defect [20]. The

first beta testing of the software is conducted by customers. Then, developers and/or the engineering team will be able to detect bugs according to customers’ feedback. Finally, when all the checkpoints are comprehensively covered, the company may proceed with the final deployment.

2.1.7 Maintenance Phase

The maintenance stage enables the developers to continuously monitor their software application for any issues that may appear in order to adjust them in time. In this stage, the software is transferred to the users and the development team might have no plans to add new features to the software [20]. The developers need to provide regular updates to the application in order to meet quality standards. SDLC is important since it not only ensures high product development standard but also effects the outcome of software development i.e. final product. Developers can facilitate implementation control using different guidelines and instructions in all stages of the SDLC. The project team can also limit the number of defects, ensuring a solid quality product.

2.2 Current Testing Methods

Manual testing is a traditional method for testing high security critical systems where human intervention is more reasonable than the machines [3]. The validation is directly performed by a human user in manual testing; therefore, providing better coverage on all aspects of the system workflow. Guaranteeing user-friendliness and reliable customer experience are the advantages of manual testing over automated testing. The pre-defined behavior set in SDLC’s initial phases compares software components’ conduct and features throughout manual testing. In other words, the main goal of manual testing is that the application satisfies the defined system requirements while running the application. Manual testing is valuable for all the mentioned facts, but it is costly and it is a burden to execute for the company. In order to improve manual testing, researchers have developed new ways of testing to reduce the cost and improve the testing interval. They are generally focusing on test optimization, analyzing three main variables: test case selection, prioritization, and scheduling [3], [6].

2.3 sOrTES

Several techniques are proposed concerning improving the efficiency of test case selections, prioritization and scheduling, and test automation; nevertheless, not all the companies consider test automation. Regarding the testing importance, all known software method-ologies have considered and included the testing process in the distinct development stages regardless of whether it is run manually or automatically. Furthermore, the main reason why testing is considered a process of particular importance is because it ensures a better and safer system; however, sometimes it causes prohibitive costs to the company and causes issues with the predefined deadlines. Taking the disadvantages of the testing process into consideration and improving them, Tahvili et.al. [3] developed a Python-based tool named sOrTES (Supportive Tool for Stochastic Scheduling of Manual Integration Test Cases), consisting of scheduling and prioritizing the manual integration test cases. Bombardier Transportation, Sweden, has evaluated this technique [3], [6]. They have proposed a new way of detecting the functional dependency and prioritizing test cases by the requirement coverage. According to sOrTES’ empirical evaluation, the tool has been able to avoid 40% of testing failures. It has increased the requirement coverage by up to 9.6%, compared to

the existing tools and techniques in the company. Considering this fact, we propose to further develop the tool and applying it to other projects in Bombardier Transportation Company in Västerås [3]. Figure 2.2shows all process of sOrTES in a previous project at the Bombardier company.

Figure 2.2: The structure of sOrTES workflow (see [3]).

As it is illustrated in the Figure2.2, the sOrTES tool passes through three different stages: data extraction that detects the dependencies and measures the requirement coverage, prioritization, and scheduling where the test cases are ranked for the execution. More details regarding these concepts will be described in the upcoming sections.

2.4 Test Optimization

In recent years, the testing process has gained increasing attention, and test optimization has become the headline for several reasons. As we discussed in the manual testing section, many companies use traditional testing methods to offer high-performance and effective products. While this is a far better approach compared to the primitive process, sometimes the costs are not affordable. Due to this fact, the companies and the researchers are focused on optimizing the testing process. In this case, they are concentrating on proposing new techniques for reducing unnecessary costs and increasing time efficiency, always underlining improving the fault detection rate. Tahvili et al. [3] states that the test optimization problem is largely based on decision-making [6]. The intended objectives are to maximize requirement coverage and early detection of defects [3], [6], [21]. Other authors have proposed new test optimization methods using these steps: selection, prioritization, and scheduling. Moreover, the intervention in the appropriate phase can automate the process of selecting the data, prioritizing, scheduling, and ranking the test cases regarding their importance in a well-studied order. The primary aim of this approach is decreasing the cost and increasing the fault detection rate [6]. However, not all the approaches mentioned are applicable to all testing facilities within the industry.

2.4.1 Test Case Selection

Selecting a set of previously generated test cases that are suitable for execution is refereed to as test case selection [3], [22]. Harman [23] defines test case selection as a process to test the modified software. Evaluating new changes in the existing software by re-executing a selected set of test cases is called regression testing [24].

Previous work [5], [6], [25] proposed several different methods for selecting a subset of already executed test cases. For example, in case of selecting an industrial site, Rikalovicde

et al. [26] proposes assigning multiple weighted criteria to each site. Then, the criterion from each candidate site are compared. Finally, considering the weight of each criteria, the most convenient site is selected.

2.4.2 Test Case Prioritization

The main task of test case prioritization is assigning execution priority to the selected test cases. Higher priority indicates which test case is better suited for execution. If the prioritization is done correctly i.e. highest priorities are assigned to the most important test cases, it is able to find system faults at a higher rater; therefore, it accelerates the testing process [6], [27]. Test case prioritization is applicable to all levels of testing [3].

Rothermel and Harrold [28] defines prioritization as finding a permutation of test cases for execution that maximizes software’s fault detection rate.

Some objectives such as complete minimization of test time and prior detection of defect can be met through the applications of test case prioritization techniques [6].

2.4.3 Test Case Scheduling

Test Case Scheduling is performed to optimize testing process. This technique consists of a combination of selection, prioritization and execution results. In previous work such as [3], [6], scheduling takes place before the execution statically, i.e. the results of the past executions do not affect the outcome of the scheduling. However, the result of every execution should be stored continuously to increase testing efficiency. These results are later considered in the scheduling algorithm [6]. This monitorization and consideration is what makes dynamic testing optimization possible and it is the main difference between scheduling and the last two sections (selection and prioritization) [28]. If a test case is executed successfully, the executions may continue according to the last prioritization schedule until there is a failed execution. The scheduling will continue until the executions are finished.

2.4.4 Multiple-Criteria Decision-Making

The testing process can be observed as a multi-criteria decision-making optimization problem [3]. The criteria is defined as a particular property for each test case [6]. Some properties of the test case can be complicated to measure. For instance, the probability of error detection is measured before the first test case execution where there is no data available regarding the failures. Still, the exact value is not available for this property while the system is running the tests. However, it is possible to compare test cases based on error detection probability and offer additional data regarding this comparison [6]. Vulnerable sub-systems can be identified based on this data. For example, the test cases that are able to detect errors faster will be scheduled to execute first. In order to continue the process, researchers have proposed several different criteria such as requirement coverage and troubleshooting; they need to be considered for optimizing the testing process.[6].

2.4.5 Requirement Coverage

Requirement coverage plays a role when each test case covers a different number of requirements.

There are many scenarios where a particular test case may test more than one requirement. Also, it is common for a requirement to be tested by several test cases. A proper way to increase the overall requirement coverage of the testing process is to execute the test cases

with the highest requirement coverage [3], [28]. Various possible scenarios are considered and applied in constructing an automated tool for measuring the requirements coverage.

2.5 Integration Testing

The testing process can test the interactions between two software modules. This is called integration testing, which focuses on determining interface correctness. The integration testing is performed when all the modules have been tested successfully. Several testing activities are performed on multiple levels to investigate if the system reaches the designated requirements or if it includes specific bugs. By splitting the system into diverse levels, it becomes more manageable to identify areas that have not been examined. The standard SDLC model is followed by almost every software development company. It includes 7 phases: requirement specification, business analysis, design, implementation, testing, development, and maintenance. Assembly, integration, system, and acceptance testing are all part of the testing phase of SDLC. Other testing methods such as regression, buddy, and alpha-beta testing can be conducted on any general testing levels with a specific purpose [28]. The testing levels can be eliminated or merged through a testing process except for the integration test performed on a multi-component system in many testing projects.

2.5.1 V-Model

V-Model is a popular presentation of the testing levels of the SDLC. In this model, shown in Figure2.3, the software development steps form a V-shaped model as opposed to the usual linear representation. This representation is designed to show how each step before the implementation is connected to its corresponding step after the implementation phase. V-model adds regression, buddy, and alpha-beta testing, which can be disseminated to the four main levels: unit, integration, system, and acceptance testing. These levels are able to be merged with each other or removed in specific circumstances. Testing projects developed in multi-component systems are adaptable to integration testing [6].

2.6 Test Case Dependencies

It is essential to consider the dependency between test cases before executing them since neglecting the dependencies will increase the probability of failure. This concern is more significant in integration testing levels where the interaction between modules is interde-pendent, and they can influence the result of each other’s execution. Test case dependency is completely dependent on the dependency between requirements they are covering. In another words, if two requirements are depended on each other, their dependency will cause a dependency between test cases which stand for testing requirements [6]. In order to create a better view, all different scenarios regarding dependencies between test cases are presented in the following sections. So suppose test case A is dependent on test case B, in that case, failure in B’s execution will causes test case A’s execution to fail as well. This is also called results dependency i.e. the execution of TC A is dependent on the results of TC B’s execution. The result of the TC B’s execution can help the tester choose if the test case A is suitable to be executed. However, the TC B can be executed even if the TC Ahas not passed. If the execution of TC B fails, it would not be optimal to execute TC A since it is already predictable that it would fail to execute. In case TC B passes, the TC Acan fail if there is a fault in the functionality it is testing. The opposite of a dependent test cases are called independent test cases. There are no chances for the latter to fail due to dependency if another test fails [6]. The dependencies between test cases need to be considered properly to decrease execution failure rate. Dependency detection practice extends to many testing contexts, i.e., when TCs are ranked for execution, a subset of test candidates are selected for automation, etc. This makes it an industrial challenge. Dependency can be functional, temporal, semantic, and abstract. When functional depen-dencies between TCs are detected, testing resources can be used more efficiently. This means that the redundant test execution can be avoided, the independent TCs can be executed simultaneously, and TCs, which test the same functionality, can be executed concurrently. Plus, all the aforementioned approaches may be utilized together to increase testing efficiency. The following definitions describes the different types of dependency.

2.6.1 Functional Dependency

As it is referred to by Tahvili et al. [3], functional dependency is the relation between system functionality and the order of running the test cases. For example if TC A is functionally dependent on TC B, the system requirements for executing TC A are not met unless TC B is executed first.

2.6.2 Abstract Dependency

In this type of dependency, the model consists of elements with hierarchical decomposi-tion [29]. In that case, test cases are dependent on each other hierarchically depending on the model structure.

2.6.3 Temporal Dependency

A test case has temporal dependency on another test case if its execution is limited by the execution of the other test case. For instance, if TC A is temporally dependent on TC B, then TC A should be executed after the execution of TC B [6]. Temporal dependency usually occurs on real-time system.

2.6.4 Causal Dependency

This type of dependency closely resembles temporal dependency. The TC A is called causally dependent on TC B if the execution of TC B depends on data generated by the execution of TC A [3].

3

Related Work

This section places the chosen topic in a research context, where its approach, methodology, and possible field results are compared with previous studies. Many of the previous empirical analysis studies have worked on test optimization. Literature has also proposed several techniques that focuses on multiple objectives and criteria [3]. As the number of modules and the functionality of the software grows, so does the number of the necessary test cases to cover all of its aspects i.e. requirements. So in order to find critical bugs as soon as possible, there is a need to prioritize the test cases regarding their importance [3].

Yoo and Harman [23] studied the Pareto efficient approach with multiple objectives for test case selection. In this case, execution cost, code coverage, and post-fault-detection history are the objectives considered to construct test case subsets. The empirical results showed that these algorithms were practical due to their good performance for single objective formulations. However, they were not efficient in terms of the pareto approach with multiple objectives for test case selections.

Chen and Lau [30] focused on test case selections for programs based on Boolean algebra specifications. The authors proposed three test case selection strategies: CUTPNFP, MNFP, and MUTP to detect the literal insertion and the literal reference faults. A comparison with an existing design (MAX-B) showed that their techniques were more effective. Test cases selected by these strategies form subsets of test cases based on the selected method. In this study, the Boolean specifications were in irredundant disjunctive standard form. Further studies can be dedicated to detecting abilities of these strategies in another state. Mirarab, Akhlaghi, and Tahvildari [18] studied possible ways to mitigate the costs of re-running TCs in a modified software system. They argue that faults can be revealed through a selected subset of test cases. In this study, a predetermined number of TCs are selected from the set of existing test cases sing an Integer Linear Programming problem with two coverage-based criteria. The selected test cases were prioritized by a greedy algorithm aiming to maximize the minimum coverage. These solutions can be considered close to the optimal solution. The proposed approach resulted in more effectiveness and consistency in testing compared to the existing techniques.

The work of Suri and Singhal [31], is focused on regression testing. They used test selection and prioritization techniques as well as an Ant Colony Optimization technique that was proposed in this regard. A subset from the test cases was selected and prioritized based on an existing algorithm. Using this technique, testers can explore possible paths close to optimal solutions and chose the best among them. ACO_T CSP tool was developed and empirically tested. It showed an effective reduction of the test suite site. However, other examples should prove the usability and effectiveness of this technique.

Hemmati and Briand [32], investigated and further compared similarity functions for test selection. Their study was based on a state machine-based testing context. They aimed at providing a high detection rate and maximize diversity among test cases. In addition, they have proposed a method based on evolutionary algorithms in order to optimize their selection. Moreover, the researchers used faults to compare detection rates in the study. They have used Jaccard Index in their test selection to improve the fault detection rate. The latter can measure the trigger-guards similarity of the respective test paths. Their investigation also proves that their similarity measure is more capable of optimizing test execution time compared to the other techniques that utilize coverage or randomness in their selection. This study can be replicated in different industrial systems.

Li et al [33] examined the most suitable fitness metric, landscape modality, and search techniques or algorithms for regression TC prioritization. They studied the code coverage

criteria. Empirical results showed that greedy algorithms have an optimal performance in a multimodal nature of the landscape except for fitness functions. The same results can be driven in future studies, which can be focused on fault detection criteria and metaheuristic algorithms.

Strandberg et al [34] were focused on automated system-level regression test prioritiza-tion. Their study analyzed SuiteBuilder and further used it as a tool for effective ranking test cases that took several factors into consideration such as code modifications, execution interval and fault detection rate. SuiteBuilder research resulted in a time-effective tool in terms of detecting bugs. Still, it also made it possible to reduce the need for manual work intervention, run a complete regression suite, and re-order test cases. Furthermore, the authors stated that this tool improved the overall regression testing process, the ability to detect failures. The future studies can address the full coverage of test cases.

Srikanth, Williams, and Osborne [35] were focused on differentiating between defects by considering their severity as well as improving fault detection rate by testing the existing and previous versions of code. The study introduced an approach called Prioritization of Requirements for testing, based on customer priority, requirements volatility, fault proneness, and implementation complexity. Their approach is applicable to the ongoing projects of industry.

Caliebe et al [36], on the other hand, studied test case selection in embedded systems, focusing on dependency-based and prioritization. Test cases were selected and prioritized through graph algorithms and were run on a particular system version. Study findings showed that TCs for regression testing were reduced. The authors presented an approach on TC prioritization and selection based on the system structure to reduce test process time. In order to achieve this, they have created a Component Dependency-Model system. They have also extracted the overall graph of the system by selecting black-box test cases considering system components and their dependencies.

Haidry and Miller [37], focused on prioritizing functional test suites, studying the effectiveness of these dependency structures. They presented a TC prioritization technique that preserves the dependencies in the test order. The test suite is prioritized using its dependency information, which explains the system’s interactions that is being tested. They were able to improved fault detection rate compared to the techniques based on function coverage. Future studies can generate test suites through combinations of existing prioritization techniques and dependency structure prioritization.

Wong et al [38] presented a unique hybrid technique. The authors, combined mini-mization, modification, and prioritization-based selection. Moreover, they used the code changelog and execution traces of TCs by examining older source codes. As the authors’ conclusions stated, this technique helped them identify an exponent subset of all TCs but with a different output behavior within the testing group. This method can easily be implemented using the ATAC tool. Lastly, the authors stated that it is important that the efficiency and effectiveness of regression testing in practice can and should be determined by further studies.

Srivastava and Tiagarajan [39] studied the way to prioritize tests in the software development process effectively. They built Echelon as a test prioritization system. As a result, the application’s set of tests was prioritized based on the program’s changes. Testers can use it in large systems because it follows a practical binary code-based approach. In this specific study, the tool utilized by the authors was a binary matching system. The authors used this system to compute an essential block granularity between the two different program versions. The build Echelon system by the authors uses changes between two program versions to order tests effectively. The study stated the importance of simple

heuristics, which can be implemented to improve the coverage of the affected program. Do et al. [40] provided an infrastructure that supported controlled experimentation through employing the regression testing techniques. The authors highlighted some challenges such as supporting replicability across experiments, aggregation of findings, cost of controlled experiments, sample representativeness, etc. The infrastructure provides the system, version, test, faults, and execution attributes. Further studies can assemble additional artifacts using the formats and tools prescribed to make them more representative and available to support experimentation.

Elbaum, Malishevsky, and Rothermel [27] compared prioritization techniques empirically. They decide the viability of these procedures at explicit altered adaptations utilizing contextual investigations and controlled examinations. For the most part, these methods with minor changes (fine-granularity and coarse-granularity) improved the pace of issue identification. Meanwhile, fault-proneness techniques brought minor improvements in this regard. Target programs are a factor that impacts the relative effectiveness of various methods. Also, the cost factors of specific testing processes affect the savings in this practice. Further studies can be focused on the sources of this variance and their incorporation into prioritization techniques.

Rothermel et al. [41] are primarily focused on TC prioritization techniques addressing specifically the regression testing. The authors aimed to make total coverage of code components that were previously not covered. In this case, they focused on revealing the faults that can be utilized to arrange the experiments for the test cases. The authors used and applied those methods to different sets of test cases that belong to different programs. As a result of this examination, the authors saw an improvement in test cases’ fault detection rate despite their price. Nonetheless, the authors stretched the need for further studies, providing several cost-benefit techniques among the techniques studied.

Kim and Porter [42], proposed a test-prioritization technique for regression testing where the software testing environment was interestingly resource-constrained in nature. In this case, the authors suggested a history-based test prioritization as a single test-optimization problem.

Spieker, Marijan, Gotlieb, and Mossige [43] studied selection and prioritization tech-niques based on automatic reinforcement learning. They used the Retecs method in this regard. The authors combined reinforcement learning methods and historical test information to select and prioritize TCs. Their used criteria were previous last execution, duration, and failure history. Retecs resulted in a very time-effective method. A reward function and historical results on previous continues integration (CI) cycles were used to provide a higher prioritization of error-prone TCs. Retecs used regression testing and reinforcement learning in an industrial context to improve the process of TC selection and prioritization in CI. The method performance can be enhanced using additional metadata in further studies.

Graves et al. [44] compared several different techniques that implemented regression testing. According to them, test suites were less effective when produced by minimization. This technique can be cost-effective only in cases where testing costs more. In addition, the selection of larger suites provided the same average results of fault detection as minimization with little analysis costs. However, in order to improve the confidence in random selection methods additional test cases may be used. The study reveals that the dataflow and the safe technique are equivalent to the average techniques in terms of cost-effectiveness while detecting the same faults by selecting a set of test cases with similar size. Other than test case selection, the dataflow technique is capable of performing different regression-based testing methods. The safe approach was able to found all faults during the empirical

evaluation process, but it was not adequate for several programs. Future work may be dedicated to study the advantages and disadvantages of test case selection methods based on regression and consider merging different sets of test cases.

3.1 sOrTES and Our Work sOrTES+

Tahvili [6], on the other hand, conducted a more in-depth study of the system integration testing under the multi-criteria optimization. Furthermore, the author suggested ESPRET as a specific tool to predict the execution time. Also, selected sOrTES as a tool for scheduling the manual integration test cases. Their work took place in the industrial sector at Bombardier Transportation, Sweden, actual industrial data were used to analyse and evaluate the aforementioned techniques and approaches. Both of the above-mentioned tools, [43] and [44], were able to reduce the number of failures in test case execution by a large margin. Furthermore, they increased the requirements coverage, showing optimization rates of test plans for framework coordination testing of implemented software development.

Tahvili et al. [3] presented sOrTES as a Python-based tool for manual incorporation of test scheduling. In this case, the dependency between requirement has been detected among modules so in other words in high level testing. The requirement and test specifications serve as inputs for this tool. Meanwhile, conditions and necessity inclusion for each TC is provided as an output. sOrTES identifies a set of dependent TCs and then ranks them for the execution according to their requirement coverage. This tool, used at Bombardier Transportation in Sweden, reduced testing failures and increased the requirement coverage. Test cases that were determined as independent and with higher requirement coverage are sorted at the top of the ranking list. Past executed test cases are removed from the testing schedule, and the remaining ones adapt to the new execution schedule. sOrTES has minimized redundant executions and troubleshooting efforts in the process of monitoring test cases’ execution results. This tool can be utilized to plan TCs for implementation in any event when a large arrangement of manual prerequisites and test details should be utilized during the process. As a result, the final software product requires less development time. It contributed to reducing testing failures up to 40% and also increased the requirement coverage up to 9.6% in an actual project at Bombardier Transportation Company.

All the aforementioned studies focused on test selection, prioritization, and scheduling using different techniques to decrease costs and optimize the software testing process along with the development life cycle. However, our study consists essentially of developing a new tool as an improvement or an extension to the previous sOrTES developed by Tahvili et al. [3]. The newly designed tool named sOrTES+ _{is a python-based tool constructed through}

four main phases: data extraction, prioritization scheduling, and dynamic visualizations. Contrary to the aforementioned studies, the dependencies between requirements have been detected by analyzing the "requires" and "provides" signal in SyRS, a lower level SRS inherit the dependency by SyRS in a current project, "Singapore R151" at Alstom and not in the modules unlike other studies did. Our study has three different scheduling techniques, including other criteria such as coverage, coverage and direct outputs, coverage and the total number of the outputs known as Greedy, Greedy DO, Greedy TO. Besides the fact that it introduces new algorithms intending to reduce the cost and optimize the requirement coverage process, it also offers a well-integrated dynamic user interface permitting users to select between the algorithms regarding their system requirements. The dynamic front end makes the sOrTES+ _{a practical tool to be used by testers in different projects at}

the Alstom company and in other companies with the same methodology of the testing procedure.

4

Method

The research conducted in this thesis is a modification of the research models presented by Holz et al. [29] and Gorschek et al. [45]. This adaption is used as a well-established methodology to realize our research goals, address research questions, and propose a set of solutions. Figure4.1 illustrates the research process followed and applied in this master thesis.

Figure 4.1: The research process followed for conducting this master thesis. This research project is formulated and written as an industrial case study. The main objectives and research goals are identified in collaboration with our industrial partner Alstom from where we reviewed their needs and retrieved the necessary data. The proposed guidelines of Runeson and Höst [24] and also Tahvili [6] are followed as a way of conducting research and different activities regarding the industrial requirements. As the industrial needs are reviewed, we first identified specific challenges of the current testing processes used in our industrial partner, the Alstom company. Then, we realized the necessity for improvement in terms of the tool performance for test optimization purposes.

Considering this, our research started by conducting a state-of-the-art review of the current approaches that exist in order to identify the gaps in the literature.

In this way, we began working on formulating the problem consisting of and influenced by both the academia and the industry input. After having a clear idea of the problem and how to address it, we proposed and developed the required solution that was later evaluated by our industrial partner and academia, which played a key role in balancing both their perspectives and needs.

4.1 Research Process

In each step of our process, we have considered the input and feedback from the industry. Therefore, each step is summarized in the following milestones.

• Research Problem. We have identified the research problems on existing automated testing approaches by studying the state-of-the-art review and practice. Presented in Section 1.

• Research Goals/Questions. Based on the identified challenges and the industry’s point of view, the research goals and questions are formulated in Section1.

• Proposed Solution. After identifying the research problems, formulating the research goals/questions, and having constant feedback and contact with our industrial partner, we were able to propose an optimized solution that addresses our problem and questions. Presented in the section5.

• Implementation. The details regarding the technologies utilized, work process, algorithm, visualization itself are presented in the section 6.

• Performance Evaluation. The comparison between our proposed tool named sOrTES+_{and the actual testing process at Alstom company are presented in Section8.}

5

Extraction Details

This section describes our proposed solution and the dependency between requirements, test cases, and how sOrTES+ _{extracts these information from actual industrial data. We pass}

through all the processes of extraction phase such as specification analyses, requirement extraction, test case extraction, mapping the test cases with their assigned requirements, measuring the requirement coverage and dependency degree of test cases.

5.1 Proposed Solution

Figure 5.1: The low-level and high-level requirements, software application, and the corresponding test cases in the Singapore R151 project.

Figure 5.2: The high-level requirements dependency.

As we mentioned before, sOrTES+ _{is analyzing the requirement specification in order to}

detect the dependencies. In this regard, the dependencies between requirements need to be detected, and later we are able to detect the dependencies between test cases via mapping each test case to the requirements it is testing. The design of the software product dictates how the dependencies between requirements are detected. Figure5.1is an overview of the requirements and test cases in the Singapore R151 project. As we can see in Figure 5.1

the requirements are divided into two main groups:

1. Software requirements (Low level requirements).

Software requirements (SRS) are defined as lower-level requirements. SRS are specific to a given software module. Usually, SRS are seen as white-box, meaning that they test the internal modules of the software while specifying the needs for each them. 2. System requirements (High level requirements).

System requirements (SyRS) are defined as high-level requirements which specify the overall behavior of the system. Contrary to software requirements, the SyRS are seen as black-box meaning they test the requirements that the entire system needs to satisfy.

A link between software requirements and system requirements designates the SRS satisfying certain SyRS. The dependency between software requirements are defined by the system requirements, in other words, a dependency between two software requirements is caused by both of them addressing the same system requirement.

In this project, there are two type of likns that indicate the dependency between the SyRS:

• “Provides”. Represents the outputs that effect SyRS.

• “Requires”. Represents the required input for a specific SyRS.

Test cases are commonly designed to test software requirements or system requirements. In this project we have analyzed test cases that were designed to test system level requirements. The information regarding test cases is provided to us in an .xlsx Excel file extracted from the Doors Next Generation database at the Alstom company.

The Table 5.1 represents two of the SyRS displayed in the Figure 5.1 and 5.2

SyRS ID Link: Requires Link: Provides

178350 Link ID=4967.178057 Link ID=4967.179736

178353 Link ID=4967.178050,Link ID=4967.179750, Link ID=4967.177804,

Link ID=4967.236787 Link ID=4967.178057

Table 5.1: The "Requires" and "Provides" links of two SyRS.

The table 5.2represents two of the SyRS shown in the Figure 5.1 and the corresponded test cases extracted by the traceability matrix for the Singapore R151 project.

SyRS ID Test case ID

178350 TC_2F_01_02_01_73038_PwStToHMI 178353 TC_2F_01_02_01_73042_PwStAuxOnOffSrv

Table 5.2: The SyRS and test cases extracted by the traceability matrix.

By analyzing the "requires" and "provides" signals of a system requirements we can detect their dependencies that effect their respective test cases as well.

5.2 Requirement Dependency

Figure 5.3: The dependency between requirements.

As it is illustrated in Figure5.3, two requirements RQ1 and RQ2 are called dependent if there is an output link in RQ1 that is connected to the RQ2 (RQ1 provides RQ2) and there is an input link in the RQ2 that is connected to RQ1 (RQ2 requires RQ1). In this case, RQ2 is dependent on RQ1 and not the other way around. In our example, showed in the5.1, the SyRS with ID 178350 depends on the SyRS with ID 178353 because the "provides" link (Link ID=4967.178057) of the SyRS 178363 serves as "requires" link (Link ID=4967.178057) for the SyRS 178350, for more information about the actual data refer to Table5.1.

5.3 Test Case Dependency

Dependency between test cases is defined by the requirements they are testing and their dependencies. Given a test case A that tests a requirement RA and a test case B that

tests a requirement RB, A is dependent on B if RA is dependent on a requirement RB. For example, as it is illustrated in Figure 5.4, T C2 is dependent on T C1 because T C2 tests RQ2 and RQ2 is dependent on RQ1 that is tested by T C1. Similar to T C2 and T C1, T C3 is dependent on T C2. In this example, T C4 is an independent test case. As an example in our actual data, shown in the Figure 5.1, as we already detected the dependency between the SyRS where the SyRS with ID 178350 depends on the SyRS with ID 178353, so the test case with ID: "TC_2F_01_02_01_73038_PwStToHMI" depends on "TC_2F_01_02_01_73042_PwStAuxOnOffSrv", for more information re-garding the data check the Table 5.2. According to our solution, the test case "TC_2F_01_02_01_73042_PwStAuxOnOffSrv" will be executed first since it is

indepen-dent and the test case "TC_2F_01_02_01_73038_PwStToHMI" will be executed next since it is dependent on the previous executed test case.

Figure 5.4: The dependency between test cases.

5.4 Requirement Extraction

The aim of this step is to extract the necessary information from requirement files. The requirement data is exported by parsing Excel database files. The extracted information from each entry includes the name, required requirements, provided requirements of the requirement.

This data will later allow us to extract information such as dependencies between requirements and test cases, number of independent, dependent and multi-dependent requirements, etc.

5.5 Test Case Extraction

The aim of this step is to extract the available information from test case files. The test case data is exported by parsing Excel database files. The extracted information from each entry includes the name and the list of the requirements that are being tested by the test case.

This data will later allow us to extract information such as dependencies between requirements and test cases, number of independent and dependent test cases, requirement coverage, etc.

5.6 Mapping Test Cases and Requirements

As it is explained in the previous sections, the dependency between test cases is defined by the requirements they are testing. This subsection explains how the dependencies are

extracted by mapping test cases and requirements together using the method proposed by [6].

5.6.1 Requirement Dependency Detection

This subsection explains how the Algorithm 1 detects the dependencies between require-ments. First, it creates a dependency network from all the requirerequire-ments. Then, If the requirement RQ1 provides the requirement RQ2 and RQ2 requires RQ1, we list RQ1 as a dependency of RQ2.

Algorithm 1 Requirement dependency detection Input:

Requirements: Set of all the requirements and their requires/provides

1: for each r1 in Requirements do

2: Set r1.dependencies to an empty list 3: foreach r2 in Requirements do 4: for each i in r1.provides do

5: if iin r2.requires and r1 is not r2 then

6: Append r2 to r1.dependencies 7: end if 8: end for 9: end for 10: end for Output:

Requirements: Set of all the requirements and their dependencies

5.6.2 Test Case Dependency Detection

This subsection explains the process of detecting the dependencies between test cases. First, each requirement is connected to its corresponding test cases that tests it. Afterwards, a new dependency graph is created that contains test cases and the requirement are replaced by additional edges between test cases. As this process is described in the Algorithm2, if RQ1 is tested by T C1 and RQ2 (dependent on RQ1) is tested by T C2, an edge is connected from T C1 to T C2 in the new dependency graph.

Algorithm 2 Test case dependency detection Input:

Requirements: Set of all the requirements and their requires/provides T estCases: Set of all the test cases and their tested requirements

1: for each t in T estCases do

2: foreach r in t.testedRequirements do 3: Append t to r.testedBy

4: end for

5: end for

6: for each t in T estCases do

7: Set t.dependencies to an empty list 8: foreach r in t.testedRequirements do 9: for each r2 in r.dependencies do

10: foreach t2 in r2.testedBy do

11: if t2 not in t.dependencies and t2 is not t then

12: Append t2 to t.dependencies 13: end if 14: end for 15: end for 16: end for 17: end for Output:

T estCases: Set of all the test cases and their dependencies

5.7 Requirement Coverage Measurement

The requirement coverage of each test case is defined as the number of requirements that it is testing. Each test case data includes a list of all the requirements that are being tested by that test case; therefore, as it is shown in the Algorithm3, the requirement coverage of each test case is simply defined as the length of the that list. We have also calculated the maximum requirement coverage by comparing the requirement coverage of each test case with the maximum requirement coverage value (initially set to zero). If a requirement coverage is greater than the maximum value, it will replace the maximum requirement coverage.

Algorithm 3 Measuring requirement coverage of the test cases Input:

T estCases: Set of all the test cases and the requirements they are testing

1: Set MaxRequirementCoverage to 0 2: for each t in T estCases do

3: Set t.requirementCoverage to length of t.testedRequirements

4: if t.requirementCoverageis greater than MaxRequirementCoverage then

5: Set MaxRequirementCoverage to t.requirementCoverage 6: end if

7: end for Output:

T estCases: Set of all the test cases and their requirement coverage