An industrial case study to improve test case execution time

(1)

Master of Science in Software Engineering

October 2020

An industrial case study to improve test case

execution time

(2)

This thesis is submitted to the Faculty of Computing at Blekinge Institute of Technology in

partial fulfilment of the requirements for the degree of Master of Science in Software

Engineering. The thesis is equivalent to 20 weeks of full time studies.

.

Contact Information:

Author:

Tejaswy Yadavalli

E-mail: teya16@student.bth.se

University advisor:

Prof. Panagiota chatzipetrou

(3)

ABSTRACT

Context. In software development lifecycle, testing plays a crucial role in maintaining the quality of

the software. Software testing can become costly if proper planning is not made on what kind of test cases should be made part of and how long testing should be carried out. It is important to notice that longer execution time of the test cases will be increase the resources spent in testing phase. Hence, in this research, focus is levied on determining how test case execution time varies when improvements are made to test environment that is executing the test cases. Here, improvements refer to changing the way test cases are executed in test environment. Improvements will be made by modifying the scripts with which test environment is built. In addition, analyze how difference in complexity levels of test cases will impact test case execution time before and after making changes to test environment scripts. Later, identify the factors that are causing the variance in test case execution time.

Objectives. The objectives of this research are to identify how test case execution time varies when

improvements are made to test environment scripts that is executing the test cases. Analyze and compare how different types of test cases such as simple, moderate and complex differ in their execution times before and after making improvements to test environment scripts. In addition, analyze the trends about how test case execution time differs as complexity of test cases increases. Later, identify the factors that caused the variance in test case execution time.

Methods. A case study was conducted at Bombardier transportation. The main sources of data collection

are values noted before and after making improvements to test environment and semi-structured interviews. Initially, internal test activity documents in company that details information about test case execution and test environment were read thoroughly. This acted as initial source to gain understanding about test environment. It is understood that test case execution is taking more time than expected because of few scripts in test environment. Later, improvements are made to test environment to reduce the test case execution time by modifying the scripts with which test environment is built and making those time-consuming scripts to run together. Quantitative data is gathered by noting down the total test case execution time before and after making improvements to test environment. In addition, execution times for various scripts in test environment are noted down. More quantitative data is collected by noting down test case execution times for different types of test cases. This information is later analyzed with methods such as descriptive statistics, hypothesis testing and graphical representation. Semi structured interviews are conducted in two phases to collect qualitative data. First phase was conducted before making improvements to test environment scripts. Later, second phase of interviews are used to validate the quantitative results and obtain feedback about results from different practitioners at case company.

Results. This research resulted in getting both qualitative and quantitative data. After making

(4)

Conclusions. Finally, it is concluded that complexity of test case will drive the teams to determine the

test execution time. This might be because, complex test cases might test a functionality in various patterns which will increase the execution time. Length of code within test case will also determine how long execution might take. It is understood that code reviews and proper maintenance of test environment is required. From the results derived in the research, there is a possibility to find new gaps and extend this research.

Keywords: Test case execution time, software quality, Improve execution time, Regression testing,

(5)

ACKNOWLEDGEMENTS

First of all, I would like to thank my supervisor Dr. Panagiota Chatzipetrou. She has guided me with great enthusiasm and technical knowledge, and she has supported me to grow as a researcher. Also, during the writing of this thesis her thorough reviewing was invaluable.

The journey of my master thesis has been a truly rewarding experience. I am grateful to Mr. Mattias Johansson, who have had given the unique opportunity to pursue it in the Bombardier, Hassleholm. The perks of both, the opportunity to be pursuing research and to be able to do so within a real-world industrial setting have been immensely satisfying. I have had the honor to contribute to research within this industry while being guided by the most prolific and profound minds within and outside the industry.

I would like to express my deep gratitude to my industry supervisor Mattias Johansson and his team for their valuable expert advice and strong support and encouragement throughout my thesis work. Their guidance and encouragement were one of the defining pillars of my confidence to overcome obstacles and minor setbacks and emerge out successfully. I am thankful to all my colleagues at Bombardier, Hässleholm who have been highly cooperative and enthusiastic.

I would like to express my heartfelt gratitude to my parents, Mr. Y. Raghava Rao and Ms. M. Sucharitha, my sister, Y. Bhuvana. Finally, I would like to thank my friends and one and all, who might have missed my mention inadvertently, for their help and support, without which my thesis work could not have been successful

.

(6)

LIST OF TABLES

Table 1: Non-functional test case characteristics ... 20

Table 2: Functional test case characteristics ... 21

Table 3: Roles and experience of interviews ... 26

Table 4: Execution time before improvements in test environment ... 35

Table 5: Execution time after improvements in test environment ... 35

Table 6: Execution time and mean value of before and after improvements for Simple test

case ... 36

Table 7: Execution time and mean value for before and the code improvements for moderate

test case ... 38

Table 8: Execution time and mean value for before and the code improvements for Complex

test case ... 39

(9)

LIST OF FIGURES

Figure 1: Software testing categories ... 9

Figure 2: Test process ... 19

Figure 3: Test envrionment invoke process ... 21

Figure 4: Sample log file ... 25

Figure 5: Comparision for simple test cas before and after improvements ... 38

Figure 6: Comparision for moderate test cas before and after improvements ... 39

Figure 7: Comparision for complex test cas before and after improvements ... 41

Figure 8: Simple test case before and after change of code ... 43

Figure 9: Moderate test case before and after change of code ... 44

Figure 10: Complex test case before and after change of code ... 45

Figure 11: Comparison of total execution times before and after code changes ... 46

(10)

1 INTRODUCTION

1.1 Problem statement

In Software Development Life Cycle (SDLC), to develop a qualitative product, testing is a crucial task [1], [2]. An efficient testing approach will result in a good quality product by saving time and effort [3]. It is seen that, in entire SDLC, 40% to 50% of cost and time are utilized by testing. Due to heavy competitive pressure in market software organizations must release a product in stipulated time [1], [2]. It is a challenge to teams to complete testing in time as well as ensure overall quality of the product. Consider testing a product with limited resources such as cost, time and testers, it is necessary for an organization to plan and use effective testing approaches . Testing a product and ensuring a rich qualitative product will challenge the developers and also increase the challenges during development [2]. Some of the benefits of software testing are –

x Cost effective - In SDLC phases if the bugs are caught in the initial stages or while development of software helps to save the money and time [4].

x Security - Risks and problems caused by the software can be detected in the earlier stages. The software can be trusted [5].

x Product quality - Quality product of delivered to customers is an essential requirements[4].

x Customer satisfaction - Testing ensures the user experience to give satisfaction to their customer [4].

However, most of the large-scale software’s are developed under complex environment [3]. Today, various embedded software’s are used in manufacturing of automobiles, mobile phones and train-controlled system. In industrial application, mostly 90% of electronic components used for manufacturing, controlling and monitoring processes are produced by Embedded Systems (ES) [6]. Quality of an ES can be determined by criteria such as removing defects, failures and ensuring safety of a system [6]. In real time applications, test suite consists of numerous test cases [2]. Typically, test cases are designed depending on the behavior of a system or software. To verify whether the test cases are testing the required behavior, there are some criteria such as execution time, cost, requirement coverage and test cases dependency [4]. According to [3], it is noticed that execution time plays crucial role to evaluate performance of test cases in testing. The performance of test cases is enhanced through analyzing and improving test case execution time [7].

Proper behavior of system is based on logical operations as well as timing of the system. For example, in order to control and operate flight system, flight control software is used. The software must execute accurately in fixed time interval to control the flight. To have this kind of accuracy, testing should be done beforehand such that execution time of the software is very quick. This requires designing test environment and test cases that runs ideally and quickly[8]. However, it is difficult to predict the execution time in real-time systems because response time for these systems needs to be accurate and it should be done in specific amount of time [8]. Execution time of a task can be guaranteed, if worst case execution time is determined [8]. Therefore, the test cases are investigated by measuring best case execution time and worst-case execution time by following different methods and techniques mentioned in[8] [9].

(11)

relevant to the requirements of user are developed [10]. While executing a task depending on input data and environment behavior certain variations are influenced on execution time [10]. Typically, minimal and maximal end-to-end execution time of a task is measured using test-cases [10]. In real time systems, it is cost efficient to write good set of test cases that are available for in limited cost and resources. ES contain microprocessors in order to fulfill the desired functionality and control the system [11]. Software is assimilated into electronic systems to improve performance of a system as well as safety such as antilock braking, electronic stability control [12]. Embedded system contains number of highly interconnected Electronic Control Units (ECUs) that interact with software to obtain required functionality [13]. ECUs contains memory unit microprocessor, microcontrollers used to execute code to generate relevant outcomes[13] . Nowadays, a car contains 30 to 70 of ECUs. Though, embedded system contains more number on ECUs, 55% of break down are tracked due to failures in electronic systems [12]. It is estimated that 30% of failures are occurred due to timing problems [12]. The failure of a system may lead to catastrophic consequences [12]. To ensure the proper execution of the ES, it is necessary to calculate execution time[7] [12]. The shortest execution time taken is called best case execution time (BCET) and longest execution time taken is called worst case execution time (WCET). These variations in the execution mainly occurs due to input data and test environment [7]. It is observed that calculation of minimal and maximum execution time of a task is measured through execution time taken by test cases [7]. To estimate timing analysis all the possible execution times are investigated [7]. Execution time also depends on properties of system such as architecture of hardware and software characteristics [7]. The overestimation of BCET and underestimation of WCET causes malfunction of ES [7].

Software testing can be performed either in manual or automated fashion. Both requires creating test cases and applying them on system under test[14]. Generally, automated testing is done by developing test cases in a chosen programming language and later executing together with required inputs. The quality of a product can be determined soon by executing series of automated test cases together in a test environment. Since testing is an expensive activity, it is required to monitor when test cases are scheduled together for execution. Sometimes in practice, execution of all feasible test cases may lead to excessive use of resources [14].

Every software is developed by having functional and non-functional requirements in mind. Hence test cases should also be created to test both types. They are functional testing and non-functional testing. Functional test cases are which test the slice of the functionality of the system whereas non-functional test cases are written to test non-functional aspects such as performance, reliability etc [15]. Among these different levels of testing techniques exists to ensure maximum reliability of software [16]. Some of the popular testing techniques for testing functional level of a software are unit testing, integration testing and system testing. Whereas non-functional testing is done using performance testing, usability, reliability, efficiency. These are some of the examples from software testing, but still there are many other kinds of testing in current world used for different purposes.

• Unit testing - In general, unit testing is written for single components or unit of in the application. These tests are usually written in the development phase. It is a smallest testable unit or component in a software. It is easy to catch the bug or defect due that is introduced in the due to change in the code [17].

• Integration testing - One or more modules are tested together as a whole. This testing follows top-down and bottom-up approach. The main objective of this testing is to verify how the components will work when combined together and tested [17].

• System testing - The whole system is tested to test the quality of a software. Functional and non-functional aspects such as reliability, maintainability and security. The requirements specifications of systems are checked [17].

x Acceptance testing - The acceptance test is performed by the end users. It mostly tested according depending on the software specifications but usually not to find out the bugs [17] • Performance testing - The stability, speed, scalability and responsive of the software testing

(12)

• Usability - A group of representative users involves to test the usability of the application. It is usually conducted at the different levels of testing [17].

• Reliability - The probability of the failure in the software is defined in the particular environment [17].

• Efficiency - The capacity of the software is called efficiency[17].

Depending on the severity of defects detected it is necessary to decide whether the integration phase have to be performed by both unit testing as well as integration testing. For example, errors such as “404: page not found” will be occurred due to integration issues. In ES it is necessary to remove higher number of defects due to safety criteria [16]. Therefore, testing needs to accomplish without compromising on time and cost to provide final system safety [9]. To ensure the safety, traditional verification and validation approach testing is performed with the test cases until test is passed without errors and defects. However, while executing automated test cases within test environment, results are generated automatically. Some cases, it is noticed that there would be delay in project delivery, because of test cases executed for longer.

Current research is being conducted at Bombardier, a large railway company in Sweden that manufactures locomotives, tracks for transportation and signaling systems. This company develops and uses ES for some of their day-to-day operations with train signaling system. Hence it is important to provide safety to an ES by testing properly to deliver a qualitative product. This requires huge effort from testers at the company to maintain the quality by performing testing and see that test results as quick as possible. Therefore, to achieve this, they developed a test environment with 34 scripts which helps to execute the test cases automatically.

There are 800 automated test cases in total which include both functional and non-functional test cases. Among these functional and non-functional test cases, they are divided into three types of test cases such as simple, moderate and complex based on different measures followed in current company. When all these test cases are executed together, they take 14 hours to complete their execution on test environment. Test environment was developed with an intension to see the test results before team members started their daily work in the morning. This is done by looking at test results, they start their day either by fixing important issues or start developing new functionalities. But as test environment is taking more than expected time to complete test execution, it is causing delay in software development and testing because they need to wait for test results during office hours. This situation repeated every day and created a problematic situation for intime deliveries. Hence, I was tasked to take the responsibility of reducing the test case execution time. Therefore, this research goal is to reduce test case execution time while executing test cases on test environment. This is planned to achieve by making improvements to any of the 34 scripts of test environment and later execute the 800 test cases to see whether improvements to test environment’s scripts helped in reducing execution time.

Note to the reader is that, here improvements are nothing but making modifications to scripts or code that test environment is developed with. Bombardier would be sometimes referred as current company in later sections of the document.

1.2 Research aim and objectives

(13)

feedback about derived results. A good amount of contribution will be made to software community by meeting this aim because it will help us to know whether test execution time has any impact on quality of the software. Following are the preliminary objectives of the thesis –

• O1: To analyze how test case execution time differs for various procedures in test environment before and after making improvement in test environment.

• O2: To understand the trends about how test case execution time evolves as complexity of test case increases.

• O3: To determine the factors that are causing differences in test case execution times before and after making improvements in test environment.

• O4: To understand how test case execution time is causing an impact on deliverables and quality of the software

1.3 Research questions

Based on the research aims and objectives, the research questions formulated for this study are reported within this section. Research questions are framed to meet the objectives of this thesis.

RQ 1. How does test case execution time differ between before and after making

improvements in test environment scripts?

Motivation – This research question is created to meet objective O1. The reason to include this question

is, with existing setup at current company where this research is performed, test environment is taking more than expected time to complete test case execution. It is obvious that some changes need to be done to reduce execution time. This can be done on scripts with which test environment is built. Hence after noting down the execution time of before and after making improvements to test environment, it is possible to compare both of the values. This would give insights about how execution time varies when changes are made to test environment that is running the test cases. Insights from this question will help the testing team at the company to look at the differences. This would also contribute to software community to know about how test environment can cause an impact in obtaining quick test results and maintain the quality of software.

RQ 2. How does test case execution times change as complexity of test case increases?

Motivation –Answering this question, will help this research to meet objective O2. The motivation to

include this research question is to see how different types of test cases will impact the test case execution times. As mentioned above, test cases that are executing in test environment at the chosen company are categorized into three different types such as simple, moderate and complex. Hence it deemed necessary to understand the trends about how test case execution time varies for different types of test cases designed with different complexity levels. Execution time can be retrieved in two scenarios, that is before and after making improvements to test environment scripts. Hence, using these values as input, it is possible to make a comparison and discuss about the trends.

Answer to this question will help the current company and software engineering domain to understand how test cases they developed will change the execution time depending on various test case types. These insights could be useful for their future projects at current company when designing the testcases and what kind of test cases they can include to minimize test case execution time.

RQ 3. What are the factors that caused high test case execution time ?

Motivation - This question is framed to meet objective O3. The motivation for including this question

(14)

of value to this research and software community because inputs are being drawn from professionals in software industry.

1.4 Expected research outcomes

Following are the outcomes that being expected after completing this research.

E1- To quantify the difference between test case execution times after making changes to the

environment that is executing the testcases.

E2 - To determine the change in execution times for different types of test cases such as simple, moderate

and complex.

E3 - To determine the factors that contributes to high execution time.

1.5 Structure of the Thesis

(15)

2 BACKGROUND STUDY & RELATED WORK

This chapter gives an overview about various researches that are conducted previously addressing the problems in software testing. But focus will be levied on discussing about the articles that are similar to the problem domain of this research. This chapter mainly discusses about three parts. First one is background study that discusses about various basic concepts in software testing as mentioned in different research papers. Whereas second section is related work that discusses about research papers that has done research on similar domain i.e., various concepts about test cases that are used while testing a product. Finally, last section discusses about research that has at most similarity to this research and how research gap is framed by addressing these research papers.

To have a better understanding about the context of this research, it is required to understand what Software testing is and analyze how it is conducted in different scenarios. Hence, literature review is conducted to identify the state of literature on test case execution time, different types of test time optimization methods and techniques proposed by various researchers. Documents that discussed about software testing, test time optimization methods are reviewed to understand how various authors tried to solve the problem with long execution times.

A detailed explanation about the reasons for selecting literature review, how literature review is conducted, search strings used are explained in section 3.2 of this document that discusses about research methods. Chapter mainly discusses about documents found by conducting literature review. Hence procedure for conducting literature is mentioned in chapter 3 which specifically discusses research methods.

2.1 Background study

2.1.1 Software testing concepts

(16)

Figure 1: Software testing categories

2.1.2 Testing Approaches –

In software testing there are two types of testing approaches, static and dynamic testing. In static testing without executing the program the verification is performed. Whereas in the dynamic testing with real input data that executes the program. In Figure 1, states that software testing categories.

Static testing Approach - In literature[17], states that with the source code such as model checking,

inspection of the code, handling the errors, checking the model and requirement functionalities. The common techniques used in the static testing are walkthrough of code, formal inspection and checking of desk.

Dynamic Testing Approach - Dynamic behavior of the application is tested while executing the

software with certain inputs the dynamic behavior is tested. In dynamic testing contains white box testing and black box testing.

1. White box testing -

Based on the internal details such as code structure and knowledge on the code then the tests are developed. High level of bugs and error ate detected in this level and resolved. It is also called as cleat box testing. In the process of finding the bugs and errors the testes will have the knowledge on the program components and its connections [17].

2. Black box testing -

In this method the internal details such as structure and knowledge on code will not know by testers. Required specifications and expected output are known by the testers. For different inputs the system is tested [17].

2.1.3 Software testing methods

Modes of testing -

(17)

• Manual testing – Manual testing is an old technique that prepares the test case manually and execute test cases to detect errors by the testers. It is laborious activity that requires some qualities for the testers such as being observant, speculative, creative, open minded, patient and skillful. On testing large set of applications, it becomes very difficult [18].

Issues with manual testing –

a) Tedious and time consuming - Testers will execute the test case. This usually consumes more time.

b) Investments on humans is more - In manual testing more testers are required for the testing.

c) Reliability is less - Because of human error the manual is unreliable.

d) Non-programmable - Programming cannot be performed in manual testing.

• Automation testing - The pre scripted test case that are executed in automation testing tool. After execution, test results are compared with the expected output to actual output. It takes less time and money comparative to manual testing. Once the test scripts are written their will not much involvement of the humans. Test scripts are usually developed in different languages such as python, TCL(Tool Command Language) and JavaScript. By the minimal human involvement the test scripts can be developed and executed. Human effort and cost can be reduced by the test design and development. Teat data is entered into the software application by using automation software and then generated test report is compared from actual vs expected. Same test unit is executed repeatedly. Using the automation software it is possible to record and replay the execution of system under went into the test. The main aim of automation is to eliminate the manual testing [18].

Automation testing benefits -

a) Fast - Compared to manual testing automation testing is faster.

b) Cost Effective - In AT, the test cases are executed by the automation tool or software that require less human resource and time.

c) Repeatable - The test case that is developed can be re-executed several times. d) Reusable - In the software several version of test suites are reused.

e) Comprehensive - In software application test suites are build that covers every feature of the application.

f) More reliable - The same operation can be performed several times on the application. g) Programmable - To find out the hidden bugs the testers will write the code.

h) Test coverage -More test coverage can be performed by using the AT.

Automation testing contains different types such as unit testing, smoke testing, functional testing, integration testing and regression testing. Definitions of these techniques is specified in chapter 1.

2.1.4 Regression testing -

Regression testing (RT) is done to ensure that functionality of previously working software remains same by retesting the software after changes being made. During the iterative development, RT acts as a gate for the continuous delivering the software. In software testing process 80% of testing cost and 50% of maintenance cost is consumed by RT. The need to RT will be increases as the increase in the iterative development [19].

(18)

after recent changes in the software the rerunning the test cases and checking whether the issue is fixed or re occurred again. As the increase in the size and complexity of software, the time and expenses are also increased. Therefore, test optimization is very much required for RT. To increase the testing efficiency and decrease the time and cost there are no of optimization techniques [20]. The current thesis is on reducing the test case execution time of RT.

2.1.5 Test execution time -

In real time systems, the time taken by test activity to execute as per the system properties. Execution time usually depends on several factors such as system properties, language used, kind of product testing etc. In software testing field execution time plays major role. As the increase in the execution time cause more maintenance, cost, time and budget [21].

Total time taken for the execution of each test case in test activity used for the estimation of the total execution time. But it is difficult to know the execution time before the execution. Therefore, time stamps are included in the test log to understand the execution time.

Execution time plays major role on the RT. As the RT is testing of whole system after adding or deleting the requirements from the stakeholder. The aim of the thesis is to decrees the execution time.

2.2 Related Work

This thesis focusses on finding the test case execution time, where test cases play a main role in identifying these values. Hence it is deemed necessary to know how various researchers interpreted test cases and what kind of research they had conducted about test cases. Much research [21], [19] has been conducted in testing domain by various researchers exploring different strategies related to test cases. Among them test case selection, test case prioritization, test case scheduling, test case optimization concepts are mainly discussed by different authors. In fact, author of the research paper [19], has conducted a survey on various test case prioritization and optimization techniques. As these concepts are similar to work that’s being done in this thesis which investigates execution time of test cases, research papers that discusses about these concepts are made part of this section.

Following sections gives an overview about information from various research papers but results from these papers will not be used in later parts of this thesis.

2.2.1 Test Optimization

In testing domain the test optimization is popular study according to literatures [19], [21]. As mentioned in the section-1 testing is time consuming as well as costly. So it is required to reduce the cost and time by increasing the test efficiency process. There are many optimization technique such as test case minimization, test case prioritization and test case selection [21]. The algorithms like ant colony optimization, artificial bee colony optimization, particle swarm optimization and generic algorithm [21]. In the following section industrial level of test optimization aspects are discussed below.

2.2.1.1 Test case selection

(19)

In literature [23]has stated selection of test cases based on the path condition. In this process the detection of faults is very effective. This experiment will generate the practical-sized test case from the large sized test cases.

Strength - Set of test cases are selected that can be executed in the required time.

Limitation - Only set of test case are executed. The remain are excluded. Where the whole application

will be tested. In this case the whole system need to be test every time whenever the changes ae made in the code. Therefore, it is not considered as test case selection method is not selected.

2.2.1.2 Test case minimization

In [23], in order to reduce the test cases from test suite generic algorithm is used for the test case minimization. The algorithm will generate the initial population that measures the coverage.

In [24], defined a evolutionary algorithms such as Particle Swarm Optimization , Artificial Bee Colony , Cuckoo Search Algorithm , Simulated annealing, Artificial Immune System , Ant colony optimization used for the minimization of test cases. Where these optimization algorithms are used on the large set of test cases and then based on the fitness level the test cases are differentiated. Then sub set of test cases are generated based on the parameters like inheritance, no of children etc.

Strength – Redundant test cases can be removed from the test suites [24]. Limitation – Minimal no of test cases can be executed [24].

2.2.1.3 Test case prioritization

In literature [25], provides the a test case prioritization algorithm by utilizing the prioritization of requirements , severity of faults and probability of failures. The test are executed according to the earlier possibility of detecting the faults, under less resources and time tests are executed

A cases study has been performed in literature [26], ROCKET is the technique used as a prioritization technique. Tests will take less time and higher in fault detection

Strength – When the new test cases or the test that has given more priority [19]. Limitation- It time consuming process [19].

2.3 Research gap

Vast research has been conducted over years on topics related to software testing [2] and also various types of testing [17]. These studies deal with different kinds of techniques employed by companies to improve testing process and contribute to deliver a qualitative software. In spite of this research in software testing domain, there is less focus in research community about how test case execution time impacts the quality of the software.

(20)

To best of author’s knowledge, no deep investigation has been done on measuring test case execution time for different types of test case types and that too in a case company. Furthermore, results from this thesis would be quite helpful to both case company and software industry to consider test case execution as one of the important factors in testing phase.

Following are the research gaps and can also be used as a motivation to conduct this research.

• No existence of a case study related to comparing of test case execution time for different types of test cases.

• Lack of research showing the significant difference between test case execution time in a real-time environment.

(21)

3 RESEARCH METHODOLOGY

This section is an outline of research design that has been used to reach the aims and objectives of this research. Runeson et al. [28] defines research as a step-by-step plan made to answer the research questions. These steps comprise of type of study employed, chosen data collection methods and data analysis methods used to analyze the collected data.

In this research, case study and literature review. Explanation about this research will provided in this chapter

3.1 Method selection

Considering the research from [28],[29] it is understood that various kinds of research methods exist to draw valuable insights and contribute to software industry. It is necessary to understand which research method best suits our research needs. Considering the feasibility to answering the research questions, a proper research method should be selected.

There are four popular types of research methods used to conduct the research specifically in software engineering stream [28]. They are survey, experiment, action research and case study. In this research, to get answers to research questions and meet the objectives, case study is selected as research method. The motivation for this selection is as follows.

x

Case Study -

It is an “exploratory, empirical inquiry that investigates a contemporary

phenomenon in real-life context [28]”. Considering the definition of case study, it seemed relevant to select case study as this research method. The primary motivation for selecting this method is, it will help this research to reach the objectives and retrieve answers to research questions. Current research aims to reduce the execution time of test cases when executed in a test environment. This can be achieved by understanding how test cases are being run currently, how test environment is designed and developed. It also requires investigating why and which scripts in test environment are consuming lot of time to execute. Later explore and implement the solution to reduce the test case execution time. All these scenarios help me to drive towards reaching all the research objectives and provide answers to research questions.

Moreover, a case study research method is a qualitative approach opt for exploratory situations, make improvements and find solutions in a real-world scenario [29]. This statement seems relevant to this research because, I need to make improvements to an existing test environment and find solutions to reduce the test case execution time. Later compare the differences in test case execution times between before and after making improvements to test environment. In addition, this research is conducted on a product which is most needful to a company that contributes a lot to automotive industry, hence it is believed that investigating and resolving the problem will benefit the case company and software engineering community.

3.2 Literature Review

(22)

This literature review acted as a support to claims made in this research about test execution time and also add solid information to background study in chapter 2. As mentioned in [30], following are the few steps used to conduct this literature review and synthesize needful information that helped to gather related information specific to this problem domain.

3.2.1 Choose resources -

Digital books and online databases are the one of the important sources of literature review. In spite of availability to many numbers of online databases, it is planned to stick to few online databases which has capability to retrieve wide variety of research documents. They are IEEE Xplore, Google Scholar, ACM digital library, Engineering village. This is because literature review is just one of the sources of this research but not the main source of information. Hence it is believed that it is not necessary to consider and evaluate documents from many databases which might lead to including resources that are irrelevant to this research.

3.2.2 Search literature in chosen database -

This step involves searching research papers in chosen online databases by formulating search strings. Hence a search string is formulated that seemed relevant to this research. “Test case selection” AND “Testing in real time systems” AND “Improving test case execution time” AND “Predicting execution time” AND “Test case prioritization” AND “Optimization of test case execution time”. Initially I read title, abstract and conclusions sections and later went through the full text of the articles. Following is the inclusion and exclusion criteria used in this research to choose articles. This is done in accordance with the guidelines mentioned in [30].

Inclusion criteria:

• Articles that discuss about optimization of test case execution time.

• Research papers that provides information about worst case execution time in embedded systems or time estimation in automotive industry or testing domain in general.

• Articles that describes information about test time optimization techniques. • Articles that has access to full text.

• Article that related to real time testing. • Articles that written in English.

Exclusion criteria:

• If an article only has general information about worst case execution time or test automation in general.

• Articles doesn’t describe provide any information related to testing, reducing test time execution.

• Articles that are published in any language other than English, mentions in websites or uncited links, presentations.

3.2.3 Snowballing

(23)

strengthen the motivation to perform this research. Hence it is decided to use this technique after finding articles relevant to this research.

Snowballing is conducted in two iterations by finding the research papers in IEEE, Google scholar. Initially, before first iteration, 5 research papers [18][19][20][21][32] are selected and references from these papers are examined. Then in first iteration. a search is made in IEEE with title of the research paper for all the 10 papers. 81 papers are seen in results, at first, I went through all the titles of research papers from the search. If information from title doesn’t give much information, I went through abstract of the research paper. Then 34 papers are selected in total during first iteration.

Then, I made a second iteration of snowballing on 34 papers, during second iteration, I went through the content of each document to find articles that are relevant to this research. Finally, after second iteration, 5 papers are found [22][23][24][25][26] that are discussed in background study and also some other sections of this thesis.

3.3 Case Study

This section presents the design and procedure followed to conduct this research on chosen case. There are different ways to present case study and its results. Design strategy for this process is flexible, but data collection and analysis methods should be conducted in a step-by-step manner. Hence it is decided to follow the guidelines provided by Per Runeson and Martin Höst to describe this case study process [20]. Following are the sections made part of this document to describe case study process.

1. Design and plan 2. Preparering data source 3. Data Collection procedure

4. Data analysis methods and procedures

3.3.1 Case and unit analysis

This document describes about a case study performed in Bombardier, a multi-national company in area of train control system and transportation that is located in Hässleholm, Sweden. Since case study is done in real life setting [28], current research focus is to know how test case execution time differs when changes are made to test environment’s scripts and later understand any patterns between the results of before and after making improvement to test environment scripts. Unit of analysis is the test environment used in case company to test their inter-locking system software.

3.3.2 Preparing data sources

A case study can be conducted using a single data collection source or many. Including different sources will fetch more data and adds a strength to the findings. It will also reduce the bias by not interpreting the results obtained from a single source [28]. As mentioned in [32], there are different data collection methods that can be classified into three degrees. This research used first degree data collection method because I was in direct contact with subjects and collected real time data. Hence, different data sources are used in this research to collect data.

(24)

3.3.2.1 Internal test activity documents

The case company provided me with some internal process documents named description about testing process, concepts and terminology, system description, test environment specification during initial steps of this research. Some of the documents acted as primary source of gathering data whereas other helped to get a basic understanding about company’s product and some other required aspects to conduct this research. These documents also helped in understanding about how test cases are written, how test activity is conducted, how to execute the test cases, how test environment is developed, what is the target system these test cases are scheduled to run. These documents are reviewed initially to get a basic understanding of the system and terminology used in the company. As I am bound under data privacy agreement with the company, I am not able to attach the document with this thesis document. Hence, a brief description about content in each of the document is described as follows.

• Description about testing process

Bombardier is a large railway company in Sweden manufactures locomotives, tracks for the transportation and signaling systems. Here, testing of a product is conducted in various steps. Test process is termed as the wayside product test process at current company. Following is the explanation about each step-in test process

.

Figure 2: Test process

Test process starts by creating a test plan for overall test activities of the product included in the project. Input to this step is a project plan or a release plan.

• In second step, based on test scope defined in test plan, requirements are analyzed to find suitable test cases from various test types. The first part of the Test Environment Specification is also developed here detailing the requirements on the test environment. The second part is developed during Test Environment implementation detailing the implementation specific details.

(25)

environment, test cases are also developed such that, they will be executed in test environment against the system under test.

• Execution is the fourth step where test cases are executed on test environment and test reports are generated.

• After each iteration through the test process, a quality analysis will be performed. This is done to identify the weak areas in the test suite so that quality can be continuously improved. Some of the techniques include completeness analysis, root cause analysis, functional combinations, exploratory tests.

Deliverables after performing above test process are test execution logs, progress reports, requirement traceability matrix to see if a functionality is fully verified or partially verified or untested.

A note to the reader is that current research is much related to third step and fourth step. Both of these steps details about how test environment is developed, configured and how test cases are executed. While executing these test cases in fourth step, execution time is taking about 14 hours and it is more than expected time. Hence, focus is levied on reducing test case execution time at fourth step so that reports are generated, and quality analysis can be performed quickly.

i. Concepts and terminology

This document gave an overview about various terms and concepts that are frequently used with in a system and its sub-systems. Radio Block Centre (RBC) is an example of the term explained in this document. Detailed description about what RBC is, types of RBC and how is it used in various layers of product is explained here.

Though these terms doesn’t mean anything to this research objectives and questions, it helped me when I started the thesis at company to get an understanding about terms used in different products, and stages at which these terms are used.

ii. System description

This document gives an overview about various products that are being developed at Bombardier. Each product is clearly explained from architectural point of view. It also gives an idea how system such RBC is linked together with other systems. Similar explanation is provided for many other systems used in company. It also gave me an idea about different processes in this company such as Alarm handling, indications and maintenance.

I referred this document to get an idea about how system under test functions and also get a general overview about various products in the company. Details about products, systems are not relevant to this thesis objectives or results. Hence a clear explanation about content in System description document is not provided in this thesis document.

iii. Test environment specification

(26)

This test environment plays a key role in meeting the objectives of this thesis, hence a detailed description about test environment is required so that reader can understand how various steps in test environment are used to execute the test cases automatically. Description about test environment that is obtained from this test environment specification document is as follows Test environment is used to execute the automated test cases without manual intervention. It contains test engine called as Yate that acts as a core to perform automated testing. This test engine has the ability to execute automated tests, start up, restart the complete system before the next script is executed to ensure a clean system. It also has the capability to start a module that is used for debugging purposes on a Linux system. This test environment is designed to automatically create a documentation of test results and store it in a log file.

Test environment is internally developed product by various developers in the company. This is developed in TCL/TK language by writing various scripts that executes when test environment is triggered. These scripts are designed to be run when test environment receives a signal called ‘maintenance terminal’ from its above processors. These above processors are developed to respond when train sends a signal about its position. An example of how a script looks is provided in later parts of the document. Figure -1 gives a brief overview about what happens when test environment is triggered and executes test cases for specific functionality, in this example it is CBR_HP. A note to reader is that, as I cannot disclose the name of the processes exactly due to data privacy agreement with company, figure-1 is filled with some fictional data.

Figure 3: Test environment invoke process

As mentioned above, different scripts are run in test environment to execute the test cases automatically. Following is the brief description about each of the script that executes sequentially while executing an automated test case.

• Pre Test - This is kind of warm up step conducted on a server before executing the test case. • Clean Up - Clean up test environment, that will stop all currently running processes from an

earlier test execution and removes shared memories.

• Kill Process - Kills all open processes related to the test environment and test script.

• IPC Remove - Removes all the shared memories, semaphores and message ques owned by the current user.

• getFeatures - Get list of features that a script dependent on. • intifeatues - Starts the test object and the test environment. • StartTCH - Starts the TCH.

(27)

• Setglobal - setup global variables depending on chosen side of the test object.

• getfreeport - Get and reserve the free portal using get free port tool. The portal parameter can be set to tcp, udp and unix. Default will be tcp.

• Start Application - Starts a process inside the test object.

• Set environment - Setup test environment in from of different variables. Depends on choose configurations.

• Send tch - Send a message to the TCH or through it and waits for prompt response from the TCH.

• Ss039sim_1 connected - Start RBC System.

• Ss039sim_2 connected - Starts the application simulator interface. • Run test - Executes the test script.

• Posttests - Executed by test environment after running the script.

• Kill process - Kills all still open processes related to the test environment and test object. • IpcRemove - Removes all shared memories, Semaphores and message queues owned by the

current users

Currently all these scripts are running sequentially while executing the test cases. Hence test environment is taking more than expected time. This is reason why I have been tasked to make modifications to these scripts and optimize the test case execution time.

iv.

Types of test cases

This document gave me an overview about different types of test cases developed in the team. It has information about how test cases are categorized, metrics for choosing a category. It is the responsibility of team members to choose the type of test case. Most of the members in the team are testers who are working on creating automated test cases and also maintain the test environment. Following is the information gathered from this document about test case type.

As mentioned above, there are 800 automated test cases in total. These test cases include both functional and non-functional automated test cases. All these 800 test cases are run in an automated fashion on test environment with the process mentioned in figure 3. All these test cases are divided into different types based on complexity of test case. Complexity is measured based on some specifications. These specifications are different for functional and non-functional test cases.

Table 1: Non-functional test case characteristics

Functional

Characteristics

Simple

Moderate

Complex

Non-Functional

Characteristics

Simple

Moderate

Complex

Network

usage

Not interacting with

any of the systems

Sending messages to

single system

Sending

messages

to

multiple

systems

(28)

No of

Navigations

<5

5 - 15

> 15

Total input size

<20

20 - 40

> 40

No of items to

verify

< 3

3 - 8

> 8

Table 2: Functional test case characteristics

3.3.2.2 Quantitative data using direct Observations

-Quantitative data is statistical where type of data is generally measured using values and numbers. It is used to reveal unknown facts by generating some patterns and showing results in understandable fashion. It is used to quantify a problem by utilizing the numerical data and transform into a statistic [33]. Considering this research, quantitative data is one of the important means of answering the research questions.

Direct observations is nothing but collecting the data in real-time environment. Here, observations are recorded and later used for analysis purposes. As this research is being conducted in real-time setting, this method helps to retrieve needful quantitative data. In addition, to meet the aim, objective O1 & O2 of this research, it requires quantitative data obtained by executing the test cases in a real-time test environment. Hence it believed that this method provides an opportunity to derive some real-time observations while examining the test case execution time in test environment [34].

This is one of the important sources of collecting quantitative data in this research. In this research, there is huge scope for retrieving direct observations, because values of test case execution time can be only be known by executing the test cases in real-time test environment. By executing test cases in test environment, quantitative values of test case execution time for three types of test cases can be known. These observations can be noted down so that they can later be used for data analysis. Considering the insights this method is about to provide, this method is considered as one of the data collection methods in this research. A clear explanation about procedure of using direct observations is provided in section 3.3.3.2.

3.3.2.3 Qualitative data using Interview -

Qualitative data is usually considered as unstructured data. It is used to make some conclusions and define a series of evidences about retrieved data. It is mostly existing in the form of sentences or represented as content rather than numerical values [33]. Qualitative data plays an important role in this research by deriving answer to RQ 3 and meet objective O3. This data is will be obtained by conducting interviews.

(29)

Considering the advantages with semi-structured interview, it is believed that this method will be quite useful to this research. Hence semi-structured interviews are made part of this research. In this research, semi-structured interviews are conducted in two phases. First phase happened before noting down direct observations whereas second phase happened during last steps of this research, that is after retrieving observations from test environment. Second phase also acted as a validation step, that is, data obtained from other data source such as direct observations should be validated from professionals. Interviews helped me to achieve this by gaining feedback. A clear description about how semi-structured interviews are conducted, how many people are interviewed, duration of interviews is given in section 3.3.3.4.

3.3.3 Data collection procedures

This section gives an overview about variables that are found after selecting the data sources, procedure followed to gather required observations. In addition to this, a clear description about interview procedure is provided. The main focus of this research to know how test case execution time differs when changes are made to test environment’s scripts and later understand any patterns between the results of before and after making improvement to test environment scripts. To achieve this, I tried to make best use of available data sources.

3.3.3.1 Variables from data sources –

Bringing above mentioned data sources in section 3.2 together, it helped me to identify the variables that are most important to reach the objectives of this thesis. To meet objectives O1 and O2, variables required are test case execution time and complexity of test case respectively. This is because objective-1 is to compare the differences in test case execution time before and after making changes to test environment. It is clearly evident variable that needs concentration while taking direct observations is test case execution time. Whereas objective 2 mainly maps to how complexity of test case is making an impact on test case execution time. Hence complexity of test case is the targeted variable for reaching objective 2.

Test case execution time –It is nothing but time taken to run the automated test cases part of a test suite

on a specified test environment. As mentioned above, in this research, there are 800 automated test cases in total. Currently it is taking around 14 hours to complete the whole execution which is quite more than expected. Hence test case execution time is the variable used to answer RQ 1, RQ 2 and meet objectives O1 and O2.

Complexity of test case – This is a metric that can defined in various ways based on the context.

Complexity of test case can be determined based on number of steps taken to complete execution, number of interfaces in test case, number of verification points etc. Similarly, in this case, test cases are divided into various types based on various conditions as mentioned in (iv) part of section 3.3.2.1. As RQ 2 is much relied on complexity of test case, it is selected as a variable in this research.

(30)

3.3.3.2 Data observations Procedure

This section gives a clear overview about procedure followed to note down observations from test environment. Steps mentioned below are performed after going through internal test activity documents and conducting first round of semi-structured interviews.

This method is used to generate statistics from noted observations. After going through internal test activity documents and conducting phase 1 interviews, I was aware that some scripts of test environment are taking longer time than expected to complete its execution within each test case. Hence, I noted the observations before making any changes to test environment. Most of the quantitative data obtained in this research is by knowing the time taken to pass each script of test environment while running a test case.

It took many steps to collect observations from test environment. Following is the step-by-step explanation about procedure followed to collect observations from test environment while executing the test cases.

• Initially, I was given access to the code of test environment that executes the test cases. It is developed in TCL/TK language. When test cases are executed, details of whole execution are logged into a text file in a .txt format. Due to data privacy constraints, I am not able to put the exact log file here. Hence log file with some mock data is prepared which looks like below in figure 4. Example shown in figure 4 is the log extracted from log file while a test case is being run and scripts PreTest and Cleanup of test environment are invoked.

• After executing three test cases from three different types, I started analyzing all the three log files to understand the flow. During this step, I marked the flow points or methods in log file. These flow points are named with name of the scripts that test case enters while execution. It is observed that there are total of 34 test scripts that are invoked in test environment when a test case is ran.

• To have a better understanding of execution flow and extract quantitative data to measure, I found it necessary to log the timestamp after executing each method. Hence, I used a function in TCL/TK language to log the date and time in format H:M:S. Following is the example.

set current_date [clock seconds]

# Start time: $current_date

send_user"1Thetimeis:[clockformat$current_date

format %H:%M:%S]"

(31)

Figure 4: Sample log file

3.3.3.3 Improvements in test environment –

A brief introduction about test environment and what I was tasked to improve is provided in above (iv) part of section 3.3.2.1. But this section gives clear insights about what kind of improvements are made to test environment, that is which script of test environment is modified to reduce the test case execution time. This is done by explaining the detailed procedure for making the improvements.

• After noting down the initial observation from steps mentioned in section 3.3.3.2 and closely examining the log files, it is understood that scripts startsim,startRbcSim, and startAdSimInterface are taking long execution time compared to other scripts.

• These details are noted and later confirmed with my mentor at current company. After receiving a positive reply, I started making modifications to these scripts in the test environment to reduce test case execution time.

• After few trials, I found that that best possible way to minimize the execution time is by running these three scripts together instead of running them sequentially. An example code from these scripts to make them run in parallel is posted in Appendix A. But due to data privacy issues, I am not able to attach the entire code with this document.

(32)

• Later, all test cases from three types are ran together and observations are recorded and later used for analytic purposes. It is observed that there is reduce of three in test case execution time. • Hence it is believed that test case execution time is reduced by making improvements in test

environment scripts to reduce test case execution time.

3.3.3.4 Semi-structured interviews procedure

This section gives an explanation how semi-structured interviews are conducted in this research. As mentioned above, interviews in this research are conducted in two phases. The reason for dividing into two phases is, I am new to testing domain and doesn’t have any relevant experience of working in a company before. Hence, to avoid any misinterpretation, it is thought to conduct

Phase 1 interview and extract needful details for conducting this research. This step also helped to discuss about information obtained through internal test activity documents with the personnel. Phase 2 is planned to make sure that quantitative results are useful to the company and also testing domain. In the phase, I started with explaining about the research topic to interviewees. Later quantitative results are shown and a clear explanation about the results, analysis is provided.

3.3.3.4.1 Planning and participant selection

Initially, interview subjects are selected with in the case company. An email invitation is sent to the interviewees explaining about the reason for this interview, date, time along with details about master thesis. A description about interview format to conduct in two phases is also explained in the email. Initially invite is only sent out for phase-1 before the research has begun, later a new invitation is sent for phase-2 after analyzing the quantitative results. Participants either responded yes or suggested a new time which suits them. Invitation was updated with relevant date, time and re-sent. Face to face interviews are conducted by meeting in the company’s meeting rooms. It was made sure that both phase interviews are recorded with lowest disturbances possible.

Interviews are done by choosing professionals in testing industry within the company. These practitioners are mainly involved in test planning, development and maintenance activities. It is decided to interview practitioners with at least two years of industrial experience specially in testing domain. This is to make sure that appropriate answers are received while conducting the interview.

Participants Test Role Experience

Participant 1 Test coordinator 13 years Participant 2 System test

engineer

6 Years Participant 3 Functional test

engineer

3 years Participant 4 Test engineer 9 years Participant 5 Automation

engineer

1.5 years Participant 6 System test

engineer

An industrial case study to improve test case execution time

Master of Science in Software Engineering

October 2020