• No results found

Combinatorial Modelling and Testing of PLC Software using ACTS

N/A
N/A
Protected

Academic year: 2021

Share "Combinatorial Modelling and Testing of PLC Software using ACTS"

Copied!
35
0
0

Loading.... (view fulltext now)

Full text

(1)

aster˚

as, Sweden

Thesis for the Degree of Bachelor of Science in Computer Science

15.0 credits

COMBINATORIAL MODELLING AND

TESTING OF PLC SOFTWARE USING

ACTS

Sara Ericsson

sen13009@student.mdh.se

Examiner: Daniel Sundmark

alardalen University, V¨

aster˚

as, Sweden

Supervisor: Eduard Paul Enoiu

alardalen University, V¨

aster˚

as, Sweden

(2)

M¨alardalen University Bachelor Thesis

Abstract

Testing is a crucial and necessary part of software development used to detect faults and make sure that the software works properly. One testing method proposed in research is named combinatorial testing, where a test suite covers all combinations of parameter values in a certain way. There is some evidence that indicated that most faults are triggered by a few number of parameter value combinations. There have been various tools proposed for using combinatorial testing in practice. ACTS is one of the most popular combinatorial research tools. Even if ACTS has been used in several industrial projects, few studies have shown how such a combinatorial testing tool can be used for modelling and test generation for industrial control software.

In this study we apply modelling and test generation using ACTS on a Programmable Logic Control (PLC) software. This kind of software is used in the safety critical domain for controlling different computer devices, such as a train control management system, which is in charge of many of the critical and safety related procedures on a train. Testing this kind of software is very important because failures can contribute to the loss of lives and money.

We show how ACTS can be applied to PLC software. We evaluated ACTS in terms of applicabil-ity (i.e., how can ACTS can be applied directly on modelling the PLCs) and efficiency in terms of generation time and test suite size. We used 17 PLC programs provided by Bombardier Transporta-tion. Based on the number of inputs in a program we divided the programs into three complexity categories: small, medium and large programs. An input space model was created for each program, where the needed information was obtained from both the programs and the engineers writing the programs. Each model was created as a system in the graphical interface of ACTS. The different algorithms and combinatorial techniques supported by ACTS were used to generate test suites by varying coverage of the parameter combinations (i.e. t-way) in the command line of ACTS. In this study we used a cut-off time of 1 hour for the test generation, as this is a realistic estimation based on discussion with industrial engineers. Our results showed that not all combinations of algorithms and combinatorial strengths could generate a test suite within the cut-off time. We argue that the results of the the modelling process and the results showing the efficiency of the test generation tool can be useful for practitioners considering to use combinatorial testing for PLC software.

(3)

1 Introduction 5 1.1 Problem formulation . . . 5 1.2 Contribution . . . 5 1.3 Overview . . . 6 2 Background 7 2.1 Combinatorial testing . . . 7 2.2 ACTS . . . 8 2.3 PLC Software . . . 9 2.4 Related work . . . 9 3 Method 11 3.1 Overall Process . . . 11

3.2 Measuring Applicability and Efficiency . . . 13

4 Results 14 4.1 Input space modelling . . . 14

4.2 Test generation . . . 20

4.2.1 Generation time and test suite size . . . 22

4.2.2 IPOG-D . . . 24

4.2.3 BaseChoice . . . 24

4.2.4 Long generation times and running out of memory . . . 24

4.2.5 Summary . . . 25 5 Discussion 26 6 Conclusion 27 6.1 Future work . . . 27 Bibliography 28 Appendix A Tables 30

(4)

List of Tables

3.1 Program categories based on information provided by three industrial engineers: small, medium and large programs. . . 11 4.1 Input variables with the same data type were found among the 17 programs. . . . 16 4.2 The data types found in the programs and their respective type in ACTS. . . 17 4.3 Number of data types of parameters created in ACTS. . . 17 4.4 The amount of parameter values of different sizes for every model before the

reduc-tion (VPP=Value per parameter). . . 18 4.5 The amount of parameter values of different sizes for every model after the reduction

(VPP=Value per parameter). . . 18 4.6 The average amount of values per parameter and the number of parameters per model. 21 4.7 The number of test suites generated under 600s for each program and algorithm.

The percent is the number of test suites under 600s out of the possible amount of test suites. . . 22 4.8 The amount of possible test suites generated under 600s for different program sizes. 22 4.9 Summary of the data collected when strength t=2. Time is given in seconds and

size in number of test cases. . . 22 4.10 Summary of the data collected when strength t=3. Time is given in seconds, and

size in number of test cases per test suite. . . 23 4.11 Results showing the data for four programs that were not included in the overall

evaluation for t=3. NA means that the data is not available. . . 23 4.12 Generation times in seconds for different sizes of programs using the BaseChoice

algorithm. . . 24 4.13 Shows at which strength the algorithms have reached the cut time (3600s), or ran

out of memory (OOM) when applied to the programs. The 7 programs not included in the table could generate all of the possible test suites, as well as the combinations marked with ”-”. . . 25 A.1 Data types per program . . . 30

(5)

2.1 ACTS GUI . . . 9

3.1 The distribution of the number of input parameters per selected program. . . 12

3.2 Methodology . . . 12

4.1 An XML file showing the input variables for a PLC program . . . 14

4.2 Parameter Values Source . . . 15

4.3 Base Choice Values Source . . . 15

4.4 Number of parameters for each modelled program . . . 16

4.5 A snippet showing two parameters from an XML representation of a system modelled in ACTS. . . 19

4.6 Output from ACTS to the command line after generating a test suite. . . 20

4.7 Test cases generated for a program using the IPOG algorithm with strength 1 saved as a CSV file. . . 21

(6)

Chapter 1

Introduction

Testing is an important activity in the development of software products [1]. A test includes inputs that stimulate the software (i.e., input parameters to start the software and a sequence of inputs). Test creation results in a set of tests called a test suite. A test framework runs the test suite against the software under test and produces a test result, which is compared to the expected result, stated in the requirements. The result of executing the test suite is a test oracle or verdict: a pass or a fail.

The creation of tests can be done manually or automatically using a software tool. The development of complex software has in some cases lead to functions with more input parameters that need to be thoroughly tested [2]. This complexity leads to a great amount of possible combinations between input parameter values and as such a great amount of tests. Executing all tests is often not particularly practical or effective, since most combinations of parameter values usually do not trigger an interesting behaviour or a software fault. There is a need to use some kind of appropriate testing method which can reduce the amount of tests but still provide high fault detection. Combinatorial testing is one of these methods proposed in the last couple of decades for automatic test generation [2,3].

1.1

Problem formulation

Combinatorial testing has been an active field of research for more than 20 years [2] with a num-ber combinatorial test generation tools being proposed. Advanced Combinatorial Testing System (ACTS) [4, 5] is one of these popular combinatorial test generation tools.

Even if ACTS was used in several industrial projects there is a lack of evidence on how this tool can be used for modelling and testing embedded software and industrial control software, like the one used in the safety-critical domain. IEC 61131-3 [6] is a programming standard for process control software and Programmable Logic Controllers (PLCs), commonly used in the engineering of embedded safety-critical systems (e.g., in the railway and power control domains).

1.2

Contribution

This thesis studies how ACTS can be used for input space modelling and evaluates the efficiency of the test generation process when applied to PLC software. The study strives to answer how ACTS can be used for test generation for a given set of PLC programs provided by Bombardier Transportation, a large company manufacturing train software and hardware. The programs are used in a train control management system (TCMS) running on PLCs. The TCMS is in charge of many of the safety related control and operational-critical functionality of a train, which makes it crucial to detect software faults as early as possible and in an efficient way. The use of automatic

(7)

test generation using ACTS could help an industrial engineering in testing the PLC software in a more efficient way. Therefore, it is relevant to evaluate the applicability of modelling the PLC programs and to study how efficient the test generation using ACTS for PLC software is.

The purpose of this thesis is to provide exploratory results that can be used in practice on how modelling and test generation for PLC software can be achieved in ACTS. In addition, we hope that an industrial engineer reading this thesis will get valuable insights and experience in how to apply combinatorial testing in industrial practice. This lead to the following research questions:

RQ1: Is ACTS applicable for modelling industrial PLC software?

RQ2: How efficient is ACTS in test generation for industrial PLC programs?

The answer to these questions was investigated by using ACTS to generate tests for 17 industrial PLC programs of different sizes from relatively small programs to large program containing many inputs. The modelling was performed manually while the tests were generated automatically using ACTS. In the end we evaluated the test generation in terms of efficiency and applicability based on the generation time and the number of obtained tests.

1.3

Overview

In Chapter 2 we present the background to this study, including the related work. Chapter 3 describes the method used to obtain the results presented in Chapter 4, which shows how we performed the case study, that is how input space modelling and test generation were performed using ACTS. The results are discussed and compared to related work in Chapter 5. The conclusions are presented in Chapter 6, which also includes suggestions for future work.

(8)

Chapter 2

Background

The process of testing is an important part of software development. There is some evidence [7] suggesting that 50% to 80% of the development budget is allocated to the testing process. Therefore, it is important to find efficient testing methods that can discover faults. As exhaustive testing is costly and practically impossible to use when dealing with industrial software [3] there is a need for applicable testing methods. There is some compelling experimental evidence showing that most faults are triggered by a combination of a few parameter values [3], which makes combinatorial testing a feasible alternative to exhaustive testing.

2.1

Combinatorial testing

Combinatorial testing can help in the detection of faults caused by combinations among parameter values, problems which otherwise might be difficult to discover or detect early in the development process. T-way testing is an approach to combinatorial testing [8, 2, 3, 9] that generates a test suite which covers all combinations of T parameter values at least once.

One of the most basic combinatorial testing methods is called pairwise testing. A test set generated using pairwise test generation covers all the possible pair of parameters in the tests [8, 10, 11]. There are a few empirical studies showing that most of the software faults (i.e., 50-97% according to a study by Kuhn et al. [8] and 60-90% according to another study by Kuhn et al. [3] from 2010) can be discovered using pairwise testing. This is a promising result that needs to be studied further for different industrial projects.

Other studies show that the number of detected faults increases with the strength of t-way testing, where t is the chosen combination strength [10]. The results of this study showed that no fault has been discovered by a test suite using a 7-way parameter combination [8], and therefore 6-way testing could be enough for most applications [5]. Since there is some evidence showing that software failures are triggered by a combination of 6 or fewer input parameter values [3, 12], we are interested in investigating the applicability and scalability of these combinatorial techniques for industrial programs. The problem of using higher strength t-way testing in the detriment of pairwise is directly related to how cost efficient the algorithms for generating more advanced covering arrays are. Algorithms, like the IPOG algorithm [13] used in ACTS, have made this test generation possible [10].

A combinatorial testing strategy called IPO (In-Parameter-Order) has been used in several different algorithms to handle problems such as reducing number of test cases and to cut down runtime [14]. The In-Parameter-Order-General (IPOG) algorithm is a variant of the IPO strategy. IPOG supports higher strength compared to IPO which is limited to 2-way testing [13,15]. The IPOG-D strategy is an extension of IPOG combined with a recursive construction approach to use smaller test sets to combine larger ones, instead of enumerating combinations which is used by IPOG,

(9)

to make it suitable for larger multi-way combinations [15]. Both IPO, IPOG and IPOG-D are deterministic [13, 15].

Modelling the software under test is an important part of using combinatorial testing since it affects the test generation efficiency [2, 16]. According to Nie and Leung [2] a combinatorial test model for the software under test consists of four elements: parameters, values of parameters, possible interaction relations among parameters and parameter constraints to exclude combinations that are not possible. A model consisting of these elements can be created by reviewing different system documents and code, along with other approaches. Defining the relations and constraints can make combinatorial testing more efficient [2]. We are interested, in this thesis, to investigate the creation of the test model for real-world programs and what are the sources for creating these modelling elements.

2.2

ACTS

ACTS is a combinatorial test generation research tool, previously known as FireEye [13], freely available and developed by researchers in the US National Institute of Standards and Technology and the University of Texas at Arlington. It was first released in 2006 [4] and it is actively updated.

ACTS supports t-way combinatorial testing, with t restricted to a value from 1 to 6. This way of combinatorial testing makes it possible to potentially uncover the faults caused by the combination of up to 6 parameters [4]. A set of parameters is defined as a system under test (SUT) [4] and ACTS supports the following data types: Boolean, Number, Enum and Range [5]. A number of algorithms for combinatorial test generation are implemented in ACTS version 2.92: IPOG [13], IPOG-F, IPOG-F2, IPOG-D and a special form of 1-way testing called Base Choice. In Base Choice each parameter is assigned a base value. The test cases are generated using the base value for each parameter except for one parameter, which is changed to all of its possible values. This process is repeated for each parameter to create the test suite [4, 5]. IPOG-D is preferred for larger systems while the other algorithms are better suited for moderate systems, with less than 20 parameters and 10 values per parameter on average [5].

There are two different test generation modes in ACTS, scratch and extend. The first one creates a whole new test set, while the second extends an existing test suite with automatically generated tests. ACTS supports, as previously mentioned, t-way testing and mixed test strength (different values on t) for grouped parameters. A set of parameters with the same strength is defined as a relation. Another feature is the possibility to add constraints, since some combinations might not be allowed and should therefore not be a part of the test set. The user can define constrains and the test set will only contain tests which fulfil these constraints. Once a test set has been generated it is possible to verify the t-way coverage, i.e. how many of the tests cover some t-way combinations [4,5].

ACTS is platform independent since it is implemented in Java. The tool can be used in several ways since ACTS provides both a graphic user interface (GUI), an application-programming interface (API) and a command line interface (CLI) [4]. The test sets can be saved to a file and exported as comma separated values, CSV-R or CSV-RC. These types of files can also be imported back to the tool [5].

Figure2.1shows the creation of a new system using ACTS GUI application. The left side of the application lets the user enter the parameters, which then are displayed on the right side in the window when added to the system. In the upper left corner there are tabs for accessing the option for adding constraints or relations.

Other combinatorial test generation tools are available. The IPO(PairTest) [11] and the AETG System [17] are used for pairwise testing implementing the IPO strategy. A list of available tools for pairwise testing can be found at [18]. In this thesis we use ACTS, as this is a mature tool that is expected to perform as well or better than the other available tools.

(10)

M¨alardalen University Bachelor Thesis

Figure 2.1: ACTS GUI used for creating a system model.

2.3

PLC Software

Programmable Logic Controllers (PLCs) are a particular kind of computers used in automation systems and are widely used nowadays in the safety-critical domain [19, 20]. Automation tasks for industrial processes, such as automated packaging [19], household appliances [20], and the control software in train systems [21] are some examples where PLCs are used. A PLC contains a microprocessor, a programmable memory and an I/O for communication [19]. The PLC runs in a cyclic manner such that it reads input, executes the program and writes the output [20].

There is an international standard for programming PLCs called IEC 61131-3 [6], published in 1993. Its purpose is to avoid the use of proprietary and closed versions of programming languages that cannot be used in other applications and hardware. There are five programming languages in IEC 61131-3: Instruction List (IL) and Structured Text (ST) are textual languages while Ladder Diagram (LAD), Sequential Function Chart (SFC) and Function Block Diagram (FDB) are graph-ical languages. The IEC 61131 standard includes not only the description of these five languages, but also the complete development cycle for developing software on this platform. [19]

2.4

Related work

Borazjany et al. [16] performed a case study in which they applied combinatorial testing on an industrial system by using ACTS to generate tests. The purpose of their study was to apply combi-natorial testing to a real-world system and evaluate its effectiveness, as well as gaining experience and insight in the practical application of combinatorial testing, including the input modelling process which is thoroughly described. A functional testing method was used for modelling. The

(11)

tests were generated to test both the functionality and the robustness of the ACTS software. As future work the authors planned to show a set of guidelines which can be used when applying combinatorial testing in practice, by conducting similar studies but on other real-world programs [16].

Lockheed Martin [22] conducted a study on introducing combinatorial testing in a large organiza-tion. The applicability of combinatorial testing was evaluated by comparing the different features contained in a set of combinatorial test tools and then applying these tools to real world problems. A number of pilot projects were conducted where ACTS was used as the primary tool. According to this study, ACTS continued to be used by a number of teams once the pilot projects had ended [22].

Lei et al. [13] conducted a study to generalize the pairwise IPO strategy to t-way testing, and im-plemented this new strategy in FireEye. Furthermore FireEye was evaluated in terms of efficiency by using different system configurations. The experiments showed that number of tests increased rapidly when the strength of t increased. FireEye and four other testing tools were applied to a Traffic Collision Avoidance System (TCAS) module. The measured results show that FireEye performed considerably better in both size of test suites and generation time for higher strength t-way testing [13].

These studies motivated us to perform this thesis work and provide some valuable initial steps towards the application of ACTS in industrial practice for PLC software.

(12)

Chapter 3

Method

In this thesis we have conducted a case study [23] in which we applied combinatorial modelling and test generation to PLC software and evaluated the applicability of using the combinatorial test generation tool ACTS. The purpose of using ACTS for a number of PLC programs is to show some insights into its usability and applicability. In addition, we want to show how this tool can be used for domain specific programs, such as PLC programs. Once the modelling was completed and all the test suites were generated, we have through this shown a set of guidelines on how to apply combinatorial testing on PLC software using ACTS.

3.1

Overall Process

The process began by selecting a number of representative PLC programs which could be used for modelling and test generation in ACTS. The first step, before the modelling could be conducted, was to choose a number of PLC programs which could be used for modelling the input space. We started by looking at a control train system already developed by Bombardier Transportation AB. In the provided Train Control Management System (TCMS) repository there were 189 PLC programs in total, written in IEC 61131-3 FBD language. Three engineers in Bombardier Trans-portation developing this system were asked to define three categories for PLC programs: small, medium and large. Based on these categories we asked them to write down the number of inputs interval for each category, which is shown in Table3.1.

Small PLC Program Medium PLC Program Large PLC Program Engineer 1 1-5 inputs 5-10 inputs 10-30 inputs

Engineer 2 1-7 inputs 7-10 inputs 10-20 inputs Engineer 3 1-5 inputs 5-10 inputs 10-20 inputs

Average 1-6 inputs 6-10 inputs 10-23 inputs

Table 3.1: Program categories based on information provided by three industrial engineers: small, medium and large programs.

In addition, we asked each engineer to select 2 programs from each category that contains typical input data types found in PLC programs written in IEC 61131-3 FBD language. The engineers selected 18 programs in total out of 189 PLC programs originally available. One program, out of 18, was removed from the case study because it was categorized wrongly as a medium size program instead as a small program. Finally, 17 programs were used for modelling and testing, i.e. 6 small, 5 medium and 6 large programs. As shown in Figure3.1 program 1-6 are small in size according to the categories defined by the industrial engineers. In addition, program 7-11 are medium and program 12-17 are considered to be large.

(13)

Figure 3.1: The distribution of the number of input parameters per selected program. TCMS Programs ACTS Input space modelling Test case generation Applicability Efficiency

Figure 3.2: Method for modelling and test generation using ACTS.

These programs were analysed by looking at their IEC 61131-3 representation in an XML format. One program corresponded to one XML file. We checked if the information needed for test gen-eration is included in the XML files. The information which could not be obtained directly from these files was gathered directly from engineers at Bombardier Transportation AB. All information obtained using this collection of information was used to model the input space using ACTS. The test cases were generated using the ACTS command line application. The command line applica-tion was used instead of the GUI in this case because this simplified the collecapplica-tion of the efficiency information needed for our evaluation (i.e., generation time and number of tests which were saved directly to a log file).

Figure3.2shows the overall method of the case study. TCMS programs are used for input space modelling and test generation in ACTS. The input space modelling is performed manually in ACTS while the tests are generated automatically. The modelling is evaluated in terms of applicability (i.e, can the selected programs be modelled completely using ACTS) and the test suite generation is evaluated in terms of efficiency.

(14)

M¨alardalen University Bachelor Thesis

3.2

Measuring Applicability and Efficiency

ACTS has been evaluated in terms of applicability and efficiency (see Figure3.2). The input space modelling is evaluated in terms of applicability. Applicability refers to its success in meeting the input space modelling goal (e.g., are we able to model all the input parameters needed for test generation). This makes this the measure key to determine the value of ACTS’s feasibility in an industrial context.

The test case generation is evaluated in terms of efficiency in terms of the generation time (i.e. the time ACTS takes to generate a test suite) and the number of test cases. These measures were collected directly from the output of the ACTS tool.

(15)

Results

The following chapter show the results of modelling and testing using ACTS for 17 programs, including the test generation details.

4.1

Input space modelling

Figure 4.1: An XML file showing the input variables for a PLC program

A model is defined by its parameters, the parameter constraints and possible relations among parameters. Each parameter is defined by a name, a data type and all the possible or permitted values [2]. In this thesis a base choice value is also assigned to each parameter. These base values are used when generating tests using the BaseChoice algorithm. The name and data type of each parameter was determined by manually analysing the PLC programs provided as XML files, by parsing trough the programs. The possible parameter values for some of the parameters were determined by analysing the XML files. These values were found by looking at the commented code in the variable-tag for some of the input variables (see Figure4.1). All Boolean values were assigned its possible values (i.e., true and false). The rest of the information needed to model the input space was obtained by asking the engineers at Bombardier Transportation developing these programs. The engineers writing the code were asked to fill in the missing information, which in most cases provided us with the rest of the parameter values. The rest of the missing values and base choice values were obtained by asking one test engineer at Bombardier Transportation, responsible for testing some of the programs used in this case study. No constraints or relations could be defined for the models. All information needed for defining the basic modelling were

(16)

M¨alardalen University Bachelor Thesis

either obtained from the code, i.e. by analysing the XML files or by asking the engineers which have written the code, or obtained from the software testers. Figure4.2and Figure4.3 show an overview of the source of information obtained in this study.

Figure 4.2: Distribution of the source of information for parameter values for each program.

Figure 4.3: Distribution of the source of information for base choice values for each program. Two different kind of input space models were created for these programs, depending on the data types found in the programs. A concrete model refer to an abstract representation in which tests

(17)

generated with this model can be directly used for testing. Abstract models differ from the concrete ones because they can not be used for testing right away. Hence, we first need to generate a set of abstract test cases. Some of the generated abstract tests need to be combined and enriched with glue information to form actual test cases [24]. Figure4.4 shows how many parameters each modelled program have.

The different data types found in the PLC programs are shown in Table 4.1. A concrete model was applied to 15 out of the 17 programs, while the remaining two used an abstract model. These two programs contained inputs with WORD as a data type. A WORD is a 16 bit data type [25]. The models for these programs are abstract in the sense that each WORD is represented as 16 boolean parameters. Each parameter represents a bit in the WORD and therefore the values of these parameters will form a complete WORD.

Figure 4.4: Number of parameters for each modelled program. Data type in XML Amount

BOOL 79 USINT 45 INT 17 UINT 12 REAL 10 UDINT 5 STRING 4 WORD 3

Table 4.1: Input variables with the same data type were found among the 17 programs. When the name, data type, possible parameter values and the base choice value have been deter-mined for each program, ACTS was used to model the input space. The ACTS GUI application was used for modelling, and for each program a system was created. The types of the parameters in the XML had to be converted to an appropriate corresponding data type in ACTS. ACTS provides the following for types: Boolean, Number, Range and Enum [5]. Table4.2 shows the data types found in the programs along with their corresponding type in ACTS. Range is here considered the same data type as Number, since Range consist of ranges of the type Number.

(18)

M¨alardalen University Bachelor Thesis

Rules In XML In ACTS Rule 1 BOOL Boolean

Rule 2 INT Number

Rule 3 USINT Number Rule 4 UINT Number Rule 5 UDINT Number

Rule 6 REAL Enum

Rule 7 STRING Enum Rule 8 WORD Boolean

Table 4.2: The data types found in the programs and their respective type in ACTS.

In ACTS the Number and Range data types support only integer values. Therefore, the type Enum was used for floating point values. The REAL values were increased by 0.5 to get all values within a range. The total number of different data types among the modelled programs are shown in Table4.3.

Data type in ACTS Amount

Boolean 127

Number 79

Enum 14

Table 4.3: Number of data types of parameters created in ACTS.

Some of the input space models ended up having a great amount of possible values per parameter (e.g, some of them up to 50000). For 39 parameters containing more than 100 possible values (see Table4.4) we tried to reduce the number of possibilities, since ACTS claims that a system with 100 or more parameters may be slow when generating tests. The reduction of the parameter values was conducted by asking a test engineer at Bombardier to reduce these ranges. As a result of this reduction of parameter values, only 3 parameters in 3 different programs had one parameter with more than 100 possible values. Table4.5 shows how many parameters with different amount of values each program had after the reduction.

To answer RQ1 we successfully modelled the input space of the 17 programs obtained from Bom-bardier Transportation using ACTS.

In our case study we found that ACTS is applicable to modelling the input space of PLC programs not only by using the information directly contained in the code, but also by using information provided by implementers and testers that was missing in the programs.

To create a system for each input model in ACTS is a straightforward task since each data type in the model was directly be converted to a corresponding type in ACTS. Figure4.5shows a snippet from a program modelled in ACTS, as an XML file.

(19)

Program No. VPP ≥ 100 No. VPP 21-99 No. VPP ≤ 20 Program 1 2 0 4 Program 2 0 0 3 Program 3 0 0 4 Program 4 0 0 3 Program 5 0 0 4 Program 6 0 0 34 Program 7 1 0 7 Program 8 0 0 7 Program 9 2 1 4 Program 10 2 3 3 Program 11 0 0 7 Program 12 8 0 13 Program 13 6 0 16 Program 14 1 0 22 Program 15 2 2 18 Program 16 14 0 0 Program 17 1 0 26 Total 39 6 175

Table 4.4: The amount of parameter values of different sizes for every model before the reduction (VPP=Value per parameter).

Program No. VPP ≥ 100 No. VPP 21-99 No. VPP ≤ 20

Program 1 0 2 4 Program 2 0 0 3 Program 3 0 0 4 Program 4 0 0 3 Program 5 0 0 4 Program 6 0 0 34 Program 7 1 0 7 Program 8 0 0 7 Program 9 0 2 5 Program 10 1 4 3 Program 11 0 0 7 Program 12 0 8 13 Program 13 0 3 19 Program 14 0 1 22 Program 15 1 3 18 Program 16 0 14 0 Program 17 0 1 26 Total 3 38 179

Table 4.5: The amount of parameter values of different sizes for every model after the reduction (VPP=Value per parameter).

(20)

M¨alardalen University Bachelor Thesis

Figure 4.5: A snippet showing two parameters from an XML representation of a system modelled in ACTS.

(21)

4.2

Test generation

Once the models have been created using the ACTS application we generated test suites for each model. ACTS provides the following algorithms for generating tests: IPOG, IPOG-F, IPOG-F2, IPOG-D and BaseChoice. When using IPOG or IPOG-F the strength can be set to 1-6, while IPOG-F2 and IPOG-D support strength 2-6 for t-way coverage. BaseChoice does not let the user set any strength, but base choice values can be assigned to each parameter. A base choice value can been seen as a typical or interesting value of the parameter. In this study a base value was assigned to each parameter during the modelling process by a test engineer experienced with testing PLC programs.

The tests were generated from the input space models and the systems created using ACTS, by running ACTS from the command-line. A batch file was created to generate several test suites from several programs at once. The output of the ACTS tool contained the relevant information, such as number of tests and generation time. This information was written to a log file (see figure

4.6). All tests were run on the same computer (8GB RAM, 2.9GHz CPU).

---System Name: Program4 Strength: 1

Mode: scratch Algorithm: ipog

Constraint Handling: Using forbidden tuples Verify Coverage: off

Parameters : 3

Constraints : 0

Covered Tuples : 12 Number of Tests : 6 Time (seconds) : 0.014 Output file: Program_4_ipog_1.txt

---Figure 4.6: Output from ACTS to the command line after generating a test suite.

The following code is part of the used batch file, showing how the ACTS application is being called and which parameters were being used during test generation:

j a v a −Xmx8g −Ddoi =! s t r e n g t h ! −Dalgo =! a l g o r i t h m ! −Doutput=c s v − j a r a c t s c m d 2 . 9 2 . j a r cmd ! v F u l l F i l e P a t h ! ! v F o l de r P a t h O ut p u t ! ! o u t F i l e n a m e ! >> %v S t a t i s t i c s%

where, Ddoi is the strength for t-way testing, Dalgo is the algorithm (IPOG/IPOG-F/IPOG-F2/IPOG-D/BaseChoice), Doutput is the type of file the generated tests should be saved to, and Xmx8g increases the java heap space for the application to 8GB. All other options were automat-ically set to default in ACTS command-line application when generating tests. The strength and the algorithm varied throughout the test generation, while the generated tests were saved to a CSV file for each test suite (See figure4.7).

Tests were generated by applying all possible algorithms for each program. The strength was in-creased (for all algorithms except BaseChoice) until one of the following cases occurred:

1. Strength is greater than the number of parameters. 2. Maximum strength is reached (t>6).

3. Test generation time is greater than one hour 4. ACTS application runs out of memory (heap space)

(22)

M¨alardalen University Bachelor Thesis

# ACTS Test Suite Generation: Sun May 15 05:20:53 CEST 2016 # * represents don’t care value

# Degree of interaction coverage: 1 # Number of parameters: 3

# Maximum number of values per parameter: 6 # Number of configurations: 6 # **************************************************** Param1,Param2,Param3 1,0,false 2,1,true 3,2,false 0,3,false 0,4,false 2,5,true

Figure 4.7: Test cases generated for a program using the IPOG algorithm with strength 1 saved as a CSV file.

When case (1) or case (2) occur no further test suites with higher strength can be generated, as the strength of the algorithm cannot be greater than the number of parameters and none of the algorithms provided by ACTS supports a greater strength than 6. The cut-off time for generating tests was set to one hour based on discussion with engineers in Bombardier testing the programs considered in this case study. No test was generated further for the remaining strength in case (3) because the measured test generation time from our initial experiments showed that time increases significantly with the strength of t. Therefore, according to this observation, test cases with higher strength that the one that does not finishes in the cut-off time takes longer than an hour to generate. Also the tests were not generated within the time limit, the tool does not show the amount of generated tests (or tests to be generated). This information is not available and can therefore not be used when evaluating the efficiency in terms of number of tests.

Program Avg. Parameters Program 1 18,8 6 Program 2 3,7 3 Program 3 3,75 4 Program 4 4 3 Program 5 3,75 4 Program 6 2 34 Program 7 18,9 8 Program 8 3,2 7 Program 9 19,4 7 Program 10 34,4 8 Program 11 2,9 7 Program 12 10,6 21 Program 13 7,7 22 Program 14 4 23 Program 15 19,4 22 Program 16 36 14 Program 17 13,67 27

Table 4.6: The average amount of values per parameter and the number of parameters per model. ACTS recommends the use of IPOG, IPOG-F and IPOG-F2 for program of moderate size, while using IPOG-D for larger programs. Moderate is defined as models with less than 20 parameters and approximately 10 values per parameter [5]. Table4.6show the number of parameters and the average amount of values per parameter for each modelled program.

(23)

4.2.1

Generation time and test suite size

IPOG IPOG-F IPOG-F2 IPOG-D

Test Suites <600s 78 60 40 50

Total test suites 92 92 75 75

Percent 85% 65% 53% 67%

Table 4.7: The number of test suites generated under 600s for each program and algorithm. The percent is the number of test suites under 600s out of the possible amount of test suites.

Program Size IPOG IPOG-F IPOG-F2 IPOG-D

Small 100% 54% 70% 80%

Medium 93% 77% 60% 84%

Large 67% 47% 37% 43%

Table 4.8: The amount of possible test suites generated under 600s for different program sizes. A test suite is considered to be generated efficiently in terms of generation time if it has been generated in less than 10 minutes according to information provided by the engineers at Bombardier Transportation. Table4.7shows how many tests for each program can be generated in less than 600 seconds. The ”Total test suites” number represents the total number of different test suites that can be generated for each program (i.e., in how many ways we can combine a modelled program in ACTS with an algorithm using all possible strengths of t, allowed by the model and the algorithm). For example program 3 has 4 parameters. The algorithm IPOG allows strength 1-6, but program 3 can only be generated with up to strength 4 since it does not have more than 4 parameters. Therefore, theoretically, it is possible to generate 4 different test suites using IPOG. Program 7 for example has 8 parameters, but in theory we can only generate 6 test suites with different coverage using IPOG, since the algorithm allows up to 6-way coverage. The last row in Table4.8 shows how many test suites were generated under 600 seconds. The results in Table4.7 show that the IPOG algorithm has covered 85% of its possible test suites under 600s and is therefore the most efficient in terms of generation time, followed by IPOG-D, IPOG-F, and IPOG-F2.

Table4.8shows how many test suites generated based on different algorithms that can be generated in less than 600s for each category of program sizes. In addition, this result shows that IPOG is the algorithm to use for generating most test suites within 600s for all program sizes. For example, all test suites for small programs using IPOG took less than 600s each to generate.

IPOG IPOG-F IPOG-F2 IPOG-D

Size Time Size Time Size Time Size Time

Avg 1167 0,049 1162 0,513 1170 1,156 2428594 5,915

Med 231 0,028 231 0,080 231 0,130 3698 0,014

Min 14 0,014 11 0,013 11 0,003 26 0,003

Max 8828 0,185 8827 5,086 8827 12,059 25488036 66,336 Table 4.9: Summary of the data collected when strength t=2. Time is given in seconds and size in number of test cases.

Table4.9 shows the average, median, minimum and maximum value for the generation time and test suite size when, based on the results from 15 of our programs when the strength is 2. No test data from program 15 and 16 were included since both of them ran out of memory using IPOG-D. Table4.10shows the results for 3-way coverage. The data in this table is based on 13 out of 17 programs. We were unable to use the rest of the programs because for these programs (the non-included programs are shown in Table4.11) only IPOG could be used for test generation within 600 seconds with IPOG and/or IPOG-F2 performing very badly in terms of efficiency.

The data in Table4.9and Table4.10show that IPOG-F, on average, generates smaller test suites than IPOG, but at the cost of a longer generation time on average. Table4.11shows that IPOG-F

(24)

M¨alardalen University Bachelor Thesis

IPOG IPOG-F IPOG-F2 IPOG-D

Size Time Size Time Size Time Size Time

Avg 5499 0,109 5446 6,978 5551 13,038 833331 2,232

Med 210 0,056 210 0,063 211 0,018 350 0,021

Min 28 0,003 28 0,003 28 0,002 42 0,005

Max 20339 0,560 20339 56,851 20592 118,307 10707624 28,569 Table 4.10: Summary of the data collected when strength t=3. Time is given in seconds, and size in number of test cases per test suite.

IPOG IPOG-F IPOG-F2 IPOG-D Program 10 3,3s >3600s >3600s 0,23s Program 12 66s 3555s >3600s 69s Program 15 23s >3600s >3600s NA Program 16 119s >3600s 2560s NA

Table 4.11: Results showing the data for four programs that were not included in the overall evaluation for t=3. NA means that the data is not available.

has a longer generation time (i.e., larger than 1 hour for 3 programs) than IPOG (i.e., less than 2 minutes for the same 3 programs).

Figure 4.8: Average test suite size using IPOG algorithm

Our experiment shows that the size of a test suite increases with the strength t used. Figure

4.8 shows the average test suite size using the IPOG algorithm for 8 programs. The average is calculated using those programs where IPOG could generate a test suite for all strengths 1 to 6. The plot shows the exponential increase in test suite size when t is greater than 4.

(25)

4.2.2

IPOG-D

According to the user guide included with ACTS [5] a moderate sized model has less than 20 parameters and approximately 10 values per parameter, and that IPOG, IPOG-F and IPOG-F2 are preferred for this size. For larger systems IPOG-D is preferred to be used instead. Six of our models under test have more than 20 parameters (i.e., program 6, 12-15 and 17 (see Table4.6)). Eight programs have more than 10 values per parameter on average (i.e., program 1, 7, 9, 10, 12 and 15-17). Three programs have more than 20 parameters and more than 10 values per parameter on average (i.e., program 12, 15 and 17). These three programs are considered large according to ACTS.

Our experiments show that program 12 runs out of memory when the strength is 5, program 15 runs out of memory when the strength is 2, and program 17 reaches the cut-off time within 1 hour when the strength is 6. Analysing the results from the other programs (i.e., which were defined as large PLC programs by engineers in Bombardier Transportation), showed that program 13 reaches the cut-off generation time of 1 hour at strength 5, program 14 creates all possible test suites, and program 16 runs out of memory when the strength is set to 2. When using IPOG and IPOG-D for our definition of large programs, our results show a smaller generation time for IPOG-D in 4 out of 14 comparable test suites, but with much larger test suites compared to IPOG. In general creates IPOG-D much larger test suites than the other algorithms for large programs when test suites could be compared for the same programs and strength.

4.2.3

BaseChoice

Generating tests using the BaseChoice algorithm results in a shorter generation time than pairwise testing (i.e., less than one tenth of a second, for all of the programs regardless program size). More details are shown in Table4.12) in which all measured generation times are represented in seconds.

Generation Time

Min Max Median Average

Small 0,001 0,002 0,002 0,002 Medium 0,002 0,004 0,002 0,002 Large 0,002 0,044 0,003 0,010 Overall 0,001 0,044 0,002 0,005

Table 4.12: Generation times in seconds for different sizes of programs using the BaseChoice algorithm.

From our experiments, we have shown that ACTS using base choice criteria is effi-cient in terms of generation time regardless the size of the programs considered.

4.2.4

Long generation times and running out of memory

As previously mentioned the cut-off time was set to 1 hour (3600 seconds). This means that if a test suite took longer than one hour to generate we stopped the process and no further tests with a higher strength were generated using the specific algorithm for the program under test. This occurred among the different algorithms and this occurrence differed from one program to another. For IPOG this stopping criteria happened 4 times, for IPOG-F 8 times, for IPOG-F2 5 times and for IPOG-D 5 times. The ACTS application also ran out of memory when generating a number of test suites. For IPOG it happened once, for IPOG-F it did not happen, 5 times for IPOG-F2 and 3 times for IPOG-D (see Table4.13for more details).

(26)

M¨alardalen University Bachelor Thesis

Program IPOG IPOG-F IPOG-F2 IPOG-D

Program 1 - 5, >3600 5, OOM 6, >3600 Program 7 - 5, >3600 4, OOM 5, >3600 Program 9 - - 4, OOM -Program 10 5, OOM 3, >3600 3, >3600 5, >3600 Program 12 4, >3600 4, >3600 3, >3600 5, OOM Program 13 5, >3600 4, >3600 4, >3600 5, >3600 Program 14 - 6, >3600 5, OOM -Program 15 4, >3600 3, >3600 3, >3600 2, OOM Program 16 4, >3600 3, >3600 4, >3600 2, OOM Program 17 - 5, >3600 5, OOM 6, >3600

Table 4.13: Shows at which strength the algorithms have reached the cut time (3600s), or ran out of memory (OOM) when applied to the programs. The 7 programs not included in the table could generate all of the possible test suites, as well as the combinations marked with ”-”.

4.2.5

Summary

To answer RQ2 we generated tests for 17 systems created in ACTS, one for each PLC program. All five algorithms, IPOG, IPOG-F, IPOG-F2, IPOG-D and BaseChoice were applied with different strength for t-way testing when possible. Our experiments show how the generation time and number of tests differ when using different algorithms combined with varying strengths of t-way coverage. Our results suggest that ACTS is not able to generate test suites under 1 hour for all programs regardless of the strength of the t-wise strategies and algorithms used. For several programs, different algorithms and t-strengths caused the ACTS to run out of memory before finishing the generation of a test suite. Overall, IPOG creates the most test suites under 10 minutes. In addition, IPOG algorithm did not run out of memory as many times as some of the other algorithms, as well as not stopping the generation after the cut-off time as often. IPOG-D was hypothesised to be suited for larger programs, hypothesis not supported by our results.

(27)

Discussion

This thesis has given us insights and experience on how to apply combinatorial modelling and testing to PLC software. Our results can be seen as an empirical exploration into the use of ACTS for PLC software. The modelling process shows how ACTS can be used for modelling a program and generating combinatorial tests.

Related to our work, Lei et al. [13] conducted a study which measures test suite size and generation time of FireEye (an earlier version of ACTS). FireEye uses a test generation algorithm for t-way testing based on the IPO strategy. Different configurations of parameters were used to perform experiment with the tool. Their experimental results show a significant increase in number of tests with the strength t, which is similar to the results in this thesis. As a consequence, the generation time and the cost of executing and checking the results of the tests is increasing.

Another study by Hagar et al. [22] describes the results of adopting combinatorial testing tools (i.e., mainly using ACTS) in several pilot projects at one company. Overall the adoption result was positive, but changing an already used testing process can be demanding on the company. Our thesis can contribute to the adoption knowledge needed by a company, an insight on how to use combinatorial testing by providing how modelling and test generation for PLC software using ACTS can be conducted. Our results can shed light on the expectations of applying ACTS for programs of different size. This thesis can help engineers at Bombardier Transportation to apply ACTS to their real-word industrial control software.

In the study by Hagar et al. [22], they use a combinatorial testing tool to check the coverage of already existing test suites. Ideally, there is an already existing test suites that can be used to cover a certain t combination [22]. When generating tests for higher strengths of t-way testing, we observed that this can be rather expensive in terms of number of tests and test generation time for larger programs. Therefore, one way to use ACTS would be to check the t-way coverage of an existing manually created test suite and then use the option to verify the extent the test suite covers the t-way combinations [5].

Borazjany et al. [16] applied ACTS to a real-world system (i.e., this system was ACTS itself). This study shows how to model the input space for generating tests, and they conclude that input space modelling is important and should be conducted thoroughly. Our results suggest that the input modelling is crucial since the input space is the foundation for the generated test suite. Our thesis suggests that the modelling of the input space cannot be done thoroughly by only using the information directly contained in the code. An ACTS user would need to use the information provided by the implementers and testers that for the information that is missing in the code.

(28)

Chapter 6

Conclusion

Testing is an important part of software development. Some studies are suggesting that creating tests based on the combination of just a few parameter values could potentially trigger a large portion of the software faults. Combinatorial testing generates test suites based on this assumption, i.e. creates test suites to contain a certain set of combination coverage among the parameter values. Research in this area resulted in the development of a number of test generation tools. ACTS is one such tool. Some studies showed how ACTS can be applied to real-word software. Nevertheless, there is a lack of evidence on how this approach can be applied to a domain specific software, such as PLC software (showing how the software can be modelled using the ACTS tool).

Our thesis starts with an empirical exploration on the use of combinatorial modelling and test generation for PLC software using ACTS. The results from modelling 17 programs provided by Bombardier Transportation show that the information needed to model the input space can be collected from both the PLC programs (directly) and from the engineers working at Bombardier Transportation (indirectly). Both abstract and concrete models were created depending on the data types used in the PLC programs. The systems created in ACTS from these models were used for generating tests using different algorithms in ACTS combined with different t-way cov-erage. Overall, the IPOG algorithm showed the best results in terms of generating the most test suites under 10 minutes (per program) for all the possible test suites. IPOG-D algorithm is not particularly applicable to large programs, as it was previously hypothesised.

6.1

Future work

This work could be researched further by looking into how applicable the test suites generated by ACTS are in terms of number of detected faults, in comparison to other testing methods, e.g. manual testing. This can give further insight on the applicability of combinatorial testing and ACTS for PLC software.

It is also possible to study how applicable and efficient ACTS is when applied to other types of software, i.e. SCADA, Simulink. This could give greater confidence on how efficient and applicable ACTS is in terms of modelling and test generation.

Another idea is to apply t-way testing to PLC software using another tool, similar to the study of Lei et al. [13]. In this study they applied FireEye and four other t-way testing tools to a part of an aircraft collision avoidance system. By comparing ACTS to other tools would show more generalizable results.

(29)

[1] P. Ammann and J. Offutt, Introduction to Software Testing. Cambridge University Press, 2008.

[2] C. Nie and H. Leung, “A survey of combinatorial testing,” ACM Computing Surveys (CSUR), vol. 43, no. 2, p. 11, 2011.

[3] D. R. Kuhn, R. N. Kacker, and Y. Lei, “Sp 800-142. practical combinatorial testing,” 2010. [4] L. Yu, Y. Lei, R. N. Kacker, and D. R. Kuhn, “Acts: A combinatorial test generation tool,”

in Software Testing, Verification and Validation (ICST), 2013 IEEE Sixth International Con-ference on. IEEE, 2013, pp. 370–375.

[5] “Acts user guide,” 2014, manual included with ACTS 2.92 testing tool.

[6] I. E. Commission, “IEC International Standard 1131-3,” Programmable Controllers, 2014. [7] G. Tassey, “The economic impacts of inadequate infrastructure for software testing,” National

Institute of Standards and Technology, RTI Project, vol. 7007, no. 011, 2002.

[8] R. Kuhn, R. Kacker, Y. Lei, and J. Hunter, “Combinatorial software testing,” Computer, vol. 42, no. 8, pp. 94–96, 2009.

[9] R. Bryce, Y. Lei, D. R. Kuhn, and R. Kacker, “Combinatorial testing,” Handbook of Software Engineering Research and Productivity Technologies, p. 196, 2010.

[10] R. Kuhn, Y. Lei, and R. Kacker, “Practical combinatorial testing: Beyond pairwise,” IT Professional, vol. 10, no. 3, pp. 19–23, 2008.

[11] K.-C. Tai and Y. Lie, “A test generation strategy for pairwise testing,” IEEE Transactions on Software Engineering, vol. 28, no. 1, p. 109, 2002.

[12] R. Bartholomew, “An industry proof-of-concept demonstration of automated combinatorial test,” in Automation of Software Test (AST), 2013 8th International Workshop on. IEEE, 2013, pp. 118–124.

[13] Y. Lei, R. Kacker, D. R. Kuhn, V. Okun, and J. Lawrence, “Ipog: A general strategy for t-way software testing,” in Engineering of Computer-Based Systems, 2007. ECBS’07. 14th Annual IEEE International Conference and Workshops on the. IEEE, 2007, pp. 549–556.

[14] S. Gao, J. Lv, B. Du, Y. Jiang, and S. Ma, “General optimization strategies for refining the in-parameter-order algorithm,” in 2014 14th International Conference on Quality Software. IEEE, 2014, pp. 21–26.

[15] Y. Lei, R. Kacker, D. R. Kuhn, V. Okun, and J. Lawrence, “Ipog/ipog-d: efficient test generation for multi-way combinatorial testing,” Software Testing, Verification and Reliability, vol. 18, no. 3, pp. 125–148, 2008.

[16] M. N. Borazjany, L. Yu, Y. Lei, R. Kacker, and R. Kuhn, “Combinatorial testing of acts: A case study,” in Software Testing, Verification and Validation (ICST), 2012 IEEE Fifth International Conference on. IEEE, 2012, pp. 591–600.

(30)

M¨alardalen University Bachelor Thesis

[17] D. M. Cohen, S. R. Dalal, M. L. Fredman, and G. C. Patton, “The aetg system: An ap-proach to testing based on combinatorial design,” Software Engineering, IEEE Transactions on, vol. 23, no. 7, pp. 437–444, 1997.

[18] J. Czerwonka, “Pairwise testing,” http://www.pairwise.org, online; accessed: 2016-04-09. [19] W. Bolton, Programmable logic controllers. Newnes, 2015.

[20] O. Pavlovic and H.-D. Ehrich, “Model checking plc software written in function block dia-gram,” in 2010 Third International Conference on Software Testing, Verification and Valida-tion. IEEE, 2010, pp. 439–448.

[21] E. P. Enoiu, A. ˇCauˇsevi´c, T. J. Ostrand, E. J. Weyuker, D. Sundmark, and P. Pettersson, “Automated test generation using model checking: an industrial evaluation,” International Journal on Software Tools for Technology Transfer, vol. 18, no. 3, pp. 335–353, 2016.

[22] J. D. Hagar, T. L. Wissink, D. R. Kuhn, and R. N. Kacker, “Introducing combinatorial testing in a large organization,” Computer, no. 4, pp. 64–72, 2015.

[23] P. Runeson and M. H¨ost, “Guidelines for conducting and reporting case study research in software engineering,” Empirical software engineering, vol. 14, no. 2, pp. 131–164, 2009. [24] L. S. G. Ghandehari, M. N. Bourazjany, Y. Lei, R. N. Kacker, and D. R. Kuhn, “Applying

combinatorial testing to the siemens suite,” in Software Testing, Verification and Validation Workshops (ICSTW), 2013 IEEE Sixth International Conference on. IEEE, 2013, pp. 362– 371.

[25] S. Kunze, “Automated test case generation for function block diagrams using java path funder and symbolic execution,” Master’s thesis, M¨alardalen University, 6 2015.

(31)

Tables

WORD BOOL INT UDINT USINT UINT REAL string Total

Program 1 0 1 2 0 1 0 0 2 6 Program 2 0 2 0 0 1 0 0 0 3 Program 3 0 3 1 0 0 0 0 0 4 Program 4 0 1 0 0 2 0 0 0 3 Program 5 0 1 0 0 3 0 0 0 4 Program 6 2 2 0 0 0 0 0 0 4 Program 7 0 1 0 0 5 0 0 2 8 Program 8 0 3 0 0 4 0 0 0 7 Program 9 0 4 0 1 1 1 0 0 7 Program 10 0 3 2 1 1 1 0 0 8 Program 11 0 1 0 0 6 0 0 0 7 Program 12 0 13 1 0 0 0 7 0 21 Program 13 0 7 4 2 1 8 0 0 22 Program 14 0 18 0 0 5 0 0 0 23 Program 15 0 13 3 1 1 1 3 0 22 Program 16 0 0 0 0 14 0 0 0 14 Program 17 1 6 4 0 0 1 0 0 12 Total 3 79 17 5 45 12 10 4 175

Table A.1: The data types for each XML program

Program t IPOG IPOG-F IPOFG-F2

Tests Time Tests Time Tests Time

Program 1 1 43 0.017 43 0.015 x x 2 1849 0.052 1849 0.245 1849 0.342 3 20339 0.138 20339 11.57 20592 13.707 4 203390 0.502 203390 926.115 203390 821.583 5 813560 0.93 813560 >3600 OOM OOM 6 1627120 0.826 Program 2 1 7 0.014 7 0.014 x x 2 14 0.014 14 0.013 14 0.003 3 28 0.003 28 0.003 28 0.002 4 x x x x x x 5 x x x x x x 6 x x x x x x Program 3 1 9 0.013 9 0.014 x x 2 18 0.016 18 0.014 20 0.003

(32)

M¨alardalen University Bachelor Thesis 3 36 0.015 36 0.014 36 0.003 4 72 0.004 72 0.002 72 0.003 5 x x x x x x 6 x x x x x x Program 4 1 6 0.014 6 0.02 x x 2 24 0.014 24 0.013 24 0.003 3 48 0.003 48 0.003 48 0.002 4 x x x x x x 5 x x x x x x 6 x x x x x x Program 5 1 5 0.016 5 0.013 x x 2 25 0.014 25 0.014 25 0.003 3 75 0.015 75 0.019 75 0.004 4 150 0.003 150 0.003 150 0.002 5 x x x x x x 6 x x x x x x Program 6 1 2 0.016 2 0.016 x x 2 14 0.021 11 0.024 11 0.006 3 34 0.056 30 0.063 33 0.016 4 85 0.294 83 0.438 86 0.078 5 220 5.698 214 9.222 216 0.449 6 538 98.043 519 237.51 525 3.641 Program 7 1 101 0.016 101 0.019 x x 2 1117 0.054 1111 0.275 1121 0.23 3 12312 0.188 12221 9.822 12676 11.44 4 123634 1.232 122210 1278.517 OOM OOM 5 782425 33.757 733319 >3600 6 4401170 64.98 Program 8 1 7 0.014 7 0.013 x x 2 42 0.018 42 0.018 42 0.006 3 210 0.021 210 0.037 211 0.018 4 842 0.026 840 0.118 844 0.087 5 1696 0.03 1706 0.34 2161 0.229 6 3360 0.03 3360 0.354 3838 0.327 Program 9 1 97 0.018 97 0.018 x x 2 2134 0.05 2134 0.286 2134 0.549 3 19206 0.189 19206 8.634 19206 19.524 4 38414 0.241 38701 63.426 OOM OOM 5 115236 13.432 115236 208.317 6 153648 0.314 153650 437.755 Program 10 1 97 0.022 97 0.023 x x 2 8828 0.165 8827 5.086 8827 12.059 3 35240 3.286 ? >3600 ? >3600 4 10951469 376.848 5 OOM OOM 6 Program 11 1 3 0.014 3 0.014 x x 2 15 0.016 15 0.015 15 0.003 3 48 0.017 46 0.019 48 0.005 4 140 0.02 148 0.027 142 0.011 5 346 0.023 311 0.044 355 0.019 6 729 0.021 729 0.061 737 0.031 Program 12 1 43 0.042 43 0.024 x x

(33)

2 2314 0.185 2255 1.176 2317 3.5 3 138452 66.472 132876 3554.562 ? >3600 4 ? >3600 ? >3600 5 6 Program 13 1 27 0.017 27 0.019 x x 2 650 0.059 648 0.34 652 0.357 3 15627 0.56 15552 56.851 15712 118.307 4 198340 36.491 ? >3600 ? >3600 5 ? >3600 6 Program 14 1 33 0.016 33 0.019 x x 2 231 0.028 231 0.08 231 0.13 3 1390 0.078 1386 1.37 1444 1.804 4 6970 0.716 6937 38.937 7801 42.313 5 28706 13.874 28214 1322.628 OOM OOM 6 83579 278.038 101604 26928.018 Program 15 1 100 0.027 100 0.034 x x 2 9701 0.325 9700 8.325 9700 72.349 3 883044 23.185 ? >3600 ? >3600 4 ? >3600 5 6 Program 16 1 36 0.023 36 0.022 x x 2 2088 0.312 2016 1.414 2102 1.084 3 125106 118.942 ? >3600 138913 2560.297 4 ? >3600 ? >3600 5 6 Program 17 1 33 0.016 33 0.019 x x 2 231 0.031 231 0.089 262 0.147 3 2140 0.136 1622 2.307 2052 4.661 4 13340 2.021 11321 126.953 13775 195.456 5 79702 77.008 ? >3600 OOM OOM 6 182211 1585.824

Table A.2: Data from test generation

Program t IPOG-D BaseChoice

Tests Time Tests Time

Program 1 1 x x x x 2 3698 0.01 108 0.002 3 21758 0.036 x x 4 284746 0.381 x x 5 894916 3363.761 x x 6 ? >3600 x x Program 2 1 x x x x 2 26 0.003 9 0.001 3 42 0.005 x x 4 x x x x 5 x x x x 6 x x x x Program 3 1 x x x x

(34)

M¨alardalen University Bachelor Thesis 2 34 0.003 12 0.002 3 54 0.006 x x 4 126 0.004 x x 5 x x x x 6 x x x x Program 4 1 x x x x 2 32 0.004 10 0.002 3 60 0.005 x x 4 x x x x 5 x x x x 6 x x x x Program 5 1 x x x x 2 35 0.004 12 0.002 3 75 0.007 x x 4 225 0.005 x x 5 x x x x 6 x x x x Program 6 1 x x x x 2 130 0.017 35 0.002 3 138 0.021 x x 4 192 0.03 x x 5 489 0.809 x x 6 983 49.681 x x Program 7 1 x x x x 2 27764 0.07 144 0.002 3 38885 0.073 x x 4 222201 0.574 x x 5 ? >3600 x x 6 ? >3600 x x Program 8 1 x x x x 2 176 0.004 22 0.002 3 350 0.018 x x 4 1400 0.024 x x 5 2828 0.095 x x 6 4340 0.152 x x Program 9 1 x x x x 2 5604 0.014 130 0.004 3 22698 0.043 x x 4 80316 0.181 x x 5 165676 66.132 x x 6 233964 169.789 x x Program 10 1 x x x x 2 171696 1.115 279 0.002 3 516040 0.233 x x 4 18523120 40.176 x x 5 ? >3600 x x 6 x x Program 11 1 x x x x 2 60 0.008 14 0.002 3 72 0.017 x x 4 189 0.017 x x 5 502 0.029 x x 6 903 0.046 x x

(35)

Program 12 1 x x x x 2 25488036 66.336 343 0.003 3 25562124 69.453 x x 4 32248860 1161.482 x x 5 OOM OOM x x 6 x x Program 13 1 x x x x 2 10692624 20.949 152 0.002 3 10707624 28.569 x x 4 11034566 26.655 x x 5 ? >3600 x x 6 x x Program 14 1 x x x x 2 12896 0.063 69 0.044 3 14058 0.076 x x 4 25147 0.17 x x 5 55563 30.059 x x 6 214558 2518.895 x x Program 15 1 x x x x 2 OOM OOM 399 0.003 3 x x 4 x x 5 x x 6 x x Program 16 1 x x x x 2 OOM OOM 491 0.003 3 x x 4 x x 5 x x 6 x x Program 17 1 x x x x 2 26096 0.126 79 0.002 3 27489 0.142 x x 4 48527 0.265 x x 5 129122 173.822 x x 6 ? >3600 x x

Table A.3: Data from test generation

Explanation to data in TableA.2and TableA.3:

Tests are measured in number of test cases in the test suite. Time is measured in seconds. OOM = Out of memory.

x = this strength cannot be combined with the model. ? = number of tests could not be defined due to cut time.

Note that the strength for BaseChoice algorithm is automatically set to 1.

Figure

Figure 2.1: ACTS GUI used for creating a system model.
Table 3.1: Program categories based on information provided by three industrial engineers: small, medium and large programs.
Figure 3.2: Method for modelling and test generation using ACTS.
Figure 4.1: An XML file showing the input variables for a PLC program
+7

References

Related documents

I analysed how variable was the ability of reproduction (seed production) trough outcrossing and selfing and whether this variation was related to differences in floral

In the present study, we investigated how transfection efficiency can be optimized by changing different parameters, such as: number of injections, vector concentrations and

40 Så kallad gold- plating, att gå längre än vad EU-lagstiftningen egentligen kräver, förkommer i viss utsträckning enligt underökningen Regelindikator som genomförts

In this thesis, we investigate the generation of configurable test-suites using test models based on finite state machines by extending the formalism and the test-case

The outputs of an RNG may be used directly as a random number or may be fed into a pseudorandom number generator (PRNG) and, the pseudorandom number generator creates a sequence

– In a MathScript node which appears as a frame  inside the Block diagram of a VI (available on the  Functions / Mathematics / Scripts &amp; Formulas 

We find that empirically random maps appear to model the number of periodic points of quadratic maps well, and moreover prove that the number of periodic points of random maps

In this thesis we have outlined the current challenges in designing test cases for system tests executed by a test bot and the issues that can occur when using these tests on a