Regression Test Selection in Multi-TaskingReal-Time Systems based on Run-Time logs

(1)

School of Innovation, Design and Engineering

MÄLARDALEN UNIVERSITY

Regression Test Selection in Multi-Tasking

Real-Time Systems based on Run-Time logs

Master’s Thesis at Mälardalen University

2009

LING ZHANG

Thesis Supervisor: Daniel Sundmark

Thesis Examiner:

Andreas Ermedahl

(2)

Abstract

Regression testing plays an important role during the software development life-cycle, especially during maintenance, it provides confidence that the modified parts of software behave as intended and the unchanged parts have no affect by the modification. Regression test selection is used to select test cases from the test suites which have been used to test the previous version of the software. In this thesis, we extend the traditional definition of a test case with a log file, containing information of which events that occurred when the test case was last executed. Based on the contents of this log file, we propose a method of regression test selection for multi-tasking real-time systems, able to determine which parts of software that have not been affected by the modification. Therefore, the test cases designed for the unchanged parts do not need to be re-tested.

(3)

Acknowledgements

Time is flying and it seems that I was always at a standstill. The problems were overcome one by one. My supervisor came up with the proposal for my thesis, it seems really cool for real-time system regression testing, adding a log file on the traditional test case to select test cases from original test suite. However, there were still more practical problems to solve, for example, which RTOS and on-line instrumentation should we use, how to get the log file from real-time system, how to make use of log file to design test case, how to design the regression test selection method.

I would like to express my gratitude to my supervisor Daniel Sundmark, thank you for your kindly help, your patience and excellent instruction. Thanks go to Andreas Ermedahl as well, thank you for your guide and reviewing.

Great thanks for my family's love and supporting, without you, I couldn't get the chance to study here and meet those excellent professors, friends and study-mates.

I am deeply grateful to my girlfriend Hanna-Guo Bohan, thanks your understanding and encouragement.

Last but not least, thanks help from my friends Adam Edvardsson, Shao Yankai, Jörgen Lidholm, Damir Isovic and other friends who taking care for my thesis.

(4)

Terminology:

Real-Time System: A system, whose correctness not only depends on the logical correctness of the result, but also on whether the result is delivered within a pre-defined timing constraint. [1].

Software Testing: "Any activity aimed at evaluating an attribute or capability of a program or system and determining that it meets its required results" [2].

Regression Testing: "Testing of a previously tested program following modification to ensure that defects have not been introduced or uncovered in unchanged areas of the software, as a result of the changes made. It is performed when the software or its environment is changed" [3].

TC- test case: "A set of input values, execution preconditions, expected results and execution post-conditions, developed for a particular objective or test condition, such as to exercise a particular program path or to verify compliance with a specific requirement" [3].

Test suite: "A set of several test cases for a component or system under test" [3].

Regression test selection: "In order to reduce the regression test expense, a subset of test cases is selected from test suite to re-test the previous system" [4].

Control flow graph: "A representation to use notation, the nodes represent basic blocks and edges represent control flow paths, all paths are traversed during the execution " [21].

"For procedure P contains a node for each simple or conditional statement in P; edges between nodes represent the flow of control between statements"[11].

(5)

1. Introduction

Software maintenance activities expend most of software development resources and money. Estimates claim that they account for 50 to 80% of the whole software life-cycle cost [5]. Regression testing is one of the necessary and expensive processes [6], which attempts to determine if modifications of the system or program have not affected the unchanged part or caused some unintended effects, and the system or subsystems comply with their specified requirements [7].

In a real-time system, the correctness of the system depends not only on the logic values of its computation but also on time, at which the results are produced[8] [9]. Hence regression testing differs when performed on real-time system and non-real-time systems (general software systems). Usually, real-time systems are multi-tasking, and have two or more tasks executing concurrently, and using semaphores, signals, and other type of events for inner-tasks communication. Such event information collected during test runs is used for regression test selection work.

There is plenty of regression test selection research in non-real-time systems [5] [6] [10] [11] . But in this thesis, we will present how the changes of the real-time system affect the regression test selection, particularly regression test selection on integration and system level. The specific contributions of this thesis are:

 The introduction of black-box and white-box regression test selection in regression test selection

 An extension of control-flow graph-based regression test selection to regression testing of real-time systems.

1.1 Real-Time System

Real-Time systems are systems, where the correctness of the system depends not only on the results of the computation, but also the time at which the result is produced[1]. In contrast, general software is mainly concerned about the logical correctness.

Timing constraint requirements appear in real-time systems since missed deadlines may destroy the system or maybe injure or even kill humans. For instance, in Autoliv’s frontal Airbag System, during a collision, crash sensors are sending signals to the central controlling system which has a micro-processor in an electronic control unit (ECU), which determines the

(7)

crash severity. If a crash happens, and the controlling system does not release the airbag within the right time period ( not too fast, nor too slow), the driver might be killed [12].

Consequently, software quality assurance is highly important in real-time system to ensure that problems are found and dealt with before product deployment.

1.2 Software testing and Regression testing

During software testing, any time when we change the environment, the hardware or software of a system, test cases should be rerun to make sure the system still meets the requirements. This process is known as regression testing.

Typical methods of regression testing include re-running previously tested test cases and checking whether previously fixed faults have re-emerged. For example, the most common approach of regression testing is building a test suite [13] made up of a standard library of test case

s that can be run each time the system is modified. We will discuss it in the next section.

1.3 Regression test selection

Because of the amount of time, human and money cost required by testing, the way of choosing suitable test cases should be the most considerable when we are executing regression tests. Testers ideally would like to spend as little time and expense as possible on regression testing, and try to make sure that recent changes did not add any new bugs. Pettersson [14] classifies two types of regression testing strategies, retest all test cases (re-test all) and select a subset of the test cases (selective regression test). A selective regression testing method try to single out test cases which reveal as much bugs, errors or failures as possible. Minimizing the number of test cases to execute whenever the charge has occurred can give large reductions in the software development cost.

In Figure1, a set of test cases (T) have been run upon the old system (S). If the program has been changed, then the modified system S is retested with the test case T’ which is a subset of the original test suite T. Choosing which subset of test cases that should be re-tested following changes is known as regression test selection.

(8)

Figure 1 Regression Test Selection

Regression test selection is not trivial. According to [14] test managers should get some balance in the regression testing selection: there exist trade-offs between time, and the cost for running the test cases and the fault detection ability of the test cases.

1.4 Thesis problem formulation and goal

The goal of this thesis is as follows, given a set of test cases T and a real-time system S and a change C to the system, determine a subset T’ of test cases in T that should be prioritized for re-execution, since its test cases have a higher probability of exercising the changes caused by

C.

1.5 Thesis outline

The remainder of the thesis is organized as follows: Chapter 2, we will review theoretical background about software testing, regression testing, and regression test selection techniques. A viewer system to monitor the events and record log files which used for our test cases will be presented. Chapter 3 describes our main contribution: how to perform the regression test selection based on log files recorded during previous testing of a multi-tasking real time system. In Chapter 4, multi-tasking Operating System-Vxworks, simulator and system viewer will be briefly described, and we will provide a simple proof-of -concept example. Chapter 5 summarizes the work of my thesis and future work. In Appendix A, we give a practical tutorial describing how to retrieve the log files in the VxWorks environment.

(9)

2.Background on Software Testing and Regression Selection

This section presents background material on software testing, regression testing, regression test selection and on-line instrumentation for logging.

2.1. Software Testing

During the software life-cycle, software testing plays an important role, as it is the one of the main guards to assure the quality of the software and the product in which the software is used. Software testing is not only about finding software bugs, but also about verifying and validating whether a software product meet given specifications and requirements.

Traditionally, testing is performed on different levels of the software development process:  Unit testing: The smallest part to verify and validate in the program system.

Programmers test the individual units of source code. A unit may be function, class, procedure or a smaller task.

 Integration testing: Component modules are combined after unit testing. Developers should be concerned with which units that may interact with other units, e.g. through data sharing, data passing, and inter-process communication.

 System testing: Traditionally, it is performed as a kind of black-box testing. Testers are not required to know the design of the code or its logic in detail. System testing detects defects after integration testing. It can for example be used to evaluate the system's installability, usability, functionality, reliability etc.

The method proposed in this thesis is based on system level testing, i.e. the whole system is assumed to be available during testing.

Usually, software testing has several testing methods, including black-box testing and white-box testing.

 Black-Box testing

In Black-box testing, testers are completely unconcerned about the inner-structure of the program. Instead, they are just concerned with whether the input-output behaviour acting as in specification.

(10)

 White-Box testing

During this strategy, white-box testers design test case based on the internal structure of the program, and examine the program's logic. Examples include techniques where all paths and every statement in the program must be executed at least once. Its design techniques include control flow testing, data flow testing, branch testing, etc. In our design, we use control-flow graphs of the tasks of the system. To meet the business and technical requirement, software testing guides software design and development. The system should be repeatedly tested to gain confidence in its quality, stability and reliability.

Traditionally, a test case consists of three parts: ID, Input, Expected-output, where

 ID is the identifier of the test case, it is a unique number following the test case to be identified.

 Input is the input for the execution for the system or program, it depends on software requirement, functional requirement, specification, design document.  Expected-output is the expected output. A description what you want the function or program to do.

2.2. Regression testing

Whenever code is modified or added to the system, we need to test the program or system again in order to make sure the changes have not corrupted already existing functionality. This process is known as regression testing.

During regression testing, a set of test cases can be re-used to save test case design time Why do we need regression testing?

1. To test whether has faults in the new version, even it is available in the previous. 2. Saving time and cost to build new test cases.

When do we need regression testing? 1. During software or system maintenance.

2. Changing class or code for the extreme programming. How do we perform regression testing?

1.In industry, usually retest-all technique is used to do regression testing, time and money expense are huge cost.

(11)

depending on software requirement, including minimization techniques, safe techniques, control-flow regression test selection techniques.

2.3. Regression Test Selection

In the last few decades, the main research on regression testing has been focusing on regression test selection techniques, which selects test cases from an existing test suite to test a modified program.

According to Rothermel [6] a typical selective retest technique proceeds as follow. Given a program P, P’ is a modified version of P, P (i) is the output for the input i to P, P'(i) refers to the output of P 's input i, a set of test case T( test suite) used to test P, then methods or techniques should be found to select test cases for P'.

In other words, again according to [6], a typical regression test select technique proceeds as follows:

1. Select a set of test cases TT' to be executed on P'.

2. Test P' on T', establish the correctness of P' with respect to T' 3. If necessary, create T'', a set of new functional test cases of P' 4. Test P' with T'', establishing the correctness of P', with respect to T'' 5. Create T''', a new test suite and test history for P', from T, T'', and T'''

In [10][14][11][18], the authors highlight several differences among regression test selection technique. The minimization technique produced the smallest and the least effective test suites, the safe technique and data-flow technique has nearly equivalent average behavior in terms of cost-effectiveness, we extend the techniques in the background of testing.

Numerous regression test selection techniques for re-using test cases have been proposed and examined. Categories of techniques include:

 Ad-hoc / Random Techniques  Minimization techniques  Safe techniques

 Data-flow techniques

 Control-flow coverage techniques

Ad-hoc / Random Techniques: Random techniques aim to randomly select some test cases from T to retest the program.

Minimization techniques: Minimization techniques aim to select a minimal set of tests that must be run to meet necessary structural coverage.

(12)

Safe techniques: Safe techniques aim to select every test that may cause the modified program to produce different output than the original program. This guarantees that the selected subset T’ contains all test cases in T which can reveal faults in the changed program

P’.

Data-flow coverage techniques: Data-flow coverage techniques are based on the observation of variables data changing. The data-flow test model intends to identify and exercise paths where a bug exists but the sequence path has not been executed.

Control-flow coverage techniques: The aim of control-flow coverage is in order to pass through condition control structures to test every individual statement, instruction and function call once [19].

All regression test selection techniques attempt to cut down the cost of re-test the modified program or system and shorten the time to test or select the subset of the existing test suite [14]. Most regression test selection techniques select test cases based on the information of the existing program code and the modified code [4][5][10][11][14][15]. Other techniques select test cases based on the information collected from the specification and requirements [16][17].

In [11], the author focuses on the application of control flow analysis to perform safe regression testing. The approach is related to white-box testing without real-time tasks. In the next section, we will present how to extend the method to also incorporate real-time system. Furthermore, we will pay more attention to the program code, structure and a log file, describing the execution of task in the real-time system.

2.4. Monitoring and Logging for Software Testing

In real-time systems, time constraints are important issues. Inside some real-time operating systems, software-based instrumentation probes are used to capture task and event information, which is used by our regression test selection method. We can view and log the state of tasks and events through some 'Viewer', such as System Viewer for Vxworks, μC/OS-View for μC/OS-View RTOS, OSE Illuminator, and the LTT (Linux Trace Toolkit). Information on tasks executed, occurred interrupts and context switches can be monitored through the tools. The software-based instrumentation probes capture information during run-time. The viewers present the information as log files, for example as text file. In the method presented in this thesis, we make use of the information stored in the log files in order to perform regression test selection.

(13)

3. Log-based Regression Test Selection:

This section describes the main contribution of this thesis, which is a method for how to select test cases for regression testing of multi-tasking real-time systems based on log files extracted during run-time. Our method of regression test selection analysis uses static analysis of the original and the modified programs combined with dynamic information collected from the execution of test cases in the original system.

In this thesis, we use the following abbreviations:

S: original real-time system S': modified real-time system T: test suite for system S t : test casetT|tT

T': a subset of T L : set of log-files

lt: lt L|ltL, ltis log file of test case t

W : set of tasks in real-time system S w : wW task in real-time system S c : cW changed task in the new system S'

3.1 Regression Test Selection based on run-time logs

In this section, we present the general idea behind our method testing for performing regression test selection in multi-tasking real-time systems based on run-time information. As started above, traditionally, a test case contains id, input and an expected output, <id,

input, expected-output>. In this thesis, we have added another element. For each test case,

there is a log lt of events information which were invoked or occurred last time the test case

was executed. Hence, our definition of a test case includes id, input, expected output, logid, <id, input, expected-output, lt>. Basically, the log file displays which tasks and other events

that have been executed in the test case. Events in the log include task activations or deactivations, semaphore sent and received calls, signals etc. Simplifying the these work and avoiding affecting the execution behavior of the system as little as possible, we have focused on three events: Task Activation, SemaphoreTake, and SemaphoreGive. Through our following analysis; we are able to determine which test cases that do not need to be re-tested.

(14)

Typical test case and regression test:

Typical test case contains: <id, input, expect output> Typical regression test:

Given a program P The modified version P’

A test case t -used to test previous program P

Find a way, making use of t to gain sufficient confidence in the correctness of P’

Goal: identify all no-obsolete tests in t that execute changed code with respect to P and P’ Regression test for this thesis:

Test case: A log-file ltis added in my method. <id, input, expect output, log-file>

Regression test: Given a system S

And the modified version system S’

A test case t which was used to test previous system S A log-file ltof test case t

Find a way, using ltin t to gain sufficient confidence that t does not affect S'

Goal: identify t in the no-obsolete tests of T does not affect the modified system S' Therefore, t will not be re-tested.

In the remainder of this section, we will describe how to perform regression test selection based on the program code structure together with log information collected during system run-time.

(15)

Figure 3: Regression Test Selection Method

In figure 3 we present the general idea of this project. Basically, after executing each test case, a log file is extracted. Based on whether the modified task or other events information are found in the log file, we perform the regression test selection according to four scenarios.  Scenario 1: Changed task C is not in log of test case t. Solved by the black-box regression test selection, as presented in Section 3.2.

 Scenario 2: Changed task C is in log of t and C does not include any semaphore calls.

Solved by the white-box regression test selection, as presented in Section 3.3.

 Scenario 3: Changed task C is in lt and semaphore is in lt . Solved by the white-box

regression test selection.

 Scenario 4: Changed task C is in lt and C includes at least one semaphore call, but

semaphore is not in lt.. Solved by the white-box regression test selection.

3.2 Black-box Regression Test Selection

This section presents the black-box regression test selection method, and provides an example to explain how it works.

(16)

3.2 .1 Black-box Regression Test Selection method

In the black-box method, we try to determine whether we need to retest test cases or not, without considering the internals of the system. As described earlier, each test case has its log file, and when test cases are run to test the real-time system, several tasks are executed. The corresponding events information can be captured using run-time instrumentation

In this step, we should prove that test case t does not need to be re-tested if we cannot find changed task C in lt. Basically the test case t is designed for specific purpose during software

testing. Consequently, when we run t, several related tasks w w_l {w|w l_t}

t  

are executed in the system, and each executed task w is logged in lt. So test case t has an

associated log file, describing the tasks that were exercised when test case t was last run. If one task cannot be found in the log file, then the task is not executed when running the test case. Therefore, if we do not find modified Task C in ltthe log file of test case t, then we do

not re-test test case t.

By this analysis, as seen in figure 4, we have shown how to handle Scenario 1.

Figure 4: Black-box regression test selection

3.2.2 Example for Scenario 1:

As an example, consider a system with tasks W, tasks w are executed to be tested by a test suite

T , tasks wiare found in the lt when test case tjis executing, as seen in figure 5.

Test cases: t1....t4

(17)

Figure 5: Relation between test cases and tasks. For example when test case t1is running, task

w1and task w2are executed

If we modify task w3, then every test case t in test suite T should be checked whether it should

be put in the test suite T', for the test case t1, w3is not executed when run t1, w3just affects

test case t2and t3, other test cases are not affected by the modified task w3either.

Therefore, for the test case t, if task C cannot be found in the log of test case t, then we do not re-test test case t, because when testers run test case t, task w is not executed.

3.3 White-Box Regression Test Selection Technique

In the previous subsection, we presented how to use the black-box regression test selection to analyze the situation described by Scenario 1. But when task C is in lt, then the regression test

selection needs to consider the inner-structure of C in order to determine if the changed part of the changed task is executed by the test case. In this section, we will discuss how to analyze when the task C and event- semaphore are in the log file.

First, considering Scenario 2, if the analysis of the inner structure of C reveals that there are no semaphore calls made by this task, we have no further information to guide us in the regression test selection process, and t needs to be re-tested (i.e,tT'). This solves the

(18)

For Scenario 3 and 4, the situation is a bit more complex. We need to analyze which parts of the inner structure of C that are affected by the changes. For this purpose, we make use of Thomas Ball’s control-flow-based regression test selection (CRTS) algorithm [22] and extend it. Hence, we first give short description of this work.

3.3.1 The intersection Graph

In this subsection, we describe the intersection graph that will be used in handling Scenario 3 and 4. As a simplification, we assume that each task in the system can be represented by a control flow graph. Here, a control flow graph, G= (V, E, s, x) is an executable representation containing a set of vertices V, a set of edges E, and a unique entry vertex s, and an exit vertex

x. Each vertex vV presents a basic block in the code of the task, each edge eE presents

a transition between two vertices.

Normally, a CFG is associated with (i.e. generated from) a single function. A program or task consisting of several functions then also have several CFGs. An additional call-graph data structure can then be used to show how the different functions may be calling one another. An alternative is to use a so called inter-procedural CFG. In such, all the function CFGs and the call-graph have been merged into one large data structure." [22]

task1( ) { get_sensor_value( ){ Calculation( ); // A return value; //B } Calculation( ){ return value; //C } }

(19)

CFG for get_sensor_value--nodes: start, A, B, exit edges: start->A, B->exit CFG for Calculation --nodes: start,C, exit edges: start->C,C->exit

CFG for task1 --nodes: get_sensor_value, Calculation edges: get_sensor_value->Calculation

If a task is modified, then we have two different control-flow graphs G and G', where G represents the old version of the task, and G' represents the new version of the changed task. As described by Ball [23], the intersection graph shows us the difference between G and G', Basically, an intersection graph I(G,G') is a combination of G and G', where all paths including changes end up in a reject exit vertex; all paths that do not include changes end up in an accept exit vertex. If accept path is passed when we run the test case, then the test case does not need to be rerun; otherwise, the test case passes the reject path, then it should be rerun. An example can be seen in figure 6.

Figure 6 :Intersection Graph of G and G'

For more information of the intersection graph, we refer to the original paper by Ball [23]. However, how can we know which path the test case pass, in the next subsection, we will show how to use the intersection graph to handle Scenario 3 and 4.

3.3.2 Regression test selection using intersection graphs and semaphores

If the changed task C is in ltand C contains semaphore calls within the code, according to the

semaphore whether is in lt or not, we try to handle Scenario 3 and 4.

Scenario 3: Changed Task C is in ltand semaphore is in lt

Intersection graph of G and G'

(20)

Solution: If all paths in the intersection graph that passes the recorded semaphore call lead to accept, then , otherwise

If the semaphore is just in paths that lead to accept in the intersection graph, then t is not included in T’, since when test case t is running, it just passes the unchanged path in the modified program; otherwise, t is in T'. Assuming Scenario 3, an example where t is not in T' can be seen in figure 7.

Figure 7: Scenario 3

Scenario 4: Changed task C is in lt and C includes at least one semaphore call, but no

semaphore call is included in lt.

Solution: If all paths that lead to reject passes the recorded semaphore call in the intersection graph, then tT', otherwise tT'

It means that the semaphore is the first basic block in reject path in the intersection graph, Assuming Scenario 4, an example where t is not in T' can be seen in figure 8.

(21)

Figure 8: Intersection graph for Scenario 4

4. Case Study in VxWorks

As a case study and a proof of concept, we have used VxWorks as the real-time operation system (RTOS)to run the multi-tasking real-time system for our proposed method for regression test selection.

For this purpose, we have made use of the following VxWorks tools:  Wind River Workbench 3.0 including:

 VxSim-simulator, since without target, it is used to run example application system in it

 System Viewer --produce log-file of the test case

4.1. VxWorks

VxWorks is a RTOS [20], and has been developed by WindRiver Systems since 1985. At the time of this thesis, the latest stable release version is VxWorks 6.7, a special RTOS designed for in embedded systems. The key features of VxWorks are the following:

 A multi-tasking kernel based on preemptive and round-robin scheduling  Isolation of user application from kernel through memory protection  Fast inter-process communication

 Error handling

 VxSim RTOS simulator.

In this thesis Workbench 3.0 IDE for VxWorks 6.6 was used. This version is based on a Eclipse framework and project management, and includes the Wind River Compiler and a Wind River GNU Compiler, a debugger, the VxSim simulator, the System Viewer, a Kernel configurator, Host and Kernel Shells for VxWorks, and Run-time analysis tools. Due to hardware availability limitation in this thesis, the VxSim simulator was used instead of a real hardware target. We built and configured the image project, and configured the System Viewer to monitor and analyze activities, then we run the test cases of the real-time system in the simulator and used the System Viewer to log the run time data. In Appendix A, we present

(22)

time data by the method proposed in this thesis.

4.2. VxWorks Simulator

The VxWorks simulator, VxSim is useful in early stages for application development, before hardware is available. It is fully integrated into the Workbench environment as a target connection. The VxSim is an application which runs on a PC, behavies exactly as the target, and can be completely configured and debugged through a standard interface. For this thesis project, a VxSim simulator was built by the requirement posted by the problem at hand.

4.3. VxWorks System Viewers

The System Viewer [20] is a analyzer for embedded software in the Wind River Workbench, which provides run-time analysis. Through it, we can visualize and troubleshoot complex target activities. Usually, some debugging tools for embedded systems, especially source-level debuggers provide only static information. However, the system Viewer logs activities on a running target, providing information that is dynamic; as it monitors the target system, logs critical information, detects deadlocks, CPU starvation, and determines application responsiveness.

In System Viewer, an event is defined as any action which taken by a task or an interrupt service routine, it may affect the state of the system. The events are divided into system events and that user defined events, they may be task, semaphores call, timestamps, context switches, watchdog timer activities, parameters, calling routines and so on.

In this thesis, we just focus on the following information:

 Context -- the tasks or interrupts executed during a test case run

 Task information: related with context, which contains task name, task identification.

 EventName-- if thereare semaphore calls or release or other type of events in the log, we can use this information to decide which task the semaphore belong to.

 EventId: Related with EventName. We can use the id to know the relationship between tasks.

4.4. Case Study Example Application

In this section, we will present an example application to check Scenario1. As explained in the Scenario 3.1, we will check whether changed task C is in log of test case t. If the changed task

C is not in the log, then we do not need to re-test the test case that created the log file.

(23)

VxWorks traditional Scheduler- Priority-based + Round-Robin, and semaphores are used for inter-task communication.

To explain the method of regression test selection for Scenario 1, we designed an application system. The application system S contains several tasks. We have designed some test cases

(T) to test the system. When the test cases were executed, the System Viewer produced

corresponding log-files. If we changed the system or modified the code inside the tasks, then we call it has a change (C) in the system. In the sample example, we have five tasks phil-1,

phil-2, phil-3, phil-4, phil-5. Then we designed 10 test cases, when we run test case ti, i was

the id of test case, several tasks were executed, and logi was the log file of test case i, was printed which had the information of tasks executed.

As seen in table 1, for example, test case t1is used to test the semaphore communication

between phil-1 and phil-2. When we run test case t1, phil-1and phil-2 are traced in log1. If we run test case t2, tasks phil-2, phil-3, and phil-4 are executed and occurred in the log.

Table 1 Sample: Log of test cases

For example, if we modify the code inside the task phil-1, test case t2 does not need to be

(24)

To summarize, test case ti can be determined by our regression test selection method to be

re-tested, if the changed C is not in the log of test case ti.

5. Conclusion and Future work:

5.1. Conclusion

This thesis has presented a method for regression test selection design for multi-tasking real-time Systems based on static and dynamic information. We have shown how to analyze the modified task and a specific log file of the each test case in the regression test suite, in order to easily and quickly determine whether the test case need to be re-tested or not. Basically, if the modified task is in the log file of the test case, we make use of control-flow analysis based on intersection graphs [19] in order to determine if the changed code is exercised by the test case or not. On the other hand, if the modified task is not in the log file of the test case, it is not selected for re-testing. The bottom line is that test cases exercising only unchanged code should not be re-executed during regression testing.

In the example case study, we made use of semaphore calls, releases as the task event, but the method works as well for mutual exclusion, message queues, pipes, message channels, and signals.

The main contribution of this work is that we brought log file in the test case, it records the task events and context information of every task, we have shown how to make use of this information in the black-box and the white-box regression test selection techniques in order to restrict the number of test cases that require to be re-tested.

5.2. Future work

In this thesis, our work focused on the black-box regression test selection and the control-flow graph inside the task. In real-time systems, several tasks may share the same code, how to handle this will be researched in future. My examiner questioned the scheduling of tasks might be changed if the code inside the task modified. For example, if the changed task has higher priority, the priority is changed, how do we select the test cases, or if large number of lines code is added inside the task, the execution time of the changed task will be longer, the tasks executed by running test case may miss deadline because of time constraints, even the priorities of tasks are not changed, then the task scheduling will be changed. An important direction for future work is that how do we deal with task scheduling changed when we design the test case. Time constraints will be discussed in test case in future.

(25)

References:

1. Damir Isovic. Real-Time System I. Department of Computer Science Mälardalen University

2. Hetzel, B. The complete Guide to software Testing, 2d ed, QED Information Sciences Inc, 1988.

3. Erik van Veenendaal,Editor.Standard glossary of terms used in Software Testing,Version 1.1 (dd. September, 29th 2005) Produced by the ‘Glossary Working Party’ International Software Testing Qualification Board

4. Mary Jean Harrold, James A.Jones, Tonyu Li and Donglin Liang. Regression test selection for JAVA Software, Proc. of the ACM Conf. on OO Programming, Systems, Languages, and Applications (OOPSLA'01)

5. Leung, H.K.N.; White, L, A cost model to compare regression test strategies, Software Maintenance 1991,Proceedings. Conference on 15-17 Oct. 1991Pages(s):201-208.

6. Gregg Rothermel and Mary Jean Harrold, Analyzing Regression Test Selection Techniques, IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL NO. 8. AUGUST 1996

7. IEEE Computer Dictionary - Compilation of IEEE Standard Computer Glossaries, Institute of Electrical and Electronics Engineers / 01-Jan-1991 / 218 pages,ISBN: 1559370793.

8. Giorgio C. Buttazzo, Hard Real-time computing systems, Kluwer Academic Publishers.1997.p1

9. J.Stankovic and K. Ramamritham. Tutorial on Hard Real-time Systems. IEEE Computer society Press, 1988.

10. Todd L.Graves, Mary Jean Harrld, Jung-Min Kim, Adam Porter, Gregg Rothermel, An Empirical Study of Regression Test Selection Techniques, IEEE, August, 2000

11. Gregg Rothermel and Mary Jean Harrold, Empirical Studies of a Safe Regression Test Selection Technique, IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL 24, NO.6, JUNE 1998.

12. Autoliv's Frontal Airbag System,

http://www.autoliv.com/wps/wcm/connect/autoliv/Home/What+We+Do/Airbags/Frontal%20 Airbags

13. Regression testing MSDN,

http://msdn.microsoft.com/en-us/library/aa292167(VS.71).aspx

14. Anders Pettersson, Analysis of Execution Behaviour for Testing of Multi-tasking Real-time systems, Department of Computer Science of Engineering, Mälardalen University, ISBN 91-88834-13-1,P27

(26)

Proc. Conf. Software Maintenance-1993, pp. 348-357,Sept.1993

16. H.K.NL.e ung and L.J. White, A Study of Integration Testing and Software Regression at the Integration Level, Proc. Conf. Software Maintenance--1990, pp. 290-300, Nov. 1990 17. A. von Mayrhauser, R.T. Mraz, and J. Walls, Domain Based Regression Testing, Proc. Conf. Software Maintenance-1994, pp. 26-35, Sept. 1994

18. Atif M.Memon, Fundamentals of Software Testing,

http://www.cs.umd.edu/~atif/Teaching/Spring2008/Lectures/15.pdf

19. Thomas Ball, On the Limit of Control Flow Analysis for Regression Test Selection, Lucent Technologies, Bell Laboratories, USA, ACM , Pages: 134 - 142 , 1998

20.Wind River System VxWorks Document, WindRiver Systems, www.windriver.com

21.Frances E.Allen, Control flow analysis, ACM, pages:2, 1970

22.Thomas Eisenbarth, Rainer Koschke, and Gunther Voge, lStatic Trace Extraction,IEEE, 29 Oct.-1 Nov. 2002 Page(s):128 - 137

Appendix:

This Appendix describes how does System Viewer work and how to configure it.

The main System Viewers tools are System Viewer Configuration, Triggering, Event Receive, Log Viewer and Analysis Suite Viewers.

For the System Viewer Configuration, we can use this to configure what we want to be logged, how to upload the log from the target to the host and how to view analysis data. Triggering tool is used to specify when to start and stop logging. The log data is collected and upload from the target by a socket listener tool called Event Receive. If we want to analyze the logged data in text format, file logs can be export as text format from the Log Viewer. A.1.

System Viewer Architecture

In this section, we present the architecture of System Viewer [20] and how to get the text log file.

 The System Viewer configuration tools are connected to the simulator, then

information is read from the simulator. The following figure is the architectural overview of it.

(27)

Figure 12: System View Architectural Overview

 System Viewer log file has several extensions, a .wvr extension, which is a raw file with log data only; a .wva extension, which is an analysis file, from its information we can know how the previous viewing session was configured. Then we can export text log file from System Viewer.

A.2.

System Viewer Configuration and Logging Information- event

This sub-section explains how to configure and generate System Viewer log files which is used in example study case.

In order to successfully configure System Viewer log we need prepare VxWorks Image Project, Simulator.

A.2.1 Creating VxWorks Image Project

We should configure a VxWorks Image Project to support System Viewer functionality in Wind River Workbench.

First, from File option in menu select New and then select VxWorks Image Project. In New VxWorks Image Project window, select simpc as a board support package and then click on Finish.

(28)

Figure 13

Next, launch simulator, new a VxWorks 6.x Simulator connection, select the image project that

one just was built.

Figure 14

Then, image created in previous step with VxWorks Image Project, when we have done, save the new kernel configuration, rebuild the image project.

(29)

First, from the Target option, select Wind River VxWorks 6.x Simulator Connection, next to VxWorks Boot parameters to select Custom simulator, which is the vxWorks file in default

file the image project just created. Then click on finish.

Figure: 15 A.2.3 System Viewer Configuration

After rebuilding image kernel, connecting Vxsimulator, right click Vxsim, entry System Viewer Configuration, select event logging level, choose Context Switch; next select the log collection mode, the log will be saved on the target in a buffer, here Continuous Upload was chosen; then, a method the target will use to upload logs, choose Socket Via TSFS, name the file name for Event Receive Configuration, and apply it.

(30)

Figure 16

As a final step, start System Viewer logging, then we can see message received in Event Receive Window, run the real-time application system that we want to analyze.

Figure 17

When we stop logging, a new window comes out from the Workbench, as seen on the next firgure.

(31)

Figure 18

Regression Test Selection in Multi-TaskingReal-Time Systems based on Run-Time logs

School of Innovation, Design and Engineering

MÄLARDALEN UNIVERSITY