Effectiveness of operational profile-based testing

(1)

School of Innovation Design and Engineering

V¨

aster˚

as, Sweden

Thesis for the Degree of Master of Science (60 credits) in Computer

Science with Specialization in Software Engineering 15.0 credits

EFFECTIVENESS OF OPERATIONAL

PROFILE-BASED TESTING

Olzhas Zhakipbayev

ozv19001@student.mdh.se

Aisulu Bekey

aby19001@student.mdh.se

Examiner: Daniel Sundmark

M¨

alardalen University, V¨

aster˚

as, Sweden

Supervisors: Wasif Afzal

M¨

alardalen University, V¨

aster˚

as, Sweden

(2)

Abstract

The operational profile based testing is currently not a well-studied topic and there are no spe-cific instructions for writing test cases for testing the program. In our thesis, we presented our idea about on the basis of what data test cases can be written. Also, in order to show the effectiveness of operational profile-based testing, we additionally described the equivalent partitioning testing technique. The software for this experiment was taken from the open-source SIR repository. We have selected software: “Account”, that was tested by two different testing methods. The test results of both techniques were compared and it was determined that the operational profile based testing technique is more effective.

(3)

1. Introduction

Operational profile based testing is supposed to generate test cases that might show characteristics of usage statistically. Thus, execution of them helps to estimate the level of the reliability in particular parts of software, since the reliability and cost of the software directly depend on the operating profile [1]. The properly chosen method of testing may guarantee a high percentage of reliability of a software.

The purpose of equivalence partitioning testing is to divide the input data into parts of equiv-alent data, then write test cases based on this separation. This testing technique spends less time on testing because test cases are not built for each input data but each block. This test method is black box testing, when the tester does not see the code, but can only see the input data. This method can also be applied to all levels of testing such as unit, integration, system, etc.

To achieve the objective of our thesis work, we selected a specific software “Account” from the SIR, that is going to be the SUT (software under test). Also, we will investigate operational profile-based testing and equivalence partitioning testing techniques. Furthermore, the selected software will be tested through each investigated testing techniques. After the software is tested, we will summarize the result of the testing in a statistical view. The statistics will be interpreted by the number of detected faults by the time.

Based on the probability occurrence of each operation in profile, the effectiveness of failure detection will be measured by the number of failures that occurred in each operation. The testing technique that finds the largest number of failures will be identified as a more effective testing technique for detecting failures. During this measurement for both testings, techniques will be given the same amount of time.

Operational profile based testing helps to understand why software fails. Through this technique attempting to quantify the reliability of software in terms of failure occurrence probability.

The operational profiles based testing technique aims to select test inputs that may expose the capability of failure’s occurrence. Before starting testing by this technique, we must determine the operational profile of a software. As is described above, the operational profile is a set of independent operations that the software performs. Operational profiles usually consist of these following things:

• customer profile - is a group or person that is purchasing the software. Also, it may consist of an array of independent customer types.

• user profile - is a set of users type that will operate the system and their capability of using the system.

• system-mode profile - the way that the system can operate. It shows the list of system modes and the capability of their occurrence.

• functional profile - after the determination of a system mode list, the list of functions must be provided that are performed during each mode. For every function in the list, the occurrence probabilities should be assigned.

The operational profile is a set of disjoints operations with the occurrence probability of each of them. The operation profile is quantified by computing the probability of each operation, which is based on evolving the functional profile. So each operation has its percentage of probability of occurrence, for instance, if operation A has 15% of occurrence and operation B has 85% then the operational profile would have a view as (A,15% and B,85%).

After the determination of the operational profile, we should begin with test planning based on the operational profile.

The experiment research method will be used in this thesis. We consider it as the most appro-priate method for this topic. To get a good experiment design, we need to have enough data to study. To compare and determine the most effective techniques, a statistical analysis of the results will be performed. To clarify the most effective testing method, the “Account” software from the SIR will be tested through the operational profile-based testing and equivalence partitioning testing techniques.

(5)

A null and alternative hypothesis will be constructed for each question under study. The null hypothesis states that there is no real difference in the effectiveness of two different testing techniques. An alternative hypothesis rejects the null hypothesis and determines that one of the testing techniques is more effective than the second.

(6)

2. Background and motivation

Now-a-days software systems play a very significant role in our daily lives. Software systems are extensively used by end-users. A quality software product should have many characteristics such as reliability, maintainability, availability, modifiability, performance, security, usability and testability, etc. Among these, our focus is on reliability and the fact that the presence of faults can negatively affect software reliability. Reliability issues in software-based systems is a well-known fact. Reliability is ruined by having faults in software that causes failure of the system [2].

The reliability of software is one of the important aspects that the software should have. It is extremely necessary to find out the probability of failures within the specific conditions because the reliability of software should respond to all requirements, and during the usage of software, failures must be minimized. A software operational profile is a set of operations that the software performs and the probability of occurrence of these operations [3]. The reliability of software and software operational profiles are quantitative characterizations of a software. The focus of the current thesis is to investigate the effectiveness of operational profile-based testing compared to the equivalence partitioning testing technique. The effectiveness of each method mentioned above will be measured in terms of the number of faults detected.

(7)

3. Problem formulation

In this thesis work, we are going to investigate and compare the operational profile-based testing technique with the equivalence partitioning testing and conclude in terms of effectiveness.

1. Investigate operational profile based testing technique 2. Examine the equivalence partitioning testing technique

3. Testing the open-source software “Account” from the SIR (Software infrastructure Reposi-tory) [4] through the techniques mentioned above.

4. Compare the results of the investigation and testing, to show the best test technique in terms of fault detection.

We will investigate the following RQ in this thesis:

RQ. What is the effectiveness in terms of faults identification of operational profile based testing technique when compared to equivalence partitioning testing technique? During the study and testing of various techniques and methods, using the results of statistical analysis, we can identify the advantages and disadvantages of each technique, including operational profile-based testing. That helps us to investigate the research question.

(8)

4. Related Work

In the article [5] authors developed an adaptive software testing approach that was combined by operational profile and coverage spectrum techniques called covrel. The objective of this work is to improve the reliability of software under test by covrel. They evaluate covrel with the subjects from SIR (software infrastructure repository) and compare the evaluation with traditional operational and coverage testing. Besides, they also, provide a prototype of the implementation of covrel for further usage. In the article they come up with an overview of covrel strategy and description of its approach, then the experiment was conducted. The results of the experiment were reported to answer the research question of whether the covrel is more effective at reliability improvement that the operational testing.

The operational approach is one of the new approaches to measure code coverage. In this article [6], research has shown that operational coverage has surpassed traditional coverage. The novel approach “operational coverage” can provide a good stopping rule. It also provides the maximum level of reliability, as it focuses on frequently used operations.

Software testing is taking a high percentage of the cost from total software development. Test cases are created to test all functions and operations in the software. These test cases are created based on the work profile for test performance. It also improves reliability and speeds up the development process [2].

Miranda and Bertolino in their article examined the correlation between the operational cov-erage of branches, operators and functions and the probability of failure of the next input test. This article [7] explores a rather interesting question that challenges many researchers in the field of testing “Is branch coverage a good measure of testing effectiveness?”. This article also discusses stopping rules for testing. Coverage measures were investigated and studied depending on the frequency of use of certain functions and operations by the user. Their coverage research showed that operational coverage is better correlated than traditional coverage.

A case study was conducted by Ali-Shahid and Sulaiman [8]. At that work the improvement of reliability was studied by generating test cases based on the testing profile and the working profile of the software. The results of the study revealed that the framework based on Software Operational Profile (SOP) and Testing Profile (TP) provides more than 40 percent of the input space coverage

One of the testing approaches for improving the reliability of software is combinatorial testing, which is described in the paper [9]. Various approaches to combinatorial testing were also consid-ered. In the paper, it was found that combinatorial testing maximizes the detection of errors and minimizes the number of test cases.

In the article [10] written by J´unior et al. conducted a study on the different uses of the software, the misalignment between the SOP (Software Operational Profile) and the tested parts of the software, the occurrence of failures in the unverified parts of the SOP. For this study, they defined the term “test profile”. The study also determined that SOP, in addition to reliability, is also related to the planning and implementation of testing activities.

In addition to software development and its reliability assessment, you also need to remember that there are limited resources for the project. One of the expensive processes is testing. For better allocation of resources during software testing, in an article by Ukimoto, Dohi, and Okamura [11], it was decided to use an operating profile. Since the reliability and cost of the software directly depend on the operating profile.

In the chapter “Improving Web Application Reliability and Testing Using Accurate Usage Mod-els” of a book “Software engineering research, Management and application” [12] were examined the impact of a Markov Operational profile on reliability, test coverage, and test efficiency. Case study were conducted for the quantity assessment of the impact. Markov operational profile is a usage model for large software, also involving the web applications.

(9)

5. Method

For our thesis work, a controlled experiment is the most appropriate research method. First, we should make a plan of how we will investigate and obtain the data regarding the test techniques and software that will be a subject of testing. The obtained data will be analyzed and interpreted from a statistical viewpoint. Then it will be presented and concluded in terms of efficiency and effectiveness comparison. The experiment will be conducted through the given steps: Context selection, hypothesis formulation, variable selection, selection of subjects.

1. Find at least 2 existed testing techniques. 2. Investigate each one in detail.

3. Compared with operational profile based testing, determine all strong and weak sides of effectiveness.

5.1. Experiment Process

1. Scope Experiment The point of the analysis of operational profile based testing and equiv-alence partitioning technique is to determine the effectiveness of each of them in regards to software under test. Concerning their time spent and the number of failures detected, from the reliability of a software. The entities that are studying in the experiment are an eval-uation of the impact of two different techniques on the reliability growth of software. The quality focus of the experiment is the reliability of the software.

2. Planning For controlling the experiment the plan should be built. Each of the steps of the plan should be fully completed. Further, each step is described in detail.

• Context Selection The operational profile based testing technique is compared to the equivalent portioning testing technique in regards to effectiveness, such as time spending and the number of failures is detected. The operational profile based testing technique is a black-box technique that selects test cases having the largest impact on failure probability in operation, as such, it is considered well suited when reliability is a major concern. The equivalence partitioning testing technique is a software testing technique that divides the input data of a software unit into partitions of equivalent data from which test cases can be derived. Furthermore, equivalence partitioning testing is con-ducted manually, meanwhile, the operational profile based testing technique is should be developed.

• Hypothesis Formulation A null hypothesis. The state of the null hypothesis is that op-erational profile based testing not more effective than equivalence partitioning testing, and in that case, the choice of a technique will be based on the simplicity of conduction. An alternative hypothesis. The state of the alternative hypothesis is that the opera-tional profile based testing technique is more effective and finds more faults than the equivalence partitioning testing.

• Variable Selection In the course of our experiment, variables (dependent and dent variables) will be determined on the basis of the research questions. The indepen-dent variable is a variable which we can control and manipulate during the experiment. According to the RQ, we can control the methods of chosen techniques that will be used in testing. The variable that we examine and observe during the study is called the dependent variable. In the experiment, observation was conducted to detect the number of faults during the testing.

• Selection of Subjects We have taken a probability sampling technique in the selection of subjects.

(10)

5.2. Operational Profile

As we already described above, the default operational profile consists of 5 profiles that need to be determined to fully develop the operational profile. The sequence of profiles is represented in Figure 1.

Figure 1. Operational Profile Development

1. Customer profile - A customer profile is an array of many independent types of clients. A consumer is one who purchases a software system. It can be a person, group, or organization, it is anyone who can purchase a product. There are also many types of consumers. Customer type is one or more customers in the group who are going to use the system in a relatively similar way and different from other types of customers.

2. “User Profile is the set of user groups and their occurrence probabilities [1]”. System users are those who directly work with the system without purchasing it, as users also can be a person, group, or organization.

3. System mode profile is a way how the system can operate. It is possible to have one system mode to be used when there are no other modes are used, but it is also possible to have multiple system modes being used. The same function or operation might have occurred in different system modes.

4. Functional Profile - we evaluate functions performed during the usage. Therefore, the proba-bility of occurrence should be assigned for each function. The function list is designed during the requirements phase, and they must be updated if changes occur. “Account” software has 6 functions. Shown in Table 1.

(11)

5. Operational Profile - operations are used at the implementation level while a functional profile is a task that is used for design. Usually, the number of operations is higher than the number of functions. A single function can be implemented by multiple numbers of operations in the system.

6. Test Selection - Test cases are selected by run executions of operations that define different states of functions.

5.2.1. Software under test

Software-artifact infrastructure repository(SIR) is an open-source repository of software-related ar-tifacts meant to support rigorously controlled experimentation with program analysis and software testing techniques, and education in controlled experimentation.

For evaluating the effectiveness of operational profile based testing, we select the “Account” software from the SIR (Software infrastructure repository). The description of the software is provided below in Table 1.

Software “Account” № Subjects Descriptions 1 LOC 108 2 Classes 1 3 Methods/Functions 12 4 Fields 4 5 Statements 7 6 Notify all 0

Table 1. Software description Below we provide the description of the functions

1. Account – the object that consists of three variables, are: balance, currency, and the limit withdrawing of money.

2. Deposit - Allows the customer to add the amount of money to the balance of a particular account.

3. Withdraw – Allows the customer to take away the amount of money, which does not exceed the amount of balance and the limit of withdrawing.

4. TransferToAccount – Allows the customer or employee to transfer a particular amount of money from one account to another.

5. ConvertToCurrency – Allows the employee to change the current currency at the particular account, with according to change of the balance by multiplying it to the coefficient of the currency.

6. WithdrawAll – Allows the customer to take away the maximum possible amount of money from the account. It should not exceed the limit of withdrawing.

7. SetBalance - Allows the administrator to set the initial balance (usually zero) at the particular account

8. GetBalance - Allows the employee or administrator to get the information regarding the amount of balance at the particular account.

9. SetMaxOverdrawn – Allows the administrator of the system to set the limit of withdrawing money simultaneously.

10. GetMaxOverdrawn – Allows the customer or employee to get the information regarding the limit of withdrawing money simultaneously.

(12)

11. GetCurrency - Allows the customer or employee to get the information regarding current currency that uses at the particular account.

12. SetCurrency - Allows the employee or admin to set the currency that will be used at the particular account

5.2.2. Developing the operational profile based on the software Account

Customer profile - Regarding our software “Account”, we identified 4 customers: bank, government, tax office, pension fund. So these customers should be given approximate occurrence probability. In Table 2 we set them a capability by percentage.

Customer profile

№ Customer Probability of occurrence 1 Bank 45 percent

2 Government 15 percent 3 Tax office 20 percent 4 Pension fund 20 percent

Table 2. Potential customers

User profile - Different potential users that can use the software were defined. Also, the occur-rence probability has been given to them to clarify who uses the software more often. It shows in Table 3.

User profile

№ User Probability of occurrence 1 Client 30 percent

2 Bank employee 50 percent 3 Admin 20 percent

Table 3. List of users

System mode profile - Software is executed in three different modes since there three different users, and for each of them the different functions and interfaces are provided.

System mode profile

№ System mode Probability of occurrence 1 Admin mode 20 percent

2 Client mode 30 percent 3 Employee mode 50 percent

Table 4. List of system mode

Functional Profile - in the description of the software, twelve different functions were provided and the source code was included. According to the description of each function, we set an ap-proximate probability of occurrence. List and appropriate capability of them are provided in Table 5. Account, deposit, withdraw, transfer, currency, withdrawAll, setMaxOverdrawn, getMaxOver-drawn, getValidCurrency, setValidCurrency, setValidBalance, getValidBalance.

(13)

Functional profile

№ Functions Probability of occurrence 1 Deposit 100 percent 2 getValidBalance 95 percent 3 Withdraw 90 percent 4 Transfer 80 percent 5 ConvertToCurrency 75 percent 6 WithdrawAll 45 percent 7 Account 35 percent 8 setValidBalance 30 percent 9 setMaxOverdrawn 25 percent 10 getMaxOverdrawn 20 percent 11 setValidCurrency 15 percent 12 getValidCurrency 10 percent

Table 5. List of functions

An implicit functional profile represents a set of values of each key input with respect to occurrence probability. For each user of the user profile, an implicit functional profile is represented in the next Figures 2, Figure 3, Figure 4.

Figure 2. Example of an implicit functional profile (Admin)

(14)

Figure 4. Example of an implicit functional profile (Employee)

Operational profile. In determining the operational profile, the main tasks are the list of operations and determine the probabilities of occurrence. To identify the operations list, need to perform three steps:

1. Divide execution into runs. 2. Identify input space.

3. Partition the input space into operations.

This list of operations can be retrieved from the functional profile. In the functional profile, 12 different functions were presented by “Account” software. One function can contain many operations. Accordingly, the number of operations is more than functions.

1. Divide execution into runs. Run is a measure of program execution time in a particular environment, usually based on a user-oriented task. Also, runs characterize operations. The concept of runs exists in order to be able to divide performance and characterize patterns of the system’s use [1].

2. Identify input space. In this step, we will determine the number of different values and combinations of inputs that the input space of the software system can take. The number of combinations of inputs and various values can be very large even for such small, not complicated programs like “Account”. By listing the set of input variables involved, we can define an input space [1].

3. Partition the input space into operations. Input space can be separated by selecting ranges and belonging to the same type of runs. Due to cost limitations, in practice, profiles are limited to a few hundred (thousands or more) elements. Because the number of elements proportionally affects the cost of development, the more elements, the greater the cost. In some cases, it is required to reduce the number of elements, by grouping run types into operations, thereby partitioned the input space [1].

(15)

5.2.3. Importance of functions in Account software

We have decided to add a new parameter to split the particular functions into four different groups. The new parameter is importance, so we can divide them in terms of the necessity of usage to keep perform the software. The more importance the operation has then more test cases for it produces. The importance of the list of functions to be used is determined by the necessary operations to perform the software. For instance, in terms of key operations of software, administration of database has a high level of importance while compared to change a photo in a user profile, it does not have the necessary importance to maintain the performance of a software.

Customer profile - In this level, we identified 4 customers: bank, government, tax office, a pension fund is the same as in Table 2. So for these functions should be given importance according to the software. In Table 6 we set them a capability by rank.

Rank is a level of importance. In our case, the importance level is measured from 1 to 5. We have decided to add a new parameter to split the particular functions into four different groups. The new parameter is importance, so we can divide them in terms of the necessity of usage to keep perform the software. The more importance the operation has then more test cases for it produces.

• 1 is a low level of importance and you can do without it.

• 5 is a high level of importance and without this function, it is not possible to work.

The importance is determined in terms of system performance and function. The importance of a function is determined by how much the software can do without this function. For example, if this function does not exist at all, or if it does not work. Whether the software can maintain its performance without this feature.

Customer profile № Customer Importance 1 Bank 5 2 Government 4 3 Tax office 3 4 Pension fund 2

Table 6. Potential customers

User profile - Software can be used by different potential users. In this part, we identified the most important users of our software.

User profile - Software can be used by different potential users. In this part, we identified the most important users of our software. They are illustrated in Table 7.

User profile

№ Users Importance 1 Client 5

2 Bank employee 4 3 Admin 4 Table 7. List of users

System mode profile - At this level, the software mode changes depending on the user. Each mode has its own percentage of importance in software. They are represented in Table 8.

System mode profile

№ System mode Importance 1 Admin mode 4

2 Client mode 5 3 Employee mode 4

(16)

Functional Profile - Twelve different functions were described in the software. According to this list, we set the approximate importance of each function. As shown in Table 9. Account, deposit, with-draw, transfer, currency, withdrawAll, setMaxOverdrawn, getMaxOverdrawn, getValidCurrency, setValidCurrency, setValidBalance, getValidBalance.

Functional profile № Functions Importance 1 Account 5 2 Deposit 5 3 getValidBalance 5 4 Withdraw 4 5 Transfer 4 6 ConvertToCurrency 3 7 WithdrawAll 3 8 setValidBalance 2 9 setMaxOverdrawn 2 10 getMaxOverdrawn 2 11 setValidCurrency 2 12 getValidCurrency 1

Table 9. List of functions 5.2.4. The test tasks

To test our software, we must go through several steps from planning to test closure.

1. Planning. For a test plan to be good, it must be concise and focused. When planning tests, you need to consider various factors and also answer questions such as: What is the main purpose of testing? What is included and what is outside the scope of testing? What limitations can affect testing? What operation support is required for the test environment? [13] The test plan is written for three reasons:

(a) Focus on important topics.

(b) Communication among themselves members of the project team and other interested parties.

(c) Helps manage change. As the project develops and the situation changes, we adapt our plans [14].

Regarding our software “Account”, the purpose of our testing is to compare two techniques, first, operational profile based testing and the second one equivalent partitioning testing to determine a more effective testing technique.

Testing the functions and operations of our software will be included within the scope of our testing. The interface of our software will be outside of our testing because it does not have an interface.

During product testing, there are usually have certain limitations that may apply to the product itself and its requirements, as well as more indirect things such as time, cost, testing tool, and so on. In our case, there is no limitation in cost and a strictly defined time, since our main goal is not to improve software through testing but to compare two different testing techniques using the software. The limitation regarding the testing tool, manual and automated testing will be more fully defined during the implementation of testing.

To test our software, we will use the Java Eclipse IDE (Integrated Development Environ-ment). More detail about it is described in the design section.

2. Analysis

Test analysis describes what should be done tested, in terms of test conditions. Since we are comparing the two different testing techniques, we should determine the test objectives of each technique. Test objectives of operational profile based testing technique:

(17)

(a) Investigate the software “Account” from SIR (b) Determine the operational profile of the software

(c) Test cases generation based on the operational profile Test objectives of equivalence partitioning testing technique:

(a) Determine the boundary values

(b) Set the specification coverage (output partitioning)

In order to start the test analysis, we should make a test basis. Test basis - is information on which test cases are based on. Regarding our chosen techniques we should collect the test cases based on the specification of a technique. For instance, an operational profile should be built from the software, and then the test cases are supposed to be generated from it. In equivalence partitioning, we should determine the boundaries of input data.

Steps that should be done before the test design begins:

(a) Build the correct behavior of the software. This step will be based on the investigation of “Account” software from SIR.

(b) Identify the expected results before the test runs. 3. Design

Test design is a process that describes how the test should be done. In order to conduct the test process in a proper way, first, we have to determine the testing tools that going to be used. As a programming language, we have chosen Java, as a platform IDE(integrated development environment) Eclipse will be used. The testing process will be implemented through the JUnit testing framework.

We are going to split the list of functions into groups by two parameters, first it is Prob-ability and second is Importance. Each group will represent a list of functions with their characteristics. This graphic card is represented in Figure 5.

• Group 1 - Will have consisted of functions of the low probability of appearance and the low importance of usage.

• Group 2 - Will have consisted of functions of the high probability of appearance and with the low importance of usage

• Group 3 - Will have consisted of functions of the high probability of appearance and with the high importance of usage

• Group 4 - Will have consisted of functions of the low probability of appearance and with the high importance of usage.

(18)

Figure 5. Graphic card

Figure 5 illustrates a graph divided into 4 groups according to two criteria (probability and importance). Our software functions cover groups 1, 3, 4. Since we do not have a function with high probability but low importance, the group corresponding to this is empty in our graphic.

(19)

5.3. Equivalence partitioning

In this software testing technique, input data is divided into equivalence classes. Therefore, this method is also called the equivalence class partitioning. Equivalent partitioning is black-box test-ing, where we do not know the internal design of the system. Therefore, for testtest-ing, we use only the input values. The presented testing method by dividing into different equivalent classes helps to reduce the number of test cases. Classification occurs by selecting similar input data in one class. If a program does not handle elements in a class, in the same way, the equivalence class should be divided into smaller equivalence classes [15]. When testing divided equivalent classes, boundary values are also defined, and these values are tested in two different classes, and values between the two classes are defined as boundary values. There are two types of equivalent classes, valid program inputs represent valid classes, and erroneous input values represent invalid classes [15]. Figure 6 and Table 10 shows an example of equivalence partitioning.

Figure 6. Equivalence Partitioning example [16] Equivalence Partitioning

Input Valid Equivalence Classes Invalid Equivalence Classes N integer number -99≤N≤99 [-99, -10],[-9, -1],[-9, -1], 0, [1, 9], [10, 99] <-99,>99, Malformed num-bers {12-,1-2-3, ...}, Non-numeric strings, {junk, 1E2, $13}, Empty value

Table 10. Equivalence Partitioning example [16] 5.3.1. Experimental Methodology

This section describes the experiment and the theory behind it. The experiment consisted of five stages: failure analysis, creating sets of test sets, tests run, and, finally, compare equivalence partitioning results with operational profile based testing. An illustrative diagram is shown in Figure 9.

(20)

Figure 7. Experimental Methodology [17] 1. Failure analysis

The selected software already had information about what kind of bugs exist in a program that could be detected during testing. Data on exact bugs was the main key for searching and studying these bugs and how they manifest themselves. Knowledge about the sort of bugs was necessary for any changes in the system and to create test cases. At the stage of failure analysis, the code was reviewed, bugs were studied and the report was written (what kind of bugs program have and how they occur, the sample test cases were also designed to identify these bugs.

2. Creation of test case sets

Before creating test case sets, we must first design test cases by using the equivalence parti-tioning is a two-step procedure: (1) identify equivalence classes, and (2) define the test cases [15]. To create a test case First, you need to study the requirements and correctly identify the input area of the program, which helps to determine the input and output variables. Secondly, divide the input domain into different categories, wherein each category partici-pants will be grouped who have at least some relation to each other. Third, When the test cases from a category display the same behavior, we based on the expected behavior, divide the set of values of each variable into disjoint subsets that span the entire area. Fourthly, equivalence classes can be combined using the multidimensional decomposition partitioning approach [18]. Invalid equivalence classes are classes that contain a combination of input that cannot be generated during the test. Some equivalence classes are not feasible due to the restriction of the requirement. The step of creating a set of test cases comes after failure

(21)

analysis. Thus, failures, program code, test technique, and test cases themselves are involved in creating test case sets.

3. Test conduction

In order to conduct testing, we have used the JUnit framework in collaboration with the open-source repository of SIR. SIR allows testing the software with the allocated test suit. The software that has been selected has several defined bugs. Based on the test suites that were prepared by the operational profiles based testing technique, we have tested the software “Account”. After that, we repeat the testing, but already with the test cases that have been build by the equivalence partitioning testing technique.

We derived operational profiles after the investigation of the source code of each software. Based on those operational profiles we generated test suits. Regarding the test suits based on equivalence partitioning, they have been generated manually based on their techniques. Here are the steps how test conducted:

• deployed the software from SIR • source-code was investigated

• based on the operational profile test suits were generated

• based on the manual script of equivalence partitioning technique we made the test suits • mutation analysis and code coverage were conducted

• the testing based on generated test cases were conducted

In Table 11 we provide the order of functions in terms of criterion “Importance”. The list starts from the most important function then ends with the least important. The Table contains the function and its description.

Table 11 Importance

№ Part Descriptions 1 a Valid account

b This function should have consist of any of params double amount, string currency, double MaxOverdraw). Should not be null.

2 a Valid amount should be deposited

b This function deposited an amount to the account. Should not be below zero, or zero and char. Should not be null. The currency in the account should be the same.

3 a Get valid balance

b This function gets a valid balance of the account. Should not be char, but an integer. Should not be null.

4 a Withdraw a valid amount

b This function withdraws an amount. Should not be below zero or zero, higher than BalanceSum, and char. Should not be null.

5 a Transfer an amount to another account

b This function transfers the amount to another account. The transferred amount should not be below zero, zero, higher than the Balance. Should not be null. Can not transfer money to itself. The amount of transferred money from first account should be removed. Another account should add this amount of money. Accounts should have the same currency.

6 a Converter of currencies in the account

b This function converts the currency. Also, it should convert the amount of money in the account regarding the currency. Each currency should have the correct rate. Should not be zero or below zero, and char. Should not be null.

(22)

Table 11 7 a Withdraw all money from the account

b This function withdraws all money from the account. Should not be below zero, zero, null. The account should not be empty(has a negative balance). After withdrawal, the balance should be zero. Should not be higher than the max limit of withdrawing. Currency should be the same.

8 a Set valid balance

b This function sets a balance to the account. Should not be char, but an integer. Should not be null.

9 a Valid number to set max overdraw sum

b This function sets max overdrawn sum. The number should not be below zero or char. The only type of input data is an integer. Should not be null. 10 a Valid number to get max overdraw sum

b This function gets max overdrawn sum. The number should not be higher than the sum that settled in the previous function, should not be below zero or char. Should not be null.

11 a Set valid currency

b This function sets the currency to the account. Should not be an integer, but the char. Should not be null.

12 a Get valid currency

b This function returns a currency that the account has. Should not be an integer, but char. Should not be null.

Table 11. Importance

In Table 12 we provide the order of functions in terms of criterion “Probability of occurrence”. The list starts from the most usage function and ends with the least usage. The Table contains the function and its description.

Table 12 Probability of occurrence

№ Part Descriptions

b This function converts the currency. Also, it should convert the amount of money in the account regarding the currency. Each currency should have the correct rate. Should not be zero or below zero, and char. Should not be null. 6 a Withdraw all money from the account

(23)

Table 12 b This function withdraws all money from the account. Should not be below zero, zero, null. The account should not be empty(has a negative balance). After withdrawal, the balance should be zero. Should not be higher than the max limit of withdrawing. Currency should be the same.

7 a Valid account

Table 12. Probability of occurrence

In Table 13 we provide the list of functions that have been sorted by arithmetical mean of importance and probability of occurrence. The value of each measure might be found in the Tables 5 and 9.

Table 13 Importance/Probability of occurrence

5 a Valid account

(24)

Table 13 b This function converts the currency. Also, it should convert the amount of money in the account regarding the currency. Each currency should have the correct rate. Should not be zero or below zero, and char. Should not be null. 7 a Withdraw all money from the account

Table 13. Importance/Probability of occurrence

In Table 14, we have made the test cases based on the Equivalence Partitioning technique, each function tested and evaluated the reliability of software, by finding the number of bugs. The number of bugs we found by this technique is provided in the results section.

Table 14 Test Cases based on Equivalence Partitioning

(25)

Table 14 c Account MyAccount = new Account();

MyAccount.setBalance(200.0); MyAccount.setCurrency(“SEK”); MyAccount.deposit(100.0);

assertEquals(300.0, MyAccount.getBalance()); // checking the amount of balance after the deposit

Account MyAccount1 = new Account(); MyAccount1.setBalance(100.0);

MyAccount1.setCurrency(“EUR”); MyAccount1.deposit(100.0);

assertEquals(100.0, MyAccount1.getBalance()); // the balance should be the same as it was before because currencies are different

MyAccount2.setCurrency(“EUR”); MyAccount2.deposit(null);

assertEquals(100.0, MyAccount2.getBalance()); // the balance should be the same as it was before because the // value of amount of deposit is null 2 a Get valid balance

c myBankAccount.setBalance(27.0);

assertEquals(“The value of the account shall be 27”, 27.0, myBankAc-count.getBalance(), 0);

(26)

MyAccount1.withdraw(250.0);

assertEquals(200, MyAccount1.getBalance(), “The balance should stay the same if amount of withdraw is greather than balance”); // the balance became negative

Account MyAccount2 = new Account(); MyAccount2.setMaxOverdrawn(200.0); MyAccount2.setBalance(200.0); MyAccount2.withdraw(200.0);

assertEquals(0, MyAccount2.getBalance(), “The balance should be zero if balance same as withdraw”); //

Account MyAccount3 = new Account(); MyAccount3.setMaxOverdrawn(150.0); MyAccount3.setBalance(200.0); MyAccount3.withdraw(null);

assertEquals(200.0, MyAccount3.getBalance(), “The balance should stay the same as it was before because the amount of withdrawn money is null”); Account MyAccount4 = new Account();

MyAccount4.setBalance(null); MyAccount4.withdraw(100.0);

assertEquals(null, MyAccount4.getBalance(), “The balance should be empty as it was before”);

(27)

Table 14 c Account MyAccount9 = new Account();

MyAccount9.setBalance(200.0); MyAccount9.setCurrency(“SEK”); Account MyAccount8 = new Account(); MyAccount8.setBalance(200.0);

MyAccount8.setCurrency(“SEK”);

MyAccount8.TransferToAccount(MyAccount9);

assertEquals(400.0, MyAccount9.getBalance(), “The balance should not be changed becuase it is not allowed to transfer money to itself”);

assertEquals(0.0, MyAccount8.getBalance(), “Balance from the source account shoulbe be romoved as well”);

Account MyAccountEur = new Account(); MyAccountEur.setCurrency(“EUR”); MyAccountEur.setBalance(50.0);

MyAccountEur.convertToCurrency(“SEK”, 10.0); // added converting the currency to SEK

MyAccountEur.TransferToAccount(MyAccount2);

assertEquals(700.0, MyAccount2.getBalance(), “The balance in account should be 700 SEK”); // here is the bug in account method that it sums balance with no checking the currency

assertEquals(“SEK”, MyAccountEur.getCurrency(), “The Accounts should have the same currency as SEK”); // here test case for checking currency of account

Account MyAccount4 = new Account(); MyAccount4.setCurrency(“SEK”); MyAccount4.setBalance(50.0);

MyAccount4.TransferToAccount(null);

assertEquals(50.0, MyAccount4.getBalance(), “Balance should be the same as it was before, because the targetAccount is empty”);

Account MyAccount5 = new Account(); MyAccount5.setCurrency(null);

MyAccount5.setBalance(50.0);

assertEquals(50.0, MyAccount5.getBalance(), “Balance should be the same as it was before, because the account who wants to transfer money has not a currency”);

(28)

MyAccount.setCurrency(“SEK”); MyAccount.setBalance(500.0);

MyAccount.convertToCurrency(“EUR”, 0.1);

assertEquals(“EUR”, MyAccount.getCurrency(), “The Currency should be EUR”); // assertEquals(50, MyAccount.getBalance()); //check the amount in account

assertEquals(50, MyAccount.getBalance(), “The balance should be 50”); Account MyAccount1 = new Account();

MyAccount1.setCurrency(“SEK”); MyAccount1.setBalance(100.0);

MyAccount1.convertToCurrency(“SEK”, 10.0); // so here is the bug of con-verting, if accounts have same currency // it is still converting and multiply to the rate

assertEquals(“SEK”, MyAccount1.getCurrency(), “Should have the same currency”);

assertEquals(100.0, MyAccount1.getBalance(), “The balance should stay the same as it was before convertings”);

MyAccount2.convertToCurrency(null, 10.0);

assertEquals(“SEK”, MyAccount2.getCurrency(), “Should have the same currency as it was before and it is not supposed to be multiplied to the rate”); Account MyAccount3 = new Account();

MyAccount3.convertToCurrency(“EUR”, 0.0);

assertEquals(“SEK”, MyAccount3.getCurrency(), “Should have the same currency as it was before and it is not supposed to be multiplied to the rate, because rate is empty”);

6 a Withdraw all money from the account

(29)

MyAccount4.setMaxOverdrawn(100.0); MyAccount4.setCurrency(“SEK”); MyAccount4.setBalance(0.0); MyAccount4.withdrawAll();

assertEquals(0.0, MyAccount4.getBalance(), “Balance should be zero”); Account MyAccount = new Account();

MyAccount.setMaxOverdrawn(100.0); MyAccount.setCurrency(“SEK”); MyAccount.setBalance(-300.0);

MyAccount.withdrawAll(); // assertEquals(0, MyAccount.getBalance(), “The amount of withdrawn money should // be 0 because it is less than 0”); assertEquals(-300, MyAccount.getBalance(), “The amount of balance should be stayed as it is, -300”);

Account MyAccount2 = new Account(); MyAccount2.setMaxOverdrawn(200.0); MyAccount2.setCurrency(“EUR”); MyAccount2.setBalance(100.0); MyAccount2.withdrawAll();

assertEquals(0.0, MyAccount2.getBalance(), “It should be same as it was because withdraw is possible just with SEK currency”);

Account MyAccount3 = new Account(); MyAccount3.setMaxOverdrawn(200.0); MyAccount3.setCurrency(“EUR”); MyAccount3.setBalance(100.0);

MyAccount3.convertToCurrency(“SEK”, 10.0); MyAccount3.withdrawAll();

assertEquals(0.0, MyAccount3.getBalance(), “Balance should be zero”); 7 a Valid account

c Account MyAccount = new Account(); MyAccount.setCurrency(“SEK”);

assertEquals(“SEK”, MyAccount.getCurrency(), “The currenct of account should be SEK”);

b This function sets max overdrawn sum. The number should not be below zero or char. The only type of input data is an integer. Should not be null. c myBankAccount.setMaxOverdrawn(100.0);

assertEquals(“Max overdrawn should be 100”, 100.0, myBankAc-count.getMaxOverdrawn(), 0);

10 a Valid number to get max overdraw sum

(30)

Table 14 c myBankAccount = new Account(100.0, “SEK”, 100.0);

c myBankAccount.setCurrency(“EUR”);

assertEquals(“EUR”, myBankAccount.getCurrency()); 12 a Get valid currency

c Account myBankAccount = new Account(100.0, “SEK”, 0.0); assertEquals(“SEK”, myBankAccount.getCurrency());

Table 14. Test Cases based on Equivalence Partitioning

In Table 15, also provided the test cases based on equivalence partitioning, but the fundamental difference is the number of test cases to each function that has been made. As we described above, the order of functions sorted by the arithmetical mean of two main measures. So we add more test cases to the functions that are at the top of the list.

Table 15 Test Cases based on Operational Profile

(31)

MyAccount.setBalance(200.0); MyAccount.setCurrency(“SEK”); MyAccount.deposit(100.0);

assertEquals(300.0, MyAccount.getBalance()); // checking the amount of balance after the deposit

Account MyAccount3 = new Account(); MyAccount3.setBalance(-350.0);

MyAccount3.setCurrency(“SEK”); MyAccount3.deposit(400.0);

assertEquals(50.0, MyAccount3.getBalance()); Account MyAccount4 = new Account(); MyAccount4.setBalance(0.0);

MyAccount4.setCurrency(“SEK”); MyAccount4.deposit(400.0);

assertEquals(100.0, MyAccount1.getBalance()); // the balance should be the same as it was before because currencies are different

MyAccount2.setCurrency(“EUR”); MyAccount2.deposit(null);

assertEquals(100.0, MyAccount2.getBalance()); // the balance should be the same as it was before because the // value of amount of deposit is null 2 a Get valid balance

(32)

Table 15 c myBankAccount.setBalance(27.0);

myBankAccount.setBalance(42.42);

assertEquals(“The value of the account shall be 42.42”, 42.42 myBankAc-count.getBalance(), 0);

myBankAccount.setBalance(0.0); assertEquals(“The value of the account shall be 0.0”, 0.0, myBankAccount.getBalance(), 0);

myBankAccount.setBalance(-35.0); assertEquals(“The value of the ac-count shall be -35.0”, -35.0, myBankAcac-count.getBalance(), 0);

myBankAccount.setBalance(9999.99); assertEquals(“The value of the ac-count shall be 9999.99”, 9999.99, myBankAcac-count.getBalance(), 0);

b This function withdraws an amount. Should not be below zero or zero, higher than Balance Sum, and char. Should not be null.

c Account MyAccount = new Account(); MyAccount.setMaxOverdrawn(150.0); MyAccount.setBalance(200.0);

MyAccount.withdraw(150.0); // assertEquals(“The amount of account should be 50”, 50, 50);

assertEquals(50, MyAccount.getBalance()); // the amount of balance should be 50

MyAccount1.withdraw(250.0);

assertEquals(200, MyAccount1.getBalance(), “The balance should stay the same if amount of withdraw is greater than balance”); // the balance became negative

Account MyAccount2 = new Account(); MyAccount2.setMaxOverdrawn(200.0); MyAccount2.setBalance(200.0); MyAccount2.withdraw(200.0);

assertEquals(0, MyAccount2.getBalance(), “The balance should stay the same if amount of withdraw is greater than balance”); //

Account MyAccount3 = new Account(); MyAccount3.setMaxOverdrawn(150.0); MyAccount3.setBalance(200.0); MyAccount3.withdraw(null);

assertEquals(200.0, MyAccount3.getBalance(), “The balance should stay the same as it was before because the amount of withdrawn money is null”); Account MyAccount4 = new Account();

MyAccount4.setBalance(null); MyAccount4.withdraw(100.0);

assertEquals(null, MyAccount4.getBalance(), “The balance should be empty as it was before”);

(33)

Table 15 b This function transfers the amount to another account. The transferred amount should not be below zero, zero, higher than the Balance. Should not be null. Can not transfer money to itself. The amount of transferred money from first account should be removed. Another account should add this amount of money. Accounts should have the same currency.

(34)

MyAccount9.setBalance(200.0); MyAccount9.setCurrency(“SEK”); Account MyAccount8 = new Account(); MyAccount8.setBalance(200.0);

assertEquals(400.0, MyAccount9.getBalance(), “The balance should not be changed because it is not allowed to transfer money to itself”);

assertEquals(0.0, MyAccount8.getBalance(), “Balance from the source account shoul be be romoved as well”);

Account MyAccount = new Account(); MyAccount.setBalance(200.0);

MyAccount.setCurrency(“SEK”);

Account AnotherAccount = new Account(); AnotherAccount.setBalance(0.0);

AnotherAccount.setCurrency(“SEK”);

MyAccount.TransferToAccount(AnotherAccount); // assertEquals(200, An-otherAccount.getBalance()); //check the balance in // transferAccount assertEquals(200.0, AnotherAccount.getBalance(), “The balance in another account should be 200”);

Account MyAccountEur = new Account(); MyAccountEur.setCurrency(“EUR”); MyAccountEur.setBalance(50.0);

MyAccountEur.convertToCurrency(“SEK”, 10.0); // added converting the currency to SEK

MyAccountEur.TransferToAccount(MyAccount2);

assertEquals(700.0, MyAccount2.getBalance(), “The balance in account should be 700 SEK”); // here is the bug in account method that it sums balance with no checking the currency

assertEquals(“SEK”, MyAccountEur.getCurrency(), “The Accounts should have the same currency as SEK”); // here test case for checking currency of account

MyAccount4.TransferToAccount(null);

assertEquals(50.0, MyAccount4.getBalance(), “Balance should be the same as it was before, because the target Account is empty”);

Account MyAccount5 = new Account(); MyAccount5.setCurrency(null);

MyAccount5.setBalance(50.0);

assertEquals(50.0, MyAccount5.getBalance(), “Balance should be the same as it was before, because the account who wants to transfer money has not a currency”);

(35)

Table 15 b This function should have consist of any of params double amount, string

currency, double MaxOverdraw). Should not be null. c Account MyAccount = new Account();

MyAccount.setCurrency(“SEK”);

assertEquals(“SEK”, MyAccount.getCurrency(), “The currenct of account should be SEK”);

Account MyAccount1 = new Account();

MyAccount1.setBalance(100.0); assertEquals(100.0, MyAc-count1.getBalance(), “The balance of account should be 100.0”); // checking a currency in account (should be SEK)

MyAccount3.setMaxOverdrawn(2000.0);

assertEquals(2000.0, MyAccount3.getMaxOverdrawn(), “The limit of with-draw of account should be 2000.0”); // checking a currency in account (should be SEK)

MyAccount3.setMaxOverdrawn(null);

assertEquals(null, MyAccount3.getMaxOverdrawn(), “The limit of withdraw of account should be 0.0”);

(36)

MyAccount.setCurrency(“SEK”); MyAccount.setBalance(500.0);

MyAccount.convertToCurrency(“EUR”, 0.1);

assertEquals(“EUR”, MyAccount.getCurrency(), “The Currency should be EUR”); // assertEquals(50, MyAccount.getBalance()); //check the amount in account

assertEquals(50, MyAccount.getBalance(), “The balance should be 50”); Account MyAccount1 = new Account();

MyAccount1.convertToCurrency(“SEK”, 10.0); // so here is the bug of con-verting, if accounts have same currency // it is still converting and multiply to the rate

assertEquals(“SEK”, MyAccount1.getCurrency(), “Should have the same currency”);

assertEquals(100.0, MyAccount1.getBalance(), “The balance should stay the same as it was before convertings”);

MyAccount2.convertToCurrency(null, 10.0);

assertEquals(“SEK”, MyAccount2.getCurrency(), “Should have the same currency as it was before and it is not supposed to be multiplied to the rate”);

MyAccount3.convertToCurrency(“EUR”, 0.0);

assertEquals(“SEK”, MyAccount3.getCurrency(), “Should have the same currency as it was before and it is not supposed to be multiplied to the rate, because rate is empty”);

7 a Withdraw all money from the account

(37)