AGENT BASED SYSTEMS IN SOFTWARE TESTING –A SYSTEMATIC MAPPING STUDY

M

U

S

I

,

D

E

V

,

S

DVA423

Thesis for the Degree of Master of Science (60 credits) in Computer Science

with Specialization in Software Engineering 15.0 credits

AGENT BASED SYSTEMS IN SOFTWARE TESTING –

A SYSTEMATIC MAPPING STUDY

Pavithra Perumal Kumaresen

ppn19004@student.mdh.se

Examiner:

Sasikumar Punnekkat

Mälardalen University, Västerås, Sweden

Supervisors: Eduard Paul Enoiu

Mälardalen University, Västerås, Sweden

Mirgita Frasheri

Mälardalen University, Västerås, Sweden

2

ABSTRACT

The emergence of new technologies in the software domain has increased the flexibility of the development process by allowing modifications at any stage throughout the development life cycle. As a result, there is an increasing need for continuous integration and testing of the software at hand. Many systems and tools are used for efficient continuous testing at a reduced cost. Agent-based software testing (ABST) is an active research area where various agent methodologies, architectures, and tools are employed to perform different types of software testing at different abstract levels. There is a constant increase in the adoption of agent methodologies for the evaluation of new testing techniques with higher levels of automation and intelligence. Therefore, there is a need to understand the existing research trends in this domain, as well as to map the current challenges for future research in this area.

This thesis aims to identify what has been studied in ABST, but also examine how the ABST studies have been conducted, identify the gaps in the current research and point to directions for future ABST research. In this thesis a systematic mapping study has been conducted with a set of 41 selected papers regarding ABST. Our results suggest that there is an increasing interest in ABST after 1999 that resulted in the development of solutions using reactive, BDI, deliberative and cooperative agent architectures for software testing. In addition, most of the ABST approaches are designed using the JADE framework and the Java programming language. In addition, these approaches are used at system-level testing for functional, non-functional, and white-box test goals. The results of our thesis also indicate that ABST research is scarcely focusing on regression testing.

3

Table of Contents

2.3 Multi Agent Systems (MAS) ... 7

3. Research Method ... 8

3.1 Definition of the Research Questions ... 8

3.2 Search Process ... 9

3.3 Screening of Papers ... 10

3.3.1 Inclusion Criteria ... 10

3.3.2 Exclusion Criteria ... 10

3.3.3 Application of Selection Criteria ... 11

3.4 Classification ... 11

3.4.1 Publication and Research Trends ... 12

3.4.2 Agent Characteristics ... 13

3.4.3 Testing Characteristics ... 14

3.5 Data Extraction and Mapping ... 15

4. Data Analysis and Results ... 16

4.1 Publication and Research Trends ... 16

4.2 Agent Characteristics ... 19

4.3 Testing Characteristics ... 24

5. Related Work ... 27

6. Threats to Validity ... 28

7. Conclusion and Future Work ... 29

References ... 30

4

List of Figures

Figure 1. An abstraction of MAS for a multinational company. ... 7

Figure 2. The Systematic Mapping Process used in this thesis. ... 8

Figure 3. Publications by years. ... 16

Figure 4. Different publication channels and research contributions used in ABST. ... 17

Figure 5. Distribution of publications by publication channels and research contributions. ... 17

Figure 6. Empirical research methods used in agent-based software testing. ... 18

Figure 7. Research contributions and empirical research methods. ... 19

Figure 8. Agent communication in JADE ... 22

Figure 9. Agent architectures and application types ... 23

Figure 10. Testing levels and types in agent-based software testing ... 25

Figure 11. Testing characteristics and types of applications ... 26

List of Tables

Table 2. Research contributions facet based on Wieringa et al. [12]. ... 12

Table 3. Agent architectures used in agent-based software testing. ... 19

Table 4. Agent development framework and tools ... 21

Table 5. Programming languages used in agent-based systems ... 23

Table 6. Results of different testing levels in agent-based software testing ... 24

Table 7. Results of different testing types in agent-based software testing ... 24

5

1.

Introduction

The provision of confidence in the correctness and efficiency of software is a crucial step in software development, as even a minor defect can ensue an undesired result or cause a complete failure of its operation. This confidence can be provided by performing a thorough and continuous testing of the system. Continuous integration and testing of the modifications made to the system throughout the development life cycle are highly costly and labor intensive. Hence test automation is being used where the designing, execution, and analysis of the testing are performed in an automated manner with the help of various tools. A survey has been conducted by Katalon Studio and KMS technology [1] to identify the possible issues in adopting test automation techniques and its implementation. Some of the most challenging issues discussed in this survey are frequent changes in the requirements, difficulties in integrating many tools, highly expensive commercial tools, and lack of knowledge regarding specific tool usage. To test a system, test cases are created by testers based on the requirements of the user and are tested against the expected outcomes to verify the correctness of the system. A system under development undergoes a lot of changes made to its requirements and it must be made sure that these changes do not affect the normal system functioning. The process of testing the updated software to help ensure that it still possesses the same functionality it had before the updates is known as regression testing [2]. A sheer amount of test cases is created and executed automatically and repeatedly to perform regression testing [3]. Usually, these test cases are not engineered to adapt on their own to the changes made in the environment and rely on the information given by test designers. As a result, they do not provide a flexible and scalable testing system.

Current continuous development and integration processes produce large and frequent changes. In this context, software development organizations need to deliver reliable and high-quality software products while having to consider more stringent time constraints [3]. To reduce the aforementioned complexities, researchers have proposed the use of machine learning and agent-based approaches in which the control is distributed among several small systems that can act autonomously according to their local strategies, such as multi-agent systems (MAS). A MAS consists of several agents that are used in a distributed system to perform various tasks throughout the development and testing phases of the software. The definition of an agent varies based on its application domain [4]. An agent can refer to a piece of software executing in a virtual environment, as well as a physical robot that operates in the real world [4].

This thesis provides the results of a systematic mapping study on the current trends in the usage of agent-based systems for software testing. We identify the areas of application and the tools, techniques, and methods used in the development of agent-based systems. Through this study, researchers can get a clearer picture of the current trends in this domain, the challenges, and the need for future research. Additionally, practitioners can identify the level of usage of agent-based systems in real world applications.

6

2.

Background

2.1.

Software Testing

Almost all the domains across the globe are built with some level of dependencies on software usage, thus increasing the demand for high-quality software. The quality of a software is improved by testing its functionalities and making sure it is free from defects. Software testing is a process of finding possible defects in the system [5].

There are several types of testing done at different levels of software development. A level of testing is categorized based on different phases in a software development life cycle [2]. The different levels of testing in a software described by P. Ammann and J. Offutt [2] are as follows: ● Acceptance testing: assesses software with respect to its requirements. It checks a

complete system based on the requirement satisfaction of a user.

● System testing: assesses software with respect to its architectural design. It checks for

the correct functioning of a system as a whole package together.

● Integration testing: assesses software with respect to its subsystem design. It checks for

the proper integration of lower units and correct intercommunication and operations among them.

● Module testing: assesses software with respect to its detailed design. It checks the

detailed lower level parts of the system architecture and their correctness. Modules are the collection of similar units under a class or package.

● Unit testing: assesses software with respect to its implementation. It checks for the

correct implementation of the units which are pieces of program code that perform specific tasks. Unit testing is the lowest level of testing.

A test can be performed to check either the functional or structural correctness of the system. Other extra functional qualities1 _{such as reliability, performance, and usability of software can} also be tested. The defects identified at various levels are removed by a process called debugging which is different from software testing [5].

2.2.

Software Agents

According to Gokulan et al. [4], the word “Agent” cannot be owned by a single community, as it has a vastly varied application domain and it is impossible to generalize it to a specific community. It can have different physical forms as mentioned earlier and it is defined with respect to its application domains [4].

One of the most popular definitions for an agent is given by Wooldridge and NR Jennings [6], according to whom, the term agent is used to denote a hardware or a software-based computer system that has properties such as:

● operate on their own without human interventions (autonomy), ● interact with other agents (social ability),

● perceive their environment and respond in a timely fashion to changes that occur in it (reactivity) and

7

● take initiative to exhibit a goal-oriented behavior (proactiveness).

Some other properties that are associated with the agents include mobility, temporal continuity, and collaborative behavior among others [4]. Agents are also capable of learning from the experience of other communicating entities which may be human, other agents in the network, or statistical controllers [4]. Agent-oriented systems, to be effective, should merge conventional agent design with approaches employed in advanced distributed systems [7]. An agent in a distributed system can communicate with other subsystems and identify the changes in the system and support other subsystems/agents without any external command.

2.3.

Multi Agent Systems (MAS)

According to B. P. Gokulan and D. Srinivasan [4], “Multi-Agent Systems (MAS) are an

extension of the agent technology where a group of loosely connected autonomous agents acts in an environment to achieve a common goal”. Application of MAS in a distributed system has

the potential to largely reduce the complexity in integrating the subsystems that may or may not be in the same physical location [8]. It also allows us to build systems in which the building blocks are interacting MAS that work together to reach one or several global goals (the global goal is the goal of the system as a whole).

Figure 1. An abstraction of MAS for a multinational company.

Furthermore, as shown in Figure 1, several MAS can be grouped to form a global system with a common goal where each subsystem is a multiagent system consisting of many agents with their own defined properties. MAS are being used for various purposes such as diagnosing subsystems in a distributed environment [9], processing the information for users in an open internet environment [10]and testing the functionalities of a system [11].

8

3.

Research Method

A systematic mapping involves several steps like identification of papers, analysis, and classification of the selected papers in the domain of interest. In this thesis, the mapping study is adapted from the process outlined by K. Petersen et al. [12] and shown in Figure 2.

Figure 2. The Systematic Mapping Process used in this thesis.

There are five steps involved in performing the mapping study starting with the definition of research questions which gives a scope for the search of studies. Based on the scope of the research, a set of search strings are derived that are applied in selected databases to identify the studies within the research area. From the identified studies, the most relevant studies are screened by applying certain inclusion and exclusion criteria. The abstracts, introduction, and conclusion sections of the filtered papers are studied to identify the keywords for classification which are aimed to answer the research questions. Finally, these papers are studied in detail to extract the data under each categorization and the obtained results are compared and discussed to perform the systematic mapping.

3.1.

Definition of the Research Questions

The main objective of this section was to form a set of research questions that help us to have a better understanding of the scope of the research. The research questions formed are as follows:

RQ1. What is the current state of agent-based software testing research?

The main objective of this question is to identify the research interest and contribution in this domain over the years. This question was divided into the following sub questions to have a clear view of the publication and research trends:

RQ1.1. What number of academic studies on agent-based software testing has been published? RQ1.2. What are the publication channels used to publish studies on agent-based software testing?

RQ1.3. What kinds of research contributions are provided by studies on agent-based software testing?

RQ1.4. What research methods have been used in empirical studies on agent-based software testing?

The main objective of the next question is to identify the characteristics of the agent-based approaches used in software testing:

9

The objective of this question is to identify the various details of the agent-based systems that are used in this specific research area. This question is divided in the following three sub-questions focusing on agent-based system architecture and implementation:

RQ2.1. What are the agent architectures used in agent-based software testing?

RQ2.2. What are the development frameworks used for implementing agent-based systems in software testing?

RQ2.3. What are the programming languages used by agent-based systems in software testing? The main objective of the final question relates to the test-related characteristics of agent-based systems used for software testing:

RQ3. What are the testing characteristics provided by agent-based software testing approaches? This question was divided into three sub questions as follows:

RQ3.1. What are the testing levels targeted by the studies on agent-based software testing? RQ3.2. What are the testing types targeted by agent-based software testing?

RQ3.3. What are the types of applications targeted by studies on agent-based software testing?

3.2.

Search Process

Searching for papers is one of the critical phases in a systematic mapping study as it tries to achieve a comprehensive coverage of the research topic under study. We devised a set of search strings for conducting the search in multiple digital libraries in the agent-based software testing (ABST) research area.

The following databases were selected to perform the search: • IEEE Xplore digital library.

• ACM digital library.

• SCOPUS scientific database.

The selected databases are pertinent to this study as these return the most manually collected publications on agent-based software testing. In addition, these databases were selected due to their advanced search features that help us to search through a specific section of the paper which may be a title, an abstract, keywords, or the entire paper and it also provides us with the facility to search for multiple keywords connected with AND and OR logic operators. Due to the lack of advanced search facilities in the Springer database, it was difficult to obtain a reasonable number of results. In addition, the results retrieved from Wiley were found in the other databases.

The first step in the search process was defining search strings based on the research questions and applying them in the selected databases. The search strings selected for this study were “software”, “testing”, and “agents”, based on the research topic. The search format varies with different databases as shown below:

• IEEE: ((("Abstract”: software) AND "Abstract”: testing) AND "Abstract”: agents) • ACM: [Abstract: software] AND [Abstract: testing] AND [Abstract: agents] • SCOPUS: TITLE-ABS-KEY (“Software” and “testing” and “agents”)

10

These keywords were searched in the abstract section of the papers, since an abstract of a paper gives an overview of the conducted research and we considered that these keywords are vital for categorizing these approaches. Searching through the entire paper led to several thousands of results. For example, 33,986 results were obtained from ACM digital library on searching these keywords in the entire paper. We argue that a relevant paper in agent-based software testing would mention the selected keywords upfront in the title and abstract. Thus, the abstract section is used for the search process and the following results were retrieved:

• IEEE: 667 papers • ACM: 860 papers • SCOPUS: 1136 papers

We mention here that there are other methods that can be used to improve the coverage of such a mapping study. In this thesis, three article databases have been used for identifying the relevant papers in this domain. Beside this method, one could use other additional search processes such as snowballing (backward and forward) [13]. The selection of a start set of papers used to perform snowballing search is one of the main challenges in this procedure. Another method used is to apply the snowballing process for the set of papers selected. Snowballing is not necessarily an alternative to the database search process. Due to the nature of this study, we used a different approach to check the reliability of the relevant literature and ensure the best possible coverage of the literature. We used the comprehensive related work collected in a paper outlying the area of test agents by Enoiu and Frasheri [3]. We selected these papers and checked their inclusion against the results obtained from our search process. All papers included in this paper have been found using our search strategy and databases. However, a detailed snowballing process could potentially extend the confidence in our search process.

3.3.

Screening of Papers

The process of screening was done to refine the search results by eliminating the duplicate entries and non-related papers based on certain screening criteria. The search strings were applied in the selected three databases which led to the overall result of 2663 papers. Due to the generic nature of the search strings a large number of results were obtained. A set of inclusion and exclusion criteria were applied to the title, abstract, and keywords sections of the papers to identify the relevant papers.

In the inclusion and exclusion criteria we focused on the relevant aspects by including the following classifiers:

3.3.1. Inclusion Criteria

• Papers that are relevant to using agent-based systems in software testing.

• Papers that include the development and maintenance of agent-based systems used in software testing.

• Papers that include a review on tools, performance, and other features of agents-based systems used in software testing.

3.3.2. Exclusion Criteria

• Papers that are not relevant to agent-based systems.

• Papers on software testing using other intelligent learning techniques like machine learning and not autonomous agents.

11

• Papers on the usage of agents to maintain and increase the system performance or other criteria rather than testing the software systems.

• Papers on testing agent-based systems rather than the usage of these systems in software testing.

3.3.3. Application of Selection Criteria

The initial screening process led to the omission of several hundreds of papers as most of the papers were about software testing techniques without usage of agent methodology or application of agent methodology in other domains other than software testing. The inclusion of the ‘software’ keyword in the search process retrieved a large set of papers concerning other domains in software engineering. The search was first conducted in the IEEE database and this led to the selection of more papers from IEEE and omission of the repeated papers with same title and authors from the other databases that were searched after IEEE. As a result of this search process, a set of 182 papers were selected with comparatively a smaller number of papers from ACM and SCOPUS as shown in Table 1.

In order to extract the most relevant papers in this domain, a second set of screening is done with more detailed analysis where if the data from the abstract, title and keyword sections was insufficient, the introduction and conclusion sections were used to apply the criteria. As a result, out of 182 papers from the initial results a set of 48 papers were selected. These 48 papers were studied in detail to perform the data extraction. During this process two papers (S06 and S09) were found to be repeated in both IEEE and SCOPUS with a slight modification in the title or the version. These repeated papers from IEEE were removed from the study. In addition, two papers from IEEE and ACM and one paper from SCOPUS were also removed since these did not provide sufficient data needed to perform the systematic mapping study. Finally, a set of 41 papers was obtained as a result of a detailed screening process as shown in Table 1.

Table 1. Screening of papers.

SCREENING OF PAPERS Search results After initial

screening After final screening After detailed study IEEE 667 90 26 22 ACM 860 67 9 7 SCOPUS 1136 25 13 12

3.4.

Classification

The classification scheme is composed of the following facets based on the research questions. Each facet addresses a research question and consequently the corresponding sub questions. The facets are as follows:

3.4.1 Publication and Research Trends 3.4.2 Agent Characteristics

12

In the following sections we will expand these facets and provide details on each of the subdimensions used for classifying the ABST approaches

3.4.1 Publication and Research Trends

We used the following categories for extracting data from each paper in relation to RQ1: RQ 1.1. Publication Rate. It identifies the number of publications over the years in this domain. RQ 1.2. Publication Type

It identifies the channels of publications in the ABST area. Some of the publication channels used in this research area are as follows:

• Journal papers • Conference papers • Workshop papers

RQ 1.3. Research Contributions

An existing classification of research approaches by Wieringa et al. [14] was used in this mapping study. There are five types of research contributions used in [12] as shown in Table 2.

Table 2. Research contributions facet based on Wieringa et al. [12].

Category Description

Validation research

Techniques investigated are novel and have not yet been implemented in practice. Techniques used are for example experiments, i.e., work done in the lab.

Evaluation research

Techniques are implemented in practice and an evaluation of the technique is conducted. That means, it is shown how the technique is implemented in practice (solution implementation) and the consequences of the implementation in terms of benefits and drawbacks (implementation evaluation). This also includes identifying problems in industry.

Solution proposal

A solution for a problem is proposed, the solution can be either novel or a significant extension of an existing technique. The potential benefits and the applicability of the solution is shown by a small example or a good line of argumentation.

Philosophical papers

These papers sketch a new way of looking at existing things by structuring the field in form of a taxonomy or conceptual framework.

Opinion papers

These papers express the personal opinion of somebody whether a certain technique is good or bad, or how things should be done. They do not rely on related work and research methodologies.

Experience papers

Experience papers explain what and how something has been done in practice. It must be the personal experience of the author.

13

RQ 1.4. Empirical Research Methods

There are several types of empirical studies done in a research work, whereas the following three types were selected in this study due to the direct identification of these keywords in the papers.

• Experiment: which includes experiments done in laboratories and industries to verify

and explain the functioning of the proposed systems.

• Case Study: which performs a detailed investigation by gathering data from a variety of

sources using different methods such as observations and interviews to have a better understanding of the subject under study.

• Comparative Study: which makes a comparison between two or more similar works to

identify their similarities and to gain insights about the level of usage of agent methodology in software testing.

3.4.2 Agent Characteristics

This category covers the characteristics of the agent-based systems (RQ2) used for software testing in this study. It consists of the following sub facets.

RQ 2.1. Agent Architecture

To have an overview of the system structure, it is important to understand its architecture which is a computational solution to a problem, showing how the parts of a system interact [15]. One of the main factors that support us to understand the evolution of agent technology is the architecture of an agent system. The agent architecture is considered as the functional brain of an agent in making decisions and reasoning to solve problems and achieving goals [16]. Several surveys and reviews have been conducted over the years on agent architectures [15] [16]. The agent architecture classification used in this study was adopted from the classification scheme used by Friedenberg and Silverman [17]. There are five different architectures for an agent-based system according to Friedenberg and Silverman [17] as described below:

• Simple reactive: Agents in the system are simple and fast whose response and reasoning

facilities are similar to that of a series of if-then logical sequences. But in an unpredicted dynamic circumstance they lack decision making capabilities and rely on other mapping agents.

• Reactive / Subsumptive: Agents are more advanced than a simple reactive agent and

they are viewed as a collection of simple behavior modules but having a hierarchical organization.

• BDI: Agents can store the state of its system and environment (belief), maintain a goal

(desire), and contain a series of mapping to convert the belief and desires into action (intention).

• Deliberative: Agents have the potential to solve complex problems, include provision

for planning, and can perform a sequence of actions to achieve a certain goal. These agents take advantage of contemporary AI technologies such as Neural Nets, Fuzzy Logic, and Expert systems to enhance planning actions.

• Blackboard / Co-operative: Many agents act as a team, where the knowledge of the state

14

RQ 2.2. Agent Development Frameworks

This classification scheme includes the different toolkits, frameworks, and platforms used for the implementation of agent-based systems. It is important to analyze and compare the various tool supports for these systems to identify the most efficient and user-friendly ones. Several reviews have been conducted to identify the development frameworks that are used in agent technology [18] [19]. Some of the active common development frameworks such as JADE, JADEX, and JACK [18] [19] were identified in this study. The usage of JADE LEAP, which is an extended version of JADE, was also identified in a few papers. The other development tools used for implementing agent-based systems are ADK, Aglet and FIPA-OS. This category aims to identify the commonly used development tool in this domain.

RQ 2.3. Programming Language

This category identifies the commonly used programming languages by agent-based systems in software testing. Some of the popular programming languages are java, python, C, ruby, C# [20].

3.4.3 Testing Characteristics RQ 3.1. Testing Levels

This category classified the papers based on the level of abstract at which ABST is performed. For our purpose we used the following testing levels by adapting some existing classifications [2] [5]. The following four main levels of testing were used in the classification2:

• Acceptance Testing. It assesses the software with respect to its customer requirements.

• System Testing. It assesses the software with respect to its system architectural design.

• Integration Testing. It checks the proper integration of lower units and the correct

operation.

• Unit/ Component Testing. It assesses software with respect to its implementation.

RQ 3.2. Testing Types

The following classification of testing types is based on the categorization outlined by Spillner et al. [5] and based on earlier concepts defined in testing research:

• Functional Testing: Test cases are created based on the functional requirements of the

system.

• Non-Functional Testing: Test cases are designed based on attributes describing the

system as a whole and some of its non-functional characteristics such as reliability, usability, and performance.

• White Box Testing: It is also known as structure-based testing, where the tester is aware

of the internal code structure or architecture. It verifies the correct flow of the control and data through the components of the system.

15

• Regression Testing: The purpose of regression testing is to check whether changes to

existing software have introduced errors to functionality that performed correctly in the software's previous version.

RQ 3.3. Types of Applications Under Test.

This category identifies the types of applications that are tested using agent-based systems. Agent-based systems have a wide range of application domains as discussed in Section 2.3. We extracted data from each paper related to the application under test using ABST. These identified types are:

• Web-based Applications: applications concerning the use of web services.

• Network Applications: applications concerning various network factors such as network

topology, network performance, and network load sharing.

• Industrial Applications: applications concerning the control systems that are used in

different industries such as factory automation and safety-critical system control. • Distributed Software Applications: applications concerning the use of digital software

services in a distributed environment.

Thus, a simple classification scheme has been formed based on the research questions. It helps to have a direct extraction of data items to each facet and reduces the complications of further deeper analysis to make a comparison of results.

3.5 Data Extraction and Mapping

A final set of papers were refined by applying the criteria mentioned in Section 3.3 and certain papers were excluded during the process of detailed study due to lack of relevant data items in the papers. The final set of papers were studied in detail to extract the data items under each category of the classification scheme. An excel sheet has been used to document the extracted data with each column representing a category from the classification scheme and each row representing a paper. Thus, for every paper that was studied, the respective column details were entered for each facet. The data items were placed under a specific facet based on a selection motivation as mentioned in Section 3.3. Once the data items were identified and filled, a systematic map was performed which included the process of counting, clustering, and mapping. The number of similar data items under each category was counted and clustered and finally the numbers of each cluster were compared to get a clear result.

16

4.

Data Analysis and Results

As mentioned in Section 3.5, the data has been extracted from the studies and the results were then analyzed both quantitatively and qualitatively. A quantitative analysis gives the quantitative results under each category whereas the qualitative analysis is used to extract information based on interpretation. The results of each category are discussed in the following sections. The final set of primary studies used for data extraction along with its brief description can be found in Appendix A where each study has been numbered as SXX (where XX refers to the numerical values from 01 to 41).

4.1.

Publication and Research Trends

The analysis under this category is done to find the frequencies of publications over the years and types of publications channels and research contributions. It also analyzes the frequencies of publications under different empirical research methods.

RQ 1.1. The variations in the number of publications over the years in this domain is shown in Figure 3. The number of publications over the years from 1999 till present are covered in this analysis. There is an increase in the growth of research contributions to this domain over the last decade with its peak being in 2010 with the highest number of 6 publications made in this domain. Only a limited number of papers with a total sum of 11 papers were published in the current decade indicating the need for more research contributions in this domain. We were able to identify only a few papers from the databases that were published in 2019 and no paper from 2020, as the search was conducted during the early days of 2020. This categorization is valuable as it indicates that although the research contributions in this domain were increasing in the last decade, a fall in the contributions in the current decade highlights the need for more research work in this domain.

Figure 3. Publications by years.

RQ 1.2. The aim of this research question is to identify the main channels where agent-based software testing studies are disseminated. The results in Figure 4a shows that almost 78% of the primary papers were published in peer-reviewed international conferences, nearly 17% in journals and just 5% were published in workshops.

17

RQ 1.3. The classification results based on the research contributions is shown in Figure 4b. It is seen that more than 50% of the studies proposed solutions for various problems in this domain. In nearly 40% of the papers, the ABST technique has been validated and only around 10% of the papers have evaluated their proposed solutions. There are no papers of other types. This shows that there is a scarce collaboration between the research and industry, as there are only a very few evaluation papers and no papers in the experience category. Most of the papers were about creating new system structures or proposing solutions to reduce the complexity of the existing system structures using the agent technology. These results highlight a need for ABST research to provide more significant practical contributions, as well as widening the proposed academic solutions towards practical evaluations.

a. Publication channels b. Research contributions

Figure 4. Different publication channels and research contributions used in ABST.

18

The map of different publication channels and research contributions over the years is shown in Figure 5. Papers have been published in conferences throughout the years whereas publications made in journals and workshops are sporadic in the ABST research. Over the two decades several contributions have been made that are proposing solutions to various ABST problems. There is a fall in the number of contributions validating the ABST in the current decade. A very few contributions in the ABST research have evaluated their proposed solutions. We observe a decrease in the number of publications in this domain between the year 2013 and 2016. Only a few papers published in this period were identified in the initial results which were also omitted during the process of screening and detailed study.

RQ 1.4. The classification of papers based on the empirical research methods is shown in Figure 6. We found that 23 studies have conducted experiments in this research area, 4 studies have used the case study method and only 2 studies have performed a comparative study. In almost 32% of the studies there is no explicit mention of any empirical evidence available.

Figure 6. Empirical research methods used in agent-based software testing.

The identified types of empirical research methods used for different research contributions are shown in Figure 7. We observe that 10 studies (S07, S11, S12, S27, S28, S30, S32, S33, S36, S38) have conducted experiment to validate the different techniques in this domain and 11studies (S03, S04, S09, S10, S15, S16, S17, S20, S25, S26, S29) have conducted experiments to propose solutions to various problems in this domain either by creating new ideas or by extending the existing technologies. Only 2 studies (S18 and S19) have conducted experiments to evaluate their proposed solutions. A few studies have also used the case study method for different research contributions. Two studies (S14 and S38) have used the case study to perform validation in this domain. One study (S06) analyzed the evaluated solutions and another study (S37) proposed a new solution using the case study method. It is interesting to see that S38 has carried out an experiment and also used the case study method in their work. Only two studies (S08 and S34) have conducted a comparative study to analyze two different methods and propose a solution. In the other studies, there has been no explicit mention of any particular empirical research method being used.

19

Figure 7. Research contributions and empirical research methods.

4.2.

Agent Characteristics

The analysis under this category gives an overall understanding of the features of the agent-based systems used in this domain over the years.

RQ 2.1. The aim of this research question is to categorize studies on ABST based on key agent architectures emerging from the papers being studied. The percentage of different types of agent architectures used in the agent-based systems and the respective primary studies for each architecture are shown in Table 3. Our results suggest that the most used architecture by the agent-based systems is the reactive architecture (~39% of papers), while the other agent architectures such as BDI, deliberative and co-operative are used in almost 20% of the overall papers, respectively. We observe that simple reactive agent architectures are rarely used in ABST research. In one study (S36), there was a lack of enough information to identify the agent architecture used in the proposed system.

Table 3. Agent architectures used in agent-based software testing.

Architecture Primary Studies Percentage

Reactive S04, S06, S10, S11, S17, S20, S23, S24, S25, S26,

S27, S28, S31, S32, S33, S37 39.02%

BDI S03, S05, S09, S14, S16, S35, S38, S40 19.51%

Co-operative S01, S02, S07, S12, S21, S22, S29, S39 19.51% Deliberative S08, S13, S15, S18, S19, S30, S34, S41 19.51%

20

Our results suggest that the reactive architecture is used in a large number of contributions and it is used for all types of applications. In most of the studies, the agents are arranged in a layered or a hierarchical format where each layer consists of agents with specific functionalities to perform a particular task. For example, in the primary study S20, a 4 layered architecture is used in which four different agents, the System management agent, Pre-processing agent, Fault localization agent and Failure analysis agent are used in four layers and perform different tasks. These agents can communicate with each other and take decisions based on the information obtained. In the studies S26, S32 and S33 a 3 layered / tiered architecture is used in which each layer can use different agents with complex functionalities.

BDI is one of the well-known agent architectures which has been used for several years by many researchers in ABST. In this type of architecture certain agents are used to identify the state of the system and make decisions based on the state and send instructions to other agents which perform the required actions to achieve the goal of the system respectively. For example, S03 has different types of agents such as Tester agents to perform the test, Facilitator agents to detect and decide what the problems are and Visualizer agents to communicate with the external world and identify the state of the system.

Co-operative agents work as a team by communicating with each other with agents having similar functionality. For example, in S01 a group of agents work together where more than one agent can have the same functionality but are capable of performing specific tasks in different platforms by using certain criteria and all these agents communicate to one central agent (blackboard agent) which is known as the broker agent to get more information on the state of the system. In S22, a system has been proposed with a group of agents performing similar tasks that can communicate and seek help from each other.

Deliberative agents make use of the AI techniques and are used in more advanced systems. In S13, S19 and S41 fuzzy logic has been used, while in S15 a neuro-fuzzy technique has been proposed. In S08 and S34 intelligent optimization algorithms were used. In some cases, techniques can be combined to enhance the functionality of an agent. For example, in S26 the agent system has a hierarchical architecture where the agents at the lower hierarchical level are reactive in nature which are expected to make decisions based on the raw sensor inputs and the agents in the high level are deliberative in nature which are expected to deal with the knowledge level view of the agent environment using the symbolic representations and past data. This is known as hybrid architecture. We mention here this type has not been used according to the adapted classification [17].

RQ 2.2. The different development tools used for the implementation of agent-based systems are discussed in Table 4. In more than half of the studies (61%) there is no explicit mention of the tools used as most of them lack practical implementations of the proposed systems. From the analysis of rest of the papers, it is evident that the most common framework used for agent development is the JADE framework (10 papers) and some of the other platforms (frameworks/toolkits) used are ADK, IBM Aglet, JADE LEAP and JADEX (extensions of JADE), FIPA OS and JACK Intelligent Agent. The identified tools and its description along with the respective primary studies are shown in Table 4.

21

Table 4. Agent development framework and tools

Tool Description Primary

Studies

AGLET

It is a java mobile agent platform that can autonomously and spontaneously move from one host to another and eases the development of agent-based applications [21].

S10

ADK

ADK (Agent Development Kit) introduced by Tryllian is a mobile component-based development platform that allows you to build reliable and scalable industrial strength applications. The ADK features dynamic tasking, using a reliable, lightweight runtime environment based on Java [22].

S03

FIPA OS

FIPA-OS is a component-based toolkit enabling rapid development of FIPA compliant agents. It is completely developed in Java. It is made as an Open Source community project, making it an ideal choice for any FIPA compliant agent development activity [23].

S04

JACK

JACK Intelligent Agents is a framework for multi-agent system development that uses the BDI software model and provides its own Java-based plan language and graphical planning tools [24].

S34

JADE

JADE (Java Agent DEvelopment Framework) is a software Framework that simplifies the implementation of multi-agent systems through a middle-ware that complies with the FIPA specifications and through a set of graphical tools that support the debugging and deployment phases [25]. S12, S14, S16, S17, S18, S19, S24, S28, S37, S38 JADEX

JADEX is a Belief Desire Intention (BDI) reasoning engine that allows for programming intelligent software agents in XML and Java. It is very flexible and can be used on-top of different middleware infrastructures such as JADE [26].

S35

JADE LEAP

JADE-LEAP is a modified version of the JADE platform that can be used in any devices which are java enabled [27]. JADE-LEAP provides the same APIs with respect to JADE so that an agent developed to run on JADE can run on JADE-LEAP with NO modifications at all [27]. The differences between JADE and JADE-LEAP are limited to the low-level communication mechanisms invisible to applications and users [27].

22

JADE is the most used agent development framework in agent-based software testing. It follows the FIPA specifications and uses ACL (Agent Communication Language) for communication [25]. It provides several advantages [25] outlined by many of the contributions in this area:

• When an agent is created by JADE it is created in a container. A group of containers form a platform. Each platform has one main container and one or many agent containers and they can be created in different systems in different locations.

• The communications can be done within agents in the same container, different containers, and different platforms without any restrictions as shown in Figure 8. • Advanced communication facility to identify messages of a certain category (such as

REQUEST, INFO, PROPOSAL, and others) in a communication is facilitated by ACL using the performative option.

• It has an Agent Management System (AMS) and Directory Facilitator (DF) which provides the agents with a unique identity and offers a yellow page service to identify the list of available agents and connect to them.

• JADE can also be used with other testing frameworks such as Junit to perform advanced tasks.

• The JADE framework is highly preferred for debugging purposes due to the availability of several support tools such as remote monitoring agent, dummy agent, sniffer agent and introspector agent.

• Agent systems implemented using JADE are more scalable and flexible as the agents in these systems are highly mobile and can be migrated between containers and can also be cloned across different containers.

• Agents in a system implemented by JADE can engage in multiple conversations simultaneously and execute several concurrent tasks.

Figure 8. Agent communication in JADE

RQ 2.3. The aim of this research question is to categorize studies on agent-based software testing based on programming languages used in these agent-based systems emerging from the papers being studied. As shown in Table 5, in more than half of the papers (~51%) the Java programming language is used in the agent-based systems. In addition, in almost 37% of the papers there is no explicit mention of the usage of any specific programming language. Other programming languages like C, C++, Python, and Perl are rarely used (in less than 5 papers each). This is due to the fact that most of the agent development frameworks support and are built in Java programming language as discussed in Table 4. The benefits of object-oriented features in a distributed heterogeneous environment is one of the key factors that brings attraction towards the Java programming language in this research area.

23

Table 5. Programming languages used in agent-based systems

Programming

language Primary studies Percentage

C/C++ S07, S27, S30 7.31%

Perl S02 2.44%

Python S11 2.44%

Java S03-S06, S09, S10, S12, S14, S16, S17, S18, S19, S24,

S25, S28, S29, S34, S35, S37, S38, S39 51.21% To summarize the results for RQ2, ABST research has used reactive, BDI, deliberative and cooperative agent architectures. The majority of these solutions are using the Java programming language and the JADE framework. We also observed that there is a lack of information about the development frameworks and programming languages which could imply that practical implementations of the proposed agent-based systems are still scarce.

The cross mapping of data from different categories has been useful for mapping the current state of the agent-based software testing research area. Nevertheless, our results have limitations since the different tool support identified did not contain any information on the respective programming language used. However, mapping has been performed between different categories which produce interesting results for better understanding of the ABST research. One such mapping has been performed between the agent architectures and the application types (as shown in Figure 9). We can observe that the deliberative agent architectures are mainly used in industrial systems and the BDI is used only in web-based and distributed software applications. The co-operative is mainly used in the distributed software applications and moderately used in all other applications. In addition, the reactive agent architecture seems to be a very popular architecture used in agent-based software testing and used in all the types of application under test.

24

4.3.

Testing Characteristics

RQ 3.1. Testing Levels: The percentage of different testing levels in this domain is shown in Table 6. It is observed that most of the studies (~68%) have used agent technology to perform system level testing, while around 34% of the studies have used agent technology to perform the Unit level testing and a very few studies (~5%) have used agent technology to perform the integration level testing. No studies are found on acceptance level testing.

Table 6. Results of different testing levels in agent-based software testing

Testing levels Primary studies Percentage

Unit S01, S02, S05, S06, S22, S24-S27, S29, S33, S36, S37,

S39 34.41%

Integration S01, S12 4.88%

System S01, S03, S04, S07-11, S13-S21, S23, S28-S32, S34,

S35, S38, S40, S41 68.29%

RQ 3.2. Testing Types: The aim of this research question is to categorize studies on ABST based on the testing areas emerging from the papers being studied. The percentage of the papers based on the types of testing done on the applications is shown in Table 7. Most of the agent-based software testing approaches are targeting functional testing (18 studies), white-box testing (19 studies) and non-functional testing (13 studies). The scarcity of agent-based software testing research in regression testing (4 studies) indicates that this area of research has not been scaled to regression test selection and continuous integration practices. It is observed that functional and white box testing are almost performed on equal rates whereas non-functional testing is comparatively less, and very few studies are performed on regression testing with the help of agent-based systems.

Table 7. Results of different testing types in agent-based software testing

Testing types Primary studies Percentage

Functional S01, S03, S09, S10, S12, S15, S17, S21, S23, S25, S28-S32, S37, S39, S41 43.9% Non-Functional S01, S02, S07, S11, S12, S14, S16, S17, S23, S25, S29, S32, S38 31.7% White Box S01, S04, S05, S08, S13, S18, S19, S20, S26-30, S33-S36, S38, S40, 46.34% Regression S01, S06, S22, S24 9.75%

Our observations suggest that many studies propose agent-based systems that can be used to perform testing of different types and levels by selecting certain test criteria and goals. For example, the primary study S01 proposed a common system using agent technology that can

25

be used for performing any type of testing at unit, system, and integration levels. The study S29 discussed the usage of agent techniques to perform both unit and system level testing. Similarly, S17 has discussed both functional and non-functional testing, while S29 discussed functional, non-functional, and white box testing. This trend behavior in our map can be observed in Table 7.

Figure 10. Testing levels and types in agent-based software testing

We also investigated the mapping between testing levels and types as shown in Figure 10. We observed that the functional, non-functional, and white box testing areas in ABST research are highly concentrated at the system level, moderately concentrated at the unit level and rarely at the integration level testing of the systems. Studies on regression testing in ABST are mostly done at the unit level and one study (S01) has proposed a common system that can be used for performing regression testing (and all the other three testing areas) at different levels. In all the three testing levels (unit, integration, and acceptance), the most commonly studied testing types are functional and white box testing, followed by non-functional testing. There are no studies concentrating specifically on acceptance level testing and only a very few studies on regression and integration testing.

Table 8. Results obtained for application types used in agent-based software testing.

Application types Primary studies Percentage

Web-based applications S01, S05, S06, S10, S14, S16, S25, S31 19.51% Network applications S02, S07, S15, S17, S27, S36 14.63% Industrial automated systems S04, S13, S18, S19, S21, S23, S26, S30 19.51% Distributed software applications S03, S08, S09, S11, S12, S20, S22, S24, S28, S29, S32-S35, S37-S41 46.34%

26

RQ 3.3: The classification of papers based on the types of applications where the agent-based system is used is shown in Table 8. Our results suggest that agent-based systems are mostly used for testing distributed software applications (~46% of papers). In addition, agent-based software testing targets other application domains such as web-based applications (19.5% of the studies), network applications (almost 15% of the studies), and industrial automation systems (19.5% of the studies).

We obtained a map of the different levels and types of testing performed on different applications as shown in Figure 11. We can observe that several contributions have analyzed the functional and white box testing at the system level of the distributed software applications, with a limited number of contributions on other types and levels of testing in this application domain. In web-based applications, the most commonly studied testing types are functional and non-functional testing at the system and unit level of the systems. In industrial automated systems the highly concentrated types and levels of testing are white box testing (6 studies) and system testing (7 studies) respectively. In network applications, only a limited number of studies were identified on different types and levels of testing. It is observed that no studies on regression and integration testing were identified in network applications and industrial automated systems.

Figure 11. Testing characteristics and types of applications

To summarize the results for RQ3, most of the agent-based software testing approaches are used to perform functional and white-box testing at system level and are targeting web-based and distributed software applications.

27

5.

Related Work

Several mapping studies have been conducted in the software testing research area [28,29,30,31,32,33,34] to identify the level of growth, usage, and effects of different types of testing in various application domains. The usage of various technologies for efficient testing has also been studied and mapped.

Over the past several years, numerous studies have been conducted to map various advanced technologies in Software Engineering (SE). To analyze the current empirical trends and its replication in Software Engineering (SE), a systematic study has been conducted by Cruz et al. [28], to find the research areas and the empirical research methods that are mostly replicated in SE. It is found that software testing is one of the most common areas in SE where replication studies are made, and a case study has been used in most of the cases in the current decade. A mapping study has been done by E. Engström and K. Petersen [29] to identify the potential gap between research and practice methods in Software testing. Several mapping studies have been conducted to study the testing trends in various application domains like Cloud applications [30], Web applications [31], and different types of testing such as Combinatorial testing [32], Structural testing [33], and Agile testing [34]. These studies aim at finding the testing methods, tools, empirical studies, areas of application, and other current trends in these fields. At present, numerous advanced techniques like intelligent agents, machine learning, etc. are highly being used in software testing. A systematic survey has been conducted by Nahal Norouzi et al. [35] to identify the different behaviors, physical forms, identifications, and other trends of the Intelligent Virtual Agents used across various domains such as education, virtual assistance, customer services and even in marketing sectors. Some studies have also been made to identify the impact of other factors in software testing, for example, the impact of the software architecture in software testing [36] and the impact of machine learning in software testing [37]. These studies were made to analyze how the different software architectures and machine learning techniques help improve the efficiency of testing, the tools used, the areas of application, empirical studies, and other trends in this domain.

To summarize, to the best of our knowledge, this is the first mapping study on agent-based software testing and more studies are needed to improve the study of agent-based software testing and its wider application to software development problems.

28

6.

Threats to Validity

In general, four types of threats to validity are addressed in a study [38]. They are construct validity, internal validity, external validity, and conclusion validity.

Construct Validity: It refers to the measure of correctness in the research method applied to the

study [38] The research method proposed by K. Petersen et al. [12] was used to conduct the systematic mapping. The steps mentioned in [12] were carried out in the same execution order. One of the major threats in a mapping study is omission of some relevant papers during the search process. This threat was mitigated by verifying several alternatives for the search strings and using some of the well-known papers in this field for validating the final string. We have cross checked the obtained results with manually obtained papers to mitigate the risks. One could improve upon this by using the snowballing technique to enhance the confidence of the results obtained in a search process.

Internal Validity: It refers to the measure of correctness of the treatment applied which may

influence the result of the study [38]. Screening of papers from the large set of samples obtained from the search result was done based on the information in the abstract, title, and keywords of the papers. In case of insufficient information available in these sections, the introduction and conclusion sections of the papers were also considered to reduce the risk of missing important papers. The classification scheme was carefully defined by iteratively revising the keywords identified in the studies and adapting the most commonly used categorizations. These facets under each classification may differ for the adoption of different classification schemes from other authors.

External Validity: It refers to the degree to which the findings of the study can be generalized

[38]. The results of this study cannot be generalized to common agent technology or software testing. It can only be considered for the reference of the agent-based systems in software testing. Despite the facets used in the classification scheme being general, the results are completely dependent on the impact of the usage of agent-based systems in software testing.

Conclusion Validity: It refers to the measure of correctness of the results obtained [38]. The

results obtained in this study were based on manual data extraction and analysis which were done iteratively and revised continuously to mitigate the risk of missing certain data in the calculations. The classification scheme used for data extraction is developed based on the revised keywords and adaptations of the commonly accepted and used categorizations. These categories are well established in both the agent and testing areas of research and have independent expert opinion from the thesis supervisors.

We also mention that this thesis does not handle any sensitive data and is based on publicly available resources and published academic sources. As this thesis builds upon previous research it is not involving any human subjects. To reduce the potential risks and threats to validity of the results obtained, this thesis is done in accordance with one of the popularly accepted research methods and various alternative measures are taken as mentioned earlier. A short description of the primary studies has also been provided in the Appendix A with an aim of maximizing the benefits to the society by making it easy to obtain a clear understanding of the research area.

29

7.

Conclusion and Future Work

The use of agent-based systems in software testing and test automation is a growing area of research that should be recognized and considered. Therefore, in this study we outline the agent-based software testing research area and conduct a systematic mapping study agent-based on identified concepts and dimensions related to agent-based systems and software testing. The study highlights that agent-based software testing refers to the application of agents (e.g., software agents, intelligent agents, autonomous agents, multi-agent systems) to software testing problems by automating complex testing tasks.

This thesis shows the results of a systematic mapping study in the agent-based software testing area. The purpose of this study is to provide a better understanding of this domain and guidance for future work. We found a total of 41 related papers in the agent-based software testing research area. Our results can be summarized as follows:

• There is an increasing interest in agent-based software testing after 1999 with the majority of studies being disseminated in conferences and proposing solutions and performing validations of ABST.

• Research in agent-based software testing has used reactive, BDI, deliberative and cooperative agent architectures. The majority of these solutions have used the Java programming language and the JADE framework.

• Most of the agent-based software testing approaches have been used to perform functional and white-box testing at system level in web-based and distributed software applications.

• The current body of knowledge concerning agent-based software testing reports only four studies on regression testing.

Through the process of conducting this systematic mapping, a variety of strong opportunities for future work were identified. Future work could include the extension of this systematic mapping study into a systematic literature review. One of the main potential gaps identified is the scarcity of research in regression and integration testing as well as the lack of practical implementations and industrial evaluations.

30

References

[1] Katalon, "The Most Striking Problems in Test Automation- A Survey," Katalon Studio & KMS Technology, May 2018. [Online]. Available: http://www.Katalon.com. [Accessed 11 April 2020].

[2] P. Ammann and J. Offutt, Introduction to Software Testing, Cambridge University Press, 2008.

[3] E. Enoiu and M. Frasheri, "Test Agents: The Next Generation of Test Cases," in IEEE

International Conference on Software Testing, Verification and Validation Workshops,

Xi'an, China, 2019.

[4] B. P. Gokulan and D. Srinivasan, "An Introduction to Multi-Agent Systems," in

Innovations in Multi- Agent Systems and Applications - 1, vol. 310, 2010, pp. 1-27.

[5] A. Spillner, T. Linz and H. Schaefer, Software Testing Foundations, USA: Rocky Nook Inc., 2014.

[6] M. Wooldridge and N. R. Jennings, "Intelligent agents: Theory and practice," The

Knowledge Engineering Review, vol. 10, no. 2, pp. 115-152, 1995.

[7] D. Ioan and B. Boldur-Eugen, "Artificial Intelligence + Distributed Systems =Agents,"

International Journal of Computers, Communications & Control (IJCCC)., vol. 4, no. 1,

pp. 17-26, 2009.

[8] A. Giret and V. Botti, "Multi Agent Systems of Multi Agent Systems," Universitat

Politecnica de Valencia, 2014.

[9] Peter Fröhlich, Iara de Almeida Móra, Wolfgang Nejdl, Michael Schroeder, "Diagnostic Agents for Distributed Systems," in Meyer JJ.C., Schobbens PY. (eds) Formal Models of

Agents. ModelAge 1997. Lecture Notes in Computer Science, vol 1760., Springer, Berlin,

Heidelberg, 1999.

[10] K. S. Decker and K. Sycara, "Intelligent Adaptive Information Agents," Journal of

Intelligent Information Systems 9., vol. 9, no. 3, pp. 239-260, 1997.

[11] M. Fraz Malik, U. Bibi, M. N. A. Khan and M. Ayaz Malik, "Multi-Agent based Functional Testing in the Distributed Environment," International Journal of Advanced

Computer Science and Applications, vol. 8, no. 8, pp. 137-143, 2017.

[12] K. Petersen, R. Feldt, S. Mujtaba and M. Mattsson, "Systematic mapping studies in software engineering," 12th International Conference on Evaluation and Assessment in

Software Engineering, vol. 17, no. 1, 2008.

[13] C. Wohlin, "Guidelines for snowballing in systematic literature studies and a replication in software engineering," in In Proceedings of the 18th International Conference on

Evaluation and Assessment in Software Engineering (EASE ’14). Association for Computing Machinery, New York, USA., 2014.

[14] R. Wieringa, N. Maiden, N. Mead and C. Rolland, "Requirements engineering paper classification and evaluation criteria: a proposal and a discussion," Requirements

Engineering, pp. 102-107, 2005.

[15] R. Girardi and A. Leite, "A Survey on Software Agent Architectures," IEEE Intelligent

Informatics Bulletin, vol. 14, no. 1, pp. 8-20, 2013.

[16] K. O. Chin, K. S. Gan, R. Alfred, P. Anthony and D. Lukose, "Agent Architecture: An Overview," Transactions on Science and Technology, vol. 1, no. 1, pp. 18-35, 2014.