Model-based Testing of supporting testing of PAC based controlling systems in industrial plants - A case study in printing plants

Full text

(1)Master of Science in Software Engineering June 2018. Model-based Testing of Supporting testing of PAC based controlling systems in industrial plants - A case study in printing plants. Jiaqi Song Jiangbiao Fu. Faculty of Computing Blekinge Institute of Technology SE–371 79 Karlskrona, Sweden.

(2) This thesis is submitted to the Faculty of Computing at Blekinge Institute of Technology in partial fulfillment of the requirements for the degree of Master of Science in Software Engineering. The thesis is equivalent to 20 weeks of full time studies.. Contact Information: Author(s): Jiaqi Song E-mail: jisa16@student.bth.se Jiangbiao Fu E-mail: jifu16@student.bth.se. University advisor: Dr. Michael Unterkalmsteiner Department of Software Engineering External advisor: Dr. Jie Xiang DinCare AS Faculty of Computing Blekinge Institute of Technology SE–371 79 Karlskrona, Sweden. Internet : www.bth.se Phone : +46 455 38 50 00 Fax : +46 455 38 50 57.

(3) Abstract Context. Testing is a very critical process to evaluate whether a related function is correctly implemented in the control system. There is an upward trend to using Programmable automation controller (PAC) based control system in the automation production context. However currently, most testing of PAC based controlling system is in manual testing, which has low efficiency and high complexity. Furthermore, there has been little research on the systematic testing of PAC in an industry environment. Objectives. Due to this problem, this study is to investigate whether a model-based testing method can overcome the challenge of manual testing and improve the testing effectiveness in PAC based controlling system. Methods. We use three steps to achieve the objective, and the first one is to implement a systematic mapping study to find existing model-based testing method that is used in the industrial area, what the process and the context. The second one is to implement a case study in a printing house, to see what the real challenge of manual testing, the third one is to find if exist Model-based testing (MBT) method could be used under such context to overcome challenges. Results. Through mapping study and case study, we found there are many testing methods and implement under diverse context, but none of them focus on the PAC based controlling system. And we found there exist MBT that also can be used in PAC based controlling system to mitigate some manual testing challenges. Conclusions. In our thesis, we implement a mapping study from 38 papers to collect data for existing model-based testing methods to have a deep understanding of this area. We found there are five main contexts that MBT are usually being used and we extract implement process and advantages and disadvantages for MBT methods which value to practitioners and researchers. And we conducted a case study in a printing house in Switzerland to observe the challenges of manual testing of PAC based controlling system. We found one exist MBT method that can be used in our context that makes the test case generation step more effective. And we proposed a simulation testing model that hopefully can address all the manual testing challenges by combined with the exist MBT method.. Keywords: Model-based testing, Programmable automation controller, Manual testing, Simulation.

(4) Acknowledgement First and most important, we thank the god Jesus for giving us the all necessary resources and letting us cross the difficulties. God has always led us. In addition, we would like to express our gratitude to many people: First and foremost, we sincerely appreciate our supervisor Dr. Michael Unterkalmsteiner for his continuous and patiently help. He gave us so many detail constructive comments and suggestions throughout the research. We cannot finish the thesis without his help. We also many thanks to our external supervisor Dr. Jie Xiang for giving us many supports. With his help that we can conduct the research in a real industrial plant. Besides, we would like to thank the examiner Dr. Jürgen Börstler for his professional feedback for our proposal and thesis. Finally, We thank all friends and family for their encouragement and help.. ii.

(5) Contents. Abstract. i. Acknowledgement. ii. List of Figures. vi. List of Tables. viii. Glossary. 1. Acronyms. 2. 1 Introduction 1.1 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Thesis Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 Thesis Organization . . . . . . . . . . . . . . . . . . . . . . . . . . .. 3 3 4 4. 2 Background and Related Work 2.1 Background . . . . . . . . . . 2.1.1 Company . . . . . . . 2.1.2 Industry automation . 2.1.3 Printing automation . 2.1.4 Model-based testing . 2.2 Related Work . . . . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. 6 6 6 6 7 7 8. 3 Research Methodology 3.1 Aim and Objectives . . . . . . . . 3.2 Research Questions . . . . . . . . 3.3 Research Design . . . . . . . . . . 3.4 Research method comparison . . 3.4.1 Systematic Mapping Study 3.4.2 Case Study . . . . . . . . 3.4.3 Literature review . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. 10 10 10 11 11 11 12 12. . . . . . .. iii.

(6) 3.4.4 3.4.5. Survey . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Controlled Experiment . . . . . . . . . . . . . . . . . . . . . 13. 4 Systematic Mapping Study 4.1 Definition and Scoping . . . . . . . . . 4.1.1 Defining Objectives . . . . . . . 4.1.2 Defining Research Questions . . 4.2 Conduct Search . . . . . . . . . . . . . 4.2.1 Search Strings . . . . . . . . . . 4.2.2 Database Search . . . . . . . . 4.3 Screening of Relevant Papers . . . . . . 4.3.1 Inclusion and Exclusion Criteria 4.3.2 Snowballing . . . . . . . . . . . 4.3.3 Quality assessment . . . . . . . 4.4 Classification Scheme . . . . . . . . . . 4.5 Data preparation . . . . . . . . . . . . 4.6 Data Extraction and Analysis . . . . . 4.6.1 Data collect for RQ 1.1 . . . . . 4.6.2 Data collect for RQ 1.2 . . . . . 4.6.3 Analysis for RQ 1.1 . . . . . . . 4.6.4 Analysis for RQ 1.2 . . . . . . . 4.6.5 Potential MBT for Case study . 4.7 Validity Threats . . . . . . . . . . . . . 4.7.1 Theoretical Validity . . . . . . . 4.7.2 Interpretive Validity . . . . . . 4.7.3 Generalizability Validity . . . .. . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . .. 14 15 15 15 15 15 16 17 17 17 17 18 20 21 23 27 30 32 33 34 34 34 34. 5 Case study and Results 5.1 Case study design . . . . . . . . . . . . . . . . 5.1.1 Objectives . . . . . . . . . . . . . . . . 5.1.2 Case description . . . . . . . . . . . . . 5.2 Preparation for data collection . . . . . . . . . 5.2.1 How to collect data . . . . . . . . . . . 5.2.2 Data analysis guidelines . . . . . . . . 5.2.3 Collection techniques and observations 5.3 Data collection and Analysis . . . . . . . . . . 5.3.1 Data collection for RQ2 . . . . . . . . 5.3.2 Testing context overview . . . . . . . . 5.3.3 Analysis for RQ2 . . . . . . . . . . . . 5.3.4 Model-based Testing of Software . . . . 5.3.5 Proposed Simulation Model . . . . . .. . . . . . . . . . . . . .. . . . . . . . . . . . . .. . . . . . . . . . . . . .. . . . . . . . . . . . . .. . . . . . . . . . . . . .. . . . . . . . . . . . . .. . . . . . . . . . . . . .. . . . . . . . . . . . . .. . . . . . . . . . . . . .. . . . . . . . . . . . . .. . . . . . . . . . . . . .. . . . . . . . . . . . . .. 36 36 36 36 37 38 38 38 39 39 39 45 46 51. iv. . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . ..

(7) 5.4. 5.3.6 Analysis for RQ3 . Validity Threats . . . . . . 5.4.1 Construct Validity 5.4.2 Internal Validity . 5.4.3 External Validity . 5.4.4 Reliability . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. 58 61 61 61 61 61. 6 Conclusions and Future Work 62 6.1 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 6.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 References. 63. Appendix A Manual Testing Activities. 70. Appendix B Steps to Make Simulation Model. 76. Appendix C Data Extraction Template. 84. Appendix D Statistics of Database Search. 86. v.

(8) List of Figures. 2.1. Model-based testing workflow . . . . . . . . . . . . . . . . . . . . .. 3.1. Relationship between Research Questions and Methodologies . . . . 11. 4.1 4.2 4.3 4.4 4.5. Process of Systematic Mapping Study . . . . . . . . . . . . . . . . . Process to screen primary paper . . . . . . . . . . . . . . . . . . . . Classification Schemes of model-based testing studies . . . . . . . . Number of Papers Per Year Published in Workshop and in Journal Context that using MBT method . . . . . . . . . . . . . . . . . . .. 14 18 19 22 32. 5.1 5.2 5.3 5.4 5.5 5.6 5.7 5.8 5.9 5.10 5.11 5.12 5.13 5.14 5.15 5.16 5.17 5.18 5.19 5.20 5.21 5.22. Process of case study . . . . . . . . . . . . . . . . . Case description . . . . . . . . . . . . . . . . . . . . Machine controlled by PAC . . . . . . . . . . . . . PAC connect to specific signal . . . . . . . . . . . . PAC . . . . . . . . . . . . . . . . . . . . . . . . . . Input and output modules . . . . . . . . . . . . . . Signal testing part of the system . . . . . . . . . . . Testing logic of software for signal testing . . . . . . Manual testing process . . . . . . . . . . . . . . . . Simulation chart of STB & DBH . . . . . . . . . . Real machine of STB & DBH . . . . . . . . . . . . Functional testing part of the system . . . . . . . . Testing logic of software for functional testing . . . Manual testing process for function . . . . . . . . . Sample STB . . . . . . . . . . . . . . . . . . . . . . Sample DBH . . . . . . . . . . . . . . . . . . . . . SFC of STB . . . . . . . . . . . . . . . . . . . . . . SFC of DBH . . . . . . . . . . . . . . . . . . . . . . Transfer testing context from real machine to model Model Testing Interface . . . . . . . . . . . . . . . Process to create model and test cases . . . . . . . Class diagram of simulation model . . . . . . . . .. 36 37 39 40 40 40 41 41 42 43 43 44 44 45 47 48 49 50 52 53 53 54. vi. . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . .. 7.

(9) 5.23 Manual testing context . . . . . . . . . . . . . . . . . . . . . . . . . 55 5.24 Testing context in simulation context . . . . . . . . . . . . . . . . . 55 5.25 Testing process in simulation model . . . . . . . . . . . . . . . . . . 56. vii.

(10) List of Tables. 4.1 4.2 4.3 4.4 4.5 4.6 4.7. Research Questions and Motivations . . . . . Classification Scheme . . . . . . . . . . . . . . Data Extraction Form . . . . . . . . . . . . . Data Extraction of context and process . . . . Data Extraction of advantages and drawbacks Data Extraction Form of contexts . . . . . . . Summarized advantages and Drawbacks . . .. 5.1. Data preparation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38. viii. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. 15 19 20 23 27 31 33.

(11) Glossary DBH German:DECKBLATTHERSTELLUNG English:Cover label production Is a Combination of hardware in printing house to press label on paper. 42 STB German:STAUBAND English:Dust-proof Is a Combination of hardware in printing house to transfer paper. 42. 1.

(12) Acronyms MBT Model-based testing. i PAC Programmable automation controller. i PLC Programmable logic controller. 4 SFR Sequential Function Charts. 33, 34 SUT System under testing. 4 SysML Systems Modeling Language. 24. 2.

(13) Chapter 1. Introduction. Testing is essential, but time and resource consuming activity in the software development lifecycle. With the increasing demand for machines, robots, conveyors and other automation components in manufacture plant, the software-based controlling system becomes much more critical and complex than ever before. To guarantee the reliability and accuracy of the controlling system, more and more efforts are required for software testing[1].. 1.1. Problem Statement. In industrial plants, software is required to be tested together with hardware. In this situation, the complexity and time are much more than pure software testing. According to the investigation report for the German Association of Machine Tool Builders (VDW) at [2], during the software development, ninety percent of the commissioning time is consumed by hardware related software development, where Seventy percent of the commission time is spent for debugging. Because of the combination of hardware and software which increase the systems’ complexity. It is hard to handle this situation [3]. Furthermore, there is a growing ratio of software development in the costs of machinery. It has increased from 20% to 40% in one decade between 1990 and 2000. This trend continues because of the development of software which shows that the primary activity of automation systems developers will be software engineering [1]. Quality assurance for evolving automated production systems is still a challenge that needs to focus on [4]. During the testing stage, it is usual to perform testing related activities manually. Manual testing has a low efficiency and errorprone, resulting in unexpected machine behaviors when the system is operating [5]. model-based testing (MBT) is a testing technique that can generate test cases based on the source code or models. In MBT, test cases are automatically derived 3.

(14) Chapter 1. Introduction. 4. and executed based on the models of the System under testing (SUT) which are created before. [6] According to the study of Schulze et al. [6] It shows that more problems with high severity score are detected and less testing and analysis effort are needed than when testing manually. In this thesis work, we investigated the model-based testing for PAC based controlling systems in industrial plants. Especially, we did a case study in Mediencentrum AG, a printing plant in Switzerland, which will upgrade all the existing Programmable logic controller (PLC) based controlling system to PAC based controlling system.. 1.2. Thesis Contributions. In the thesis work, we have implemented a systematic mapping study and a case study to explore an effective model-based testing method for PAC based controlling systems, our main contributions are concluded as follows: • We conducted mapping study to identify using context, advantage, and disadvantages of current model-based testing methods which used in the industrial area. These data support practitioners to find what kind of MBT is proper for them. • We conducted a case study in a printing house and identified the main disadvantages and limitations of manual testing under PAC based controlling systems. Practitioners can prepare beforehand if they face the same challenges. • We combined a current model-based testing method and a proposed simulation model that mitigate the manual testing challenge of PAC based controlling system. Practitioners can reference our work to do more try in this area.. 1.3. Thesis Organization. The rest of this thesis is structured as follows. Chapter 2 presents the background and related work of our research in testing aspect of the automated controlling system. Chapter 3 describes the methods that were used in this work. Chapter 4 detail shows how we implemented the systematic mapping study and the extracted data was also analyzed. Chapter 5 presents the results for case study and each research question is analyzed and discussed, and we also described one MBT method.

(15) Chapter 1. Introduction. 5. and proposed simulation model. Chapter 6 draws the conclusion and future work that can be continued based on this research..

(16) Chapter 2. Background and Related Work. 2.1 2.1.1. Background Company. Unimail Systems is a software company that focuses on automation control area that offers related service in Europe. It is a is a subsidiary of Agraf which has more than 120 years of family history in the printing industry. Mediencetrum AG is a new printing house in Switzerland that use second-hand production line devices. In the thesis case, Unimail Systems is responsible for controlling these devices by using the PAC based controlling system.. 2.1.2. Industry automation. Recently, the industry automation gradually recognized by various industries. People use control systems, such as computers and robot to deal with different processes’ information technologies to instead of people. Earlier, the industrial automation can increase the productivity. Now, the most important contribution of the industrial automation is to increase quality and flexibility in an industrial process. For example, after using automated machinery to install pistons into the engine in the automobile industry, the error rate declined from 1-1.5% to 0.00001% [7]. All in all, the advantage of industry automation conclude lower operating cost, high productivity, high quality, high flexibility and high safety [8]. For a software developer, the most important challenge is to shorten the time of product development and the cost of the system startup. After achieving these goals, the products will be brought to the market faster and more efficiently.. 6.

(17) Chapter 2. Background and Related Work. 2.1.3. 7. Printing automation. Newspapers and magazines are circulated in our daily life. In the last decades, many large printing companies introduce advance printing technology and equipment [9], and these devices are normally running over ten years controlled by the old system until now. In 21 century, with the development of technology and the conception of industry 4.0, it necessary for the printing industry to make an automated improvement. In order to be competitive, the printing industry needs application expertise, engineering solutions, and manufacturing excellence to optimize the operations. From print control systems to operator interface systems, the software developer needs to increase efficiency, improve printing quality and reduce waste.. 2.1.4. Model-based testing. model-based testing is used in executing software testing or system testing based on the model-based design. The tester can use the models to shows the desired behavior of a SUT. The tester can also display the test environment and the testing strategies. The steps are as following [10]: Firstly, the SUT’s desired behavior is described as a model abstractly. Then, the model derives test cases. It’s the functional test on the abstract level. Corresponding abstract test suite derives the executable test suite. In the end, the system will communicate with the executable test suite. Next, we present a diagram to show how does it work. In the diagram, the test oracle determines whether a test has passed or failed. The executor runs the test cases and gets the result of the testing.. Figure 2.1: Model-based testing workflow.

(18) Chapter 2. Background and Related Work. 2.2. 8. Related Work. In industry automation area, many challenges are in the development of software [3]. Quality assurance is still a challenge that worth be paid attention [11]. Many pieces of research focus on this part in recent decades. Vyatkin et al. [12] proposed a framework which is based on a modeling formalism of Event Systems/Net Condition. It supports to model realistic hierarchically organized automation systems in a closed cycle. The framework is used for formal modeling and validation for the automation system that improves the system’s reliability and robustness. Mazzolini et al. [13] proposed an approach based on model checking method and model coverage properties for the verification of logic control systems in the industrial automation area. The approach to a great extent reduces the effort of related control solution verification and validation. Racchetti et al. [14] proposed Timing Diagram Testing for Verification Validation for industrial automation systems. It generates test cases based on the timing diagrams of software requirements. These test cases are then running on PLC to check the system’s conformance through an algorithmic verification. It improves the testing effectiveness. All these approaches are mainly under PLC based controlling system. MBT is a popular method used in industry as it can provide techniques for test cases automatic generation based on the models extract from the specific system [15]. MBT is very appealing because it has three main uses: they can be used to handle explicit requirements descriptions, they can be part of the specification document, they can be used to generate test sequences [16]. Dias-Neto et al. [17] made a survey about the quantitative and quantitative analysis of MBT approaches from 78 technical papers. The result shows MBT methods are rarely combined in the software development process, and most methods stay in academic research which is not evaluated or transferred to the industrial context. In our study, we implemented a case study under a real industrial environment that make a try to transfer academic to industry context. To find if exist current MBT could be used under PAC based controlling system. According to [18], it shows many activities of industrial control software development and testing still have to be implemented manually in most instances. Schulze et al. [19] made a comparison between manual testing implemented by a tester without any automation in a software company and model-based testing conducted by a tester in a research center. The result shows MBT hold 60% higher severity score than manual testing. It means MBT can detect more issues with less testing and analysis. In our study, we are trying to investigate an MBT method that can address the challenges of manual testing encountered under PAC based controlling.

(19) Chapter 2. Background and Related Work. 9. system in the case company. However, for the relevant related work, none of them are the research on the testing area of PAC based controlling system and the benefit of these methods is unknown under PAC context. Most of these methods are mainly focus on the test case generate or just on hardware control logic. As PAC based controlling system usually being tested should think about the logic of server program at the same time. We are the first to implement a case study under PAC based controlling context to identify challenges of manual testing and then we combined a current MBT method with proposed simulation model try to address the manual testing challenge..

(20) Chapter 3. Research Methodology. 3.1. Aim and Objectives. The aim of our research is to investigate effective model-based testing for PAC based controlling systems in industrial plants. We make a set of objectives as follows to meet this aim. • Investigate the existing Model-based testing methods • To understand the production context of the printing house • To find out whether there exists a model-based testing method that can be used under PAC based controlling system.. 3.2. Research Questions. In our research, we are focusing on the model-based testing method in the industrial area. At first, we need to know some existing model-based testing methods used in industry and acquire a deep understanding of these methods. We plan to analyze them in theory based on our practical experience and knowledge. Hence the RQ1 is formulated as follows: RQ1: Which model-based testing methods are used for industry plant? RQ 1.1 What is the context of use and implement ion process of the methods. RQ 1.2 What are the advantages and drawbacks of the methods.. Currently, testing of PAC based controlling systems is done manually in our case company. We intend to study the challenges and limitations of this approach. 10.

(21) Chapter 3. Research Methodology. 11. Hence the RQ2 is formulated as follows: RQ2: What are the main challenges of manually testing PAC based controlling systems in industrial plants?. To improve the effectiveness of the testing process. We plan to find whether there exists model-based testing that could be used in the case company. Hence the RQ3 is formulated as follows: RQ3: Which model-based testing method could be used to test a PAC based controlling system? To what extent can it address the challenges encountered in the case company?. 3.3. Research Design. We choose systematic mapping study and case study to answer the research questions. Figure 3.1 shows the relationship among them:. Figure 3.1: Relationship between Research Questions and Methodologies. 3.4 3.4.1. Research method comparison Systematic Mapping Study. Systematic mapping study is a method to classify the related field of interest based on designed scheme or structure.[20] Normally the study of these fields is less explored. [21] As the field of testing of PAC based system controlling system is rarely studied. And the production context is diversification in industry automation area. We want to get an overview of this topic. By implementing.

(22) Chapter 3. Research Methodology. 12. this method, we can find core relevant papers of our studies and then build a classification in the area and find MBT methods using in different contexts which can give us a high-level understanding structure of this topic.. 3.4.2. Case Study. For the case study, we have an opportunity to do our research in a real production context in a printing house. It is easy to get real and range of data, based on the actual situation, we can more likely to develop a model-based testing method for PAC controlling system. The case study is a research method to explore the phenomenon or subject in a real context. It is suitable to use case studies for the industry to evaluate software engineering methods and tools because scale-up problems can be avoided.[22] In order to answer the RQ2, we are as manual testing stuff that fully participates in the project and will collect data of testing activities, manual testing process, effort, resources needed. Based on collected data, we can find actual problems and limitless and try to solve them in the proposed model. For RQ3, we will try to find if there exists an MBT testing method from systematic mapping study that can be used under the case company context. If exist, we will implement it to see to what extend address the challenge from RQ2. If not exist, we will analyze why these methods can not be used in the case company.. 3.4.3. Literature review. Literature review is a method conducted to collect useful information from literature for the defined problem domain area [23]. It focuses on the aggregating and synthesizing evidence and should formulate a very specific goal. [20] In our study, we are trying to discover the trend of the research of MBT and then structure understand deeply about this area. We do not have a very specific goal that needs to gather evidence. Hence, this method is not proper for the study.. 3.4.4. Survey. Survey is a research method have three main objectives: descriptive, explanatory, explorative [21]. that normally need a relatively large sample of people from a.

(23) Chapter 3. Research Methodology. 13. population of interest, and then researcher use collected information from the individual samples to make inferences about the broader population [24]. Our study focuses on the MBT that using in the industrial automation area not ideas from different practitioners. And it hard for us to find a predetermined population in this area. Hence, the survey could not conduct in our study.. 3.4.5. Controlled Experiment. The controlled experiment is “ an empirical inquiry that manipulates one factor or variable of the studied setting.” [21]. It normally conducted in the environment of a laboratory which offers a high level of control [21]. Our study is to find exist MBT method that could be used in a real context of PAC based controlled system. It hard to manipulates one factor or variable in the real production context. Hence, we do not use the controlled experiment in the study..

(24) Chapter 4. Systematic Mapping Study. We implemented the Systematic Mapping Study(SMS) to know more about Modelbased testing method in the automated controlling system area. The steps we are following was described by K. Petersen et al [20]. Figure 4.1 shows the SMS process:. Figure 4.1: Process of Systematic Mapping Study We are two people in thesis group, we define the scope and keywords by discussing together. For paper selection, we conduct the search together and we independently select relevant papers respectively. Then we sum all the papers together to exclude duplicate papers. Then we check each other’s selected papers by reading abstracts and introduction to make a consensus what papers should be finally selected. For data extraction, to make the result of the data consistent, we made a data extraction form template, as shown in Appendix C. We can follow the template to extract data.. 14.

(25) Chapter 4. Systematic Mapping Study. 4.1. 15. Definition and Scoping. 4.1.1. Defining Objectives. The method of systematic mapping study is mainly to help us solve the first research question as shown in section 4.1.2. We want to know more knowledge about this area (Testing in industry plant). We main to identify related research and results for our study.. 4.1.2. Defining Research Questions. RQ1: Which model-based testing methods are used for industry plant? RQ 1.1 What is the context of use and implementation process of the methods. RQ 1.2 What are the advantages and drawbacks of the methods. To answer these questions, we intend to know more about existing model-based testing methods of the automated controlling system.. Table 4.1: Research Questions and Motivations Research Questions RQ 1.1 What is the context of use and implementation process of the methods.. RQ 1.2 What are the advantages and drawbacks of the methods.. 4.2 4.2.1. Motivation Give an overview of different methods conducted under different contexts, what is the character of related methods. Presents an overall comparison of different methods, and this result is good for us to get useful information to have a better understanding of the related method and to find whether there exist MBT method could be used for PAC based controlling system.. Conduct Search Search Strings. We defined the search string based on PICOC(Population, Intervention, Comparison, Outcome, Context ).

(26) Chapter 4. Systematic Mapping Study. 16. We are main to research about testing, so our population is defined below Population: test , testing , verification , validation , inspection , commissioning , "quality evaluation" , "quality analysis" , "quality assurance" , "quality assessment" The context mainly focus on industrial plant and automation area and related component. Intervention: "model-driven" , "model driven" , "model-based" , "model based" , "MBT" We focus on the method of Model-based testing to improve the testing effectiveness f, PAC based control system in the industrial plant. Context: "industrial controller" , "automation system" , "automation controller" , " industrial automation" , "automation and control" , "manufacturing systems" , "industrial plants" , "production plants" , "production systems" , "production line" We are not to compare MBT method with other testing methods. We just compare the data (context, advantage, and drawbacks) which extract from papers within MBT. Hence, we do not need to reflect relevant keywords in the search string. We also not need to accomplish or improve specific method as we do not have any methods in mind. Our plan is to investigate not to make outcome from MBT methods. We also did not use the outcome to form search string. We use three aspects (Population, Context, Intervention) form into Boolean expression, we combined the keywords of them by using Joiner ‘&’: population & Context & Intervention. And then we conduct the search in the digital database by using combined Boolean.. 4.2.2. Database Search. We mainly use five digital databases to implement search: Scopus, ACM, IEEE, ScienceDirect, Wiley Appendix D shows the search string and the number of results..

(27) Chapter 4. Systematic Mapping Study. 4.3 4.3.1. 17. Screening of Relevant Papers Inclusion and Exclusion Criteria. Inclusion: Papers focus on the testing area in industrial area and papers mention about Model-based testing in software engineering or Industrial automation engineering. Free in the BTH library databases. Exclusion: Papers are not in the testing area and papers are not in software engineering or Industrial automation engineering. Papers are non-peer-reviewed or non-English papers. Paper is a duplicate article. Full-text not available for the paper.. 4.3.2. Snowballing. Snowballing is a technique to find primary papers by “following the references from or to one paper to find other relevant papers” [21] After screen papers, one person conducted the snowballing method to see if there are related papers that also fulfill our inclusion criteria by reviewing the references and citations of each screened paper. The other person checked the newly selected papers to make a consensus. We used the backward snowballing that following the reference list to find papers.. 4.3.3. Quality assessment. We implemented quality assessment to find primary papers. We are mainly concern three parts: 1. Is the method related to model-based testing? 2. Is the testing method used for industry area? 3. Is the testing method clear described? As figure 4.2 shows the process of how to find the 38 primary studies..

(28) Chapter 4. Systematic Mapping Study. 18. Figure 4.2: Process to screen primary paper. 4.4. Classification Scheme. We develop a classification scheme before the data extraction to answer RQ1. As figure 4.3 shows the Classification Scheme. Primary studies are classified into three different categories: venue, method and study focus. Through the classification scheme, we can have an overview more understand the screened paper: where the paper come from, what research method they are using, what the study focus..

(29) Chapter 4. Systematic Mapping Study. 19. Figure 4.3: Classification Schemes of model-based testing studies. Table 4.2: Classification Scheme venue. journal Conference. method. experiment survey. Case study. Study focus. academic industrial. [25] [33] [39] [48] [57] [43] [43] [25] [42] [51] [38] [30] [25] [26] [43] [53] [49]. [26] [5] [27] [28] [29] [30] [34] [35] [36] [37] [38] [40] [41] [42] [43] [44] [45] [49] [50] [51] [52] [53] [54] [58] [15] [59] [60] [46] [34] [53] [48] [30] [26] [5] [27] [39] [29] [40] [32] [33] [44] [45] [46] [34] [52] [53] [54] [55] [56] [57] [47] [48] [49] [15] [50] [59]. [31] [32] [46] [47] [55] [56]. [31] [35] [37] [60]. [41] [36] [58] [28]. [39] [28] [33] [36] [47] [5] [27] [29] [30] [40] [31] [41] [42] [32] [44] [45] [46] [34] [35] [51] [52] [54] [55] [56] [57] [37] [58] [38] [48] [15] [50] [59] [60].

(30) Chapter 4. Systematic Mapping Study. 20. As table 4.2 shows, most papers are from the conference and papers are cover three research method, which can give us a dierent perspectives of our topic. And most papers focus on industrial, which is very proper for us to refer and conduct research in the real industry automation plant.. 4.5. Data preparation. For RQ 1.1, We need the data: (a) The method being implemented under what kind of industrial plant, e.g. Printing house or other kinds of the production line (b) The process of the method is implemented For RQ 1.2, We need the data: (a) Advantages and drawbacks mentioned in the paper After the classification scheme, we extract data from the main studies. According to the table 4.3, we can collect all of the information to answer RQ1. Each data extraction field has a data item and a value. We can get the general study information and the specific information for the RQ1 in the database collection. Table 4.3: Data Extraction Form Data item General Information Study ID Article Title Author Name Year of Publication Venue Contribution Research method. Value. Relevant RQ. Integer Names of the articles Set of names of authors Calendar year Name of publication venue The model or the theory of study outcomes The research method of the research process. None None None None None None None. Specific Information Keywords. The keywords we find in the abstracts. None.

(31) Chapter 4. Systematic Mapping Study. The context of implementation Process advantage drawbacks. The method being used under what kind of industry plant, e.g. Printing house or other kinds of production line The process of the method being implemented Advantages and drawbacks mentioned in the paper Advantages and drawbacks mentioned in the paper. 21. RQ1.1. RQ1.1 RQ1.2 RQ1.2. In order to understanding, structuring, and classifying operating study on the Model-based testing method in the automated controlling system area. We mainly use a qualitative method to analyze the data extracted from the important papers. As we have reviewed many papers, we use tables to present the collected data. To fully understand the MBT methods, three essential data that we collect are the context of the method being used, the implementation process of the MBT method, advantages and disadvantages of MBT methods. Readers can have a good overview of the data through tables. Based on the collected data, we summarized the data of contexts and advantages and disadvantages. To find a potential MBT method for the case study, we select method to follow two steps: first highly think about the context that similar to the case company, after find potential MBT methods we then compare them according to the advantages and disadvantage. To see which one to a greater extent that could potentially address the challenges encountered in the case company.. 4.6. Data Extraction and Analysis. In this chapter, we generate the result based on the application of the SMS protocol yielded. We selected 38 papers to answer RQ1 from the initial search result of 2687 publications of the model-based testing of software engineering. Based on the different venues classified for the data extraction, we show the result in Figure 5.1. It’s common to analysis the specific venues of publication.

(32) Chapter 4. Systematic Mapping Study. 22. Figure 4.4: Number of Papers Per Year Published in Workshop and in Journal Figure 4.4 shows the number of Papers Per Year Published in the conference and in Journal It’s 23 papers of the 38 papers are published in the conference, the other are journal articles. What’s more, the most number of papers published in International Conference on Software Engineering. Such as Engineering of Complex Computer Systems (ICECCS), Model-Driven Engineering and Software Development (MODELSWARD), Emerging Technologies and Factory Automation. We can distinguish the papers into two periods with the time of the papers published. The period of time (before 2010) can be seen as the introductory period because the number of papers before 2010 is only 5. And there is 0 paper during the period from 2002 to 2004. Actually, the first relevant paper is published in 2001 which introduce a way to self-verifying systems.[61] And then, the number of papers is more than 5 from 2012 to 2015, the most number of papers in 2013 and 2017. In the end, there is a decreasing period (2015–2016). The papers published in the conference is more than in journals. It’s 39% papers are published in journals, and 61% papers are published at conferences. Based on the research questions mentioned in the last chapter, we structure this chapter. We introduced the implementation process of the methods and the advantages and drawbacks of the methods. Table 4.2 shows the screened paper from journal and conference..

(33) Chapter 4. Systematic Mapping Study. 4.6.1. 23. Data collect for RQ 1.1. The generic MBT process is described in section 2.1.4. As Table 4.4 shows, we extracted the specific MBT process steps that described in the papers. Table 4.4: Data Extraction of context and process Reference. Context. Mathias et al. (2014)[30]. A laboratory plant which sorts and marks workpieces.. Gupta et al. (2014)[26]. A simple communication system – INRES to study the use of MBT in the industrial automation.. 1. Develop the abstract model. 2. Develop the abstract test suit. 3. Bridging levels of abstraction and test harness.. It use for PLC programs which integrated in automated systems.. 1. First sends to the I/O module an input combination that corresponds to the input event of this step 2. observes the sequence of outputs that are generated at the end of every cycle by the PLC in response to this input change 3. elaborates a test verdict on the basis of the observed and expected output sequences; once the verdict obtained, the input combination that corresponds to the next test step is sent to the I/O module if the verdict is positive (OK). 4. Test execution is stopped otherwise (verdict KO). Ambert et al. (2017)[25]. Process 1. Extraction of information from Timing sequence diagram (TSDs) and Failure mode and effects analysis (FMEA) to a sequence diagram(SD). 2. Transfer the test code manually 3. Inject the fault if the Precondition is met. 4. After the defeat, the Precondition for a test case, start FI (Fault Inject) and monitor the reaction of the plant. 5. Log the result and stop the system..

(34) Chapter 4. Systematic Mapping Study. Hilken et al. (2015)[5]. A demonstration plant. Borja et al. (2013)[51]. A center for particle physics research.. Rauter et al. (2013)[43]. A control application for a heat production plant (HPP). Schaefer et al. (2012)[46]. The method studied 4 complementary automotive cases: a car front lighting system, the electronic control of a car driver seat management, a wiper system of a car and the behaviors of a car steering column.. 24. 1. The documentation of requirements is written in a test model specification of a PLC software 2. Generate test cases by using the domain-specific languages of Sequential Function Charts 3. Write PLC code independently from the specification of creating the test model 4. Data traces (Inputs and outputs of the PLC and with timestamps) are recorded during the test case execution 5. Running the learned MSPN in parallel to the actual process to see whether event traces are reflected in the MSPN. 1. Correct the important BIP components and connectors. 2. Correct the behavior of the BIP components. 3. Testing and verification by the monitor components. 4. Compare the real value and the test oracle of the three conditions after executing the PLC logic. 5. If the values are same, the transition return is performed. If not, an error in the program has been found. 1. writing SUT input. 2. Start the execution of a number of cycle of PLC. 3. Waiting for the update outputs of PLC and OPC. 4. Reading the outputs of SUT Passing the determination.. 1. Specifying a Systems Modeling Language (SysML) test model of SUT. 2. Translate the SysML model into SysML4MBT model 3. Test designer generate test cases and the expected behavior of the SUT automatically. 4. Export test cases into TestInView platform and generated test cases into concrete scripts. 5. Comparing the execution results with the expected result automatically..

(35) Chapter 4. Systematic Mapping Study. Can long et al. (2017)[52]. A welding and material handling cell.. Jonathan et al. (2011)[53]. A vehicle front axle unit.. Mirko et al. (2005)[54]. The software embedded in automotive control systems.. Rauter et al. (2016)[41]. A distributed manufacturing system for Programmable Logic Controller (PLC) devices.. Brand et al. (2008)[39]. The battery model fragment.. Mordinyi et al. (2012) [28]. A basic artifact in the developing power plants.. 25. 1. Generate test cases and build a test sequence 2. Put the input sequence into the programmable controller 3. Execute the implemented program. 4. Compare the output. 1. Make a UML behavioral model of the SUT environment. 2. Generate test cases with model coverage criteria. 3. Translate these configurations into test scripts. 4. Execute the test scripts on a Physical Test Bench to compare the real result and the expected result. 1. Requirement-based test design: Define the logical test scenarios by the requirement specification of the embedded system. 2. Model-based test design: Refine the requirementbased logical tests to executable model-based test scenarios. 3. Consistency check: Check if the executable test scenarios cover all of the logical test scenarios. 1. Create empty test 2. List features 3. For all test steps: a. Find the components’ test cases. b. Exam test steps constraints. 4. For all test cases: a. exam feature requirement. b. Fulfill feature requirement. c. Configure propertytemplates d. add to test. 1. The simulation from QR model to QR TS. 2. The conversion of the TS into an LTS with the test purpose. 3. The product of the system LTS with the test purpose. 4. Minimize and ensuring determinism → CTG. 5. Extraction of controllable test cases from the CTG. 1. Tool domain interfaces and method calls. 2. Connect tool and code resources to verify if fulfill all restrictions. 3. Transfer data models of tool instances to the data model of the tool domain. 4. Generate test cases with the signature of the tool domain..

(36) Chapter 4. Systematic Mapping Study. 26. Noten et al. (2017)[40]. A cupcake manufacturing production line.. 1. Describe implicit physical interactions. 2. Describe workpiece state. 3. Modeling complex workpieces traveling through the line. 4. Prevent UML validation issues.. Keränen et al. (2012)[34]. Two use cases were proposed based on the MBT in HIL prototype platform architecture; the first use online MBT, and the second offline MBT.. 1. Creating the abstract reference model according to SUT requirements specification. 2. Checked to ensure the model has the desired behavior. Automatic test generation.. Iyenghar et al. (2010)[35]. A target evaluation board MCB1700 with a set of LEDs.. Estevez et al. (2012)[37]. A heat treatment line. Vain et al. (2017)[57]. A time-critical distributed system.. You, D. et al. (2015)[58]. A PCA (PatientControlled Analgesia) infusion pump system.. 1. Transform the real-time system’s requirements specification to design models. 2. Automatic code generation. 3. Transform the specific test models to executable test cases. 4. Execute test cases with inserting the test stimuli via the monitor into the embedded target. 5. Visual the result with the target debugger. 1. Define “what” the control system has to do. 2. Select the equipment needed to the control system. 3. Define the software architecture according to the functional specification. 4. assure the definition of a complete and consistent model. 1.Construct the test model. 2. Test propose specification. 3. Test generation. 4. Test deployment. 5. Test execution. 1. Device Configuration 2. Sensors and Actuators simulation. 3. Generate test case. 4. Generate mutation..

(37) Chapter 4. Systematic Mapping Study. Guan, J. et al. (2015)[56]. 4.6.2. An industrial software system.. 27. 1. Generate test models with Constructing the CREMTEG. 2. Generate test models with Generating Test Paths from the CREMTEG. 3. Test Adequacy Criteria with three test criteria. 4. Generate test data. 5. Input the test data to the test automation tool. 6. Inspect output log files, analyzes the results to make sure if the test is successful. 7. Generate final test report.. Data collect for RQ 1.2. Based on our understanding of the papers, we concluded the advantages and drawbacks of papers in table 4.5: Table 4.5: Data Extraction of advantages and drawbacks Reference Mathias et al. (2014)[30]. Gupta et (2014)[26]. al.. Ambert et al. (2017)[25]. Advantages The author can record the reaction of the plant and document the test coverage of Fault and the robustness of the plant by FI (Fault Inject) The paper explored the method for testing PLC’s using nondeterministic test models can be used in system migration, integration, and regression testing. It are reliable to detect transfer and output fault in the consecutive PLC cycles by observing the sequence of outputs of these cycles. All states and input/output events have transferred to variables and let the test more logical by using mathematical theory.. Drawbacks Need a lot of work to do, such as the extraction of the CFG and overlying a test function. Because they need to face more complex scenarios. The author needs further research to reassure the obtained results. And they need to generate the test case with other coverage criteria. The paper needs extensions to the model rather than generate TTCN-3 test cases. This method only for singlethreaded PLC that integrated into automated systems. During the implementing stage, real machines are not taken into consideration..

(38) Chapter 4. Systematic Mapping Study. Hilken et (2015)[5]. al.. Borja et (2013)[51]. al.. Based on the combination of model-based testing, model learning techniques, and anomaly detection, the systematic approach let tester or commissioning engineers to have a good view of related events and inputs/outputs. And after the MSPN is completed(different events are defined) and there is anomaly category that helps them to make decisions and found anomalies. The author can achieve early error detection by testing the UNICOS application at the same time. The implementation and interpretation of the test are achieved automatically by a simple specification. The complexity of building a BIP model is hidden by establishing conversion rules between UNICOS and BIP.. 28. Test cases will grow due to the complexity of the plant. In this situation, heuristic or filters are needed to support them to reduce the number of anomalies. Do not have automatically potential anomaly detection to improve the testing efficiency.. Native C functions which cannot explore should be avoided as much as possible. And BIP treat them as black box.. The author consider the testing requirement from functional aspect. The author achieves the test coverage goal by the test generation tool.. It is awkward for control engineers because the state transition based models, while the PLC model is data flow oriented. The effort of modeling is high. FAT testing cannot cover the cost.. Schaefer et al. (2012)[46]. The prototype proposed an integrated approach and continuous process and the 4 experiment are successful.. It is a trouble that translates the abstract test cases into concrete ones manually.. Can long et al. (2017)[52]. The paper reduce the number of test cases by getting signal features extracted from simplified plant models and don’t need a detailed plant model.. Author uses a communicating Moore machine with Boolean signals and algorithms in the implementation process. This makes the proposed framework execution cost high.. Rauter et (2013)[43]. al..

(39) Chapter 4. Systematic Mapping Study. Jonathan et al. (2011)[53] Mirko et (2005)[54]. al.. Rauter et (2016)[41]. al.. Brand et (2008)[39]. al.. Mordinyi et al. (2012) [28]. Noten et (2017)[40]. al.. Keränen et al. (2012)[34]. The paper proposed the integrated approach and continuous process to save time and to increase safety and quality. Improve the description of test scenarios and the design of the systematic test. The author improved maintainability and quality of the overall production process by reduction of the pre-defined test case definitions. The author can find test cases which might not be considered. Extend the test coverage with only use the system’s specification. The effectiveness is verified. The method provided by the author is more intuitive to define transformations. ATL is easily adaptable. The developer is more freedom. The author found that SysML is a valuable instrument supporting the complex systems.. The use of COTS cost savings. And the test automation relieves manual testing and debugging tasks, saving the cost too.. 29. The author needs to concentrate more on the nature of the test model and the purpose of the test cases in the test framework. Limited use scenario, the method only for automotive embedded software. The author need to simplify the test procedure generation in the dynamic SUT models.. The author needs more in-depth analysis.. The Eclipse environment is hard to install. The training effort is high. It will cost a lot of time to deep learn ATL. Because of the complexity of SysML, it is difficult to devise a suitable methodology supported by an optimally aligned tool. It needs a lot of effort. The Qtronic online testing algorithm has some performance problems. The experimental evaluation indicated that online MBT with Qtronic is not a silver bullet for all potential faults. The issue of modeling complex embedded systems is solved for successful use of MBT in HIL environments in the future..

(40) Chapter 4. Systematic Mapping Study. Iyenghar et al. (2010)[35]. The memory overhead for online testing is smaller than the traditional approach. And it is possible to deploy the model-based testing (online) in embedded targets.. Estevez et (2012)[37]. al.. Defect errors at early stages. Reduce the cost and development time. The use of model improve the design of the system.. Vain et (2017)[57]. al.. Increase the trust of the test result.. You, D. et al. (2015)[58]. Generate tests are offline with existing tools to reduce online cost. The method is effective to defect 93% seeded faults. And it improves the observability.. Guan, J. et al. (2015)[56]. 4.6.3. 30. The author need to extend this method to a real application scenario The use of MathML for specifying semantic which need to be expressed by mathematical editor make it hard to create related constraints. The author need to integrate these techniques to extend the test verification. The authors’ method takes a fixed mean delay of each components. When deriving test messages the author did not consider the loop transitions.. Analysis for RQ 1.1. In order to study the implementation process of the methods, we need to understand the context of use of the methods at first. Then, we can understand the process deeply. 4.6.3.1. Contexts. Based on the data collection, we find the methods used in different contexts for different papers. While we ordered each context divided into 5 types of systems with the number of occurrences as shown in Table 4.6.

(41) Chapter 4. Systematic Mapping Study. 31. Table 4.6: Data Extraction Form of contexts Contexts Production system Embedded system Critical system Communication system Cyber-physical systems. Reference [25] [5] [27] [30] [40] [41] [43] [31] [28] [29] [36] [52] [55] [51] [37] [53] [15] [50] [59] [38] [39] [42] [32] [44] [45] [46] [34] [35] [54] [56] [58] [47] [60] [57] [48]. Occurrences. [26] [49]. 2. [33]. 1. 20 13 2. The 5 types of systems Most of the methods are processed in the embedded system and production system. Embedded system is embedded into a mechanical or electrical system and has a special function. People usually perform various operations using high-efficiency control equipment combined with embedded systems. A production system is an information system that performs daily business. It includes production data, production data processing systems and production networks. Critical system concludes 4 typessafety critical, mission critical, business critical and security critical. And it must be highly reliable and maintain it because it will not generate excessive costs. In the communication system, it is a collection of individual communications networks, transmission systems, relay stations, tributary stations, and data terminal equipment (DTE), and they can interconnection and interoperation together. Cyber-Physical Systems are systems that tightly connected with the real world around and its on-going processes, and at the same time through internet providing and using, data-accessing and data-processing services [62]. To analyze RQ1.1 of how were these MBT methods are implemented and the contexts of using are the highlight. As shown in figure 4.5, about 53% of the papers are studied in the production system. And about 34% of the papers are in the.

(42) Chapter 4. Systematic Mapping Study. 32. embedded systems. Overall, according to the data collection, we can know that MBT is widely being used in dierent systems.. Figure 4.5: Context that using MBT method 4.6.3.2. Process. For the process of the design step of the model-based testing., all of the studies conclude 2 steps: 1. Develop the model. 2. Test cases generation. It’s the most common steps of the process of the MBT method used in the plant system. These papers use different techniques in the testing activity that make each process for the MBT is unique. For example, Borja et al. [51] use BIP (Behavior, Interactions, and Priorities) framework to execute the PLC program. So the process in each paper is unique and not possible to synthesize them. Different contexts with the related process can provide a reference for practitioners.. 4.6.4. Analysis for RQ 1.2. According to the table 4.6, for advantage, Some methods combined with MBT to help the tester to detect the potential problem that may occur. Like [5] [26] [63][45]. some method proposes the new technology to improve verify more quickly[64]. Some method support test generation and requirement mapping to help the tester to verify the system [65][66]. Overall, MBT can combine different methods or techniques in a specific context to improve the testing process..

(43) Chapter 4. Systematic Mapping Study. 33. For Drawbacks, when import a new method or technique to MBT, there may some other needed preparatory work need to do. Like [5][26][64]. And other drawbacks are manually to complete test case.[45] In the specific context, related drawbacks of MBT method have to be thought about that should not influence the test efficiency too much. In table 4.7, We made a summary of the main advantages and drawbacks that contained in these MBT methods based on the data we extracted in table 4.6. One MBT method contains one or several of them. Table 4.7: Summarized advantages and Drawbacks Advantages Improve the description of test scenarios Save time and cost Increase safety and quality. Drawbacks Effort and cost is high Steps are complex Using current scenarios are limited. Improve test coverage and the trust of test result. Real machine are not considered. Improve robustness. Difficult translate abstract case into concrete. Let test more logic and feasible by using mathematical theory or method. Mathematics are hard to understand and have some problem. 4.6.5. Potential MBT for Case study. According to the context, we found three are two papers [25][5]that highly similar to the case company. Guignard et al. [25] proposed a conformance relations testing by using a Mealy machine which can be useful to establish model-based testing methods of PLC. It can let tester determine the smallest duration of each test step about the number of PLC cycles. It reduces the time and effort for testing and fulfill the testing goal and give reliable verdicts. Ladiges et al. [5]proposed an approach which is assessed on a demonstration plant. It combined MBT with model learning method. The MBT is first to use SFR language to depict the test model based on requirements. And then generate test cases according to the model. The state of sensor and components are represented by value as input and expected output. When compare the Melay machine mentioned in [25] and Sequential Function.

(44) Chapter 4. Systematic Mapping Study. 34. Charts (SFR) in [5], we found melay machine has a drawbacks that it hard to represent the status of sensors and actuators while SFR can easily to cover this. Finally, we choose SFR as the potential method that has a try in the case study.. 4.7. Validity Threats. We analysis the validity threats of the SMS in this section, and identifying the validity threats which will impact the research result. Tthe validity threats in three aspects: Theoretical Validity, Interpretive Validity, Generalizability Validity [67] and we also provide our mitigation strategies.. 4.7.1. Theoretical Validity. Some method based testing methods in the papers are not in a real machine so the research result is not reliable is a theoretical validity threat of the research. We do not have enough time to ensure all of the results is correct. Because the process of the paper may be only used in the specific context. What’s more, the author may not present controversial and negative views of their process of the method. To reduce the threat, we analyzed and summarized the process of these modelbased testing methods, extracted information from them, and also discussed their advantages and disadvantages.. 4.7.2. Interpretive Validity. The fault analyzes method and the misunderstanding of the context may cause the fault research result is the interpretive validity. Because we do not have the appropriate expertise, we don’t understand all of the contexts in the papers. In order to avoid the misunderstanding of the contexts and choose an Inappropriate method for our context. We separated the contexts into 5 systems, and then analyze the advantages and drawbacks of the method. So we mitigate this threat and find the appropriate method for our context.. 4.7.3. Generalizability Validity. Because the MBT approach used in the different environment, our context used the PAC instead of PLC used in the papers. So some methods of MBT cannot be.

(45) Chapter 4. Systematic Mapping Study. 35. promoted in our environment. To mitigate the generalizability threat, we found many relevant studies by snowballing and searching. These relevant papers used the model-based testing in the different environment. In this way, we are more likely to find proper MBT methods that suit our environment..

(46) Chapter 5. Case study and Results. This chapter we describe our case study implement process. The case study design we following the guideline that introduced by Per Runeson & Martin Höst [68] We used four steps to conduct case study as shown in figure 5.1. The section number also contained in the figure for the specific step.. Figure 5.1: Process of case study. 5.1 5.1.1. Case study design Objectives. The case study is mainly to answer the question of RQ2 and RQ3. The research questions are: RQ2: What are the main challenges of manually testing PAC based controlling systems in industrial plants? RQ3: Which model-based testing method could be used to test a PAC based controlling system? To what extent can it address the challenges encountered in the case company?. 5.1.2. Case description. As figure 5.2 shows: the case we are studying is about testing in industrial automation area, specifically we conduct the case study in a printing house and focus on the testing of PAC based system.. 36.

(47) Chapter 5. Case study and Results. 37. The case study was implemented in a printing house in Switzerland, it is a brand new industrial plant that using second-hand devices. And these devices need a control system to control them. We are responsible to use PAC based system to control these machines. It is significantly important and there is a lot of manual testing work to make they running well. Due to the time of the project, we had around 4 months to implement our case study.. Figure 5.2: Case description. 5.2. Preparation for data collection. To answer the two research questions, there are some data that should be planned to collect. As the table 5.1 shows, For RQ2 there are three kinds of data that should be collected or prepared: Testing activities and process, Manual testing content, Resources and eort needed, Challenges. Based on the data collected from RQ1 and RQ2, we are to answer the RQ3..

(48) Chapter 5. Case study and Results. 38. Table 5.1: Data preparation Research Questions What are the main challenges of manually RQ2 testing PAC based controlling systems in industrial plants?. 5.2.1. Data1. Data2. Data3. Testing activities and process. Manual testing content. Resources and effort needed. How to collect data. We implemented the case study by full participation in the project to make some devices controlled by PAC based system. And we were as the manual tester to implement the all manual testing. While implementing the case study, we use a camera to record the context and use the note to record data of testing activities and resources and effort. Specifically, the context data is about hardware and software operating environment. Testing activities are about testing content which should be implemented in the case company. The effort is about how much time for the tester to implement the related testing activity. The resource is about how many copies of paper and number of the label that should be used to support the related testing activity. These data is first-hand data from us. We mainly focus on the testing parts, the other parts contained in the project like cable connection, PCs and Server install, software install that were ignored to seek data.. 5.2.2. Data analysis guidelines. Based on the collected data from the case study, first, we analyze the data related to manual testing to conclude the challenges of it. And then we analyze the related data to make a comparison between manual testing and the new testing method. In this way, we can know to what extent we mitigate the challenge of manual testing.. 5.2.3. Collection techniques and observations. Based on Lethbridge et al. (2005) [69], there are three levels of data collection techniques. The first level is that researchers collect data in real time by directing participate in the project. The second level is the researcher indirectly collect raw data without joining in the project, like observe videos etc. The third level is a.

(49) Chapter 5. Case study and Results. 39. research analysis current document like report or database etc to collect data. We are using the first-degree collection technique. We did research and at the same time, we are also as manual tester stuff to implement testing work. Our perspective is directly on the project. We can collect data that we needed and we can notice more detail information which may be ignored.. 5.3 5.3.1. Data collection and Analysis Data collection for RQ2. During the case study, we implement the research and doing manual testing of PAC based controlling system at the same time. Many data were collected, we first record the testing context, and then we detail to record the testing activities and resources and effort needed for manual testing. Appendix C shows the data.. 5.3.2. Testing context overview. In printing house, there are two aspects that need to be tested. One is signal testing and the other is functional testing. As figure 5.3 shows, physical machines are directly controlled by PAC which makes related machine response right action at the right time.. Figure 5.3: Machine controlled by PAC 5.3.2.1. Signal testing. Figure 5.4 shows the very basic and important step is to connect cables to specific ports. In this way, PAC can receive a signal from the sensor and send control commands to the machine. Figure 5.5 and figure 5.6 show the PAC’s input and the output module..

(50) Chapter 5. Case study and Results. Figure 5.4: PAC connect to specific signal. Figure 5.5: PAC. Figure 5.6: Input and output modules. 40.

(51) Chapter 5. Case study and Results. 41. The signal testing part of the system that needs to be tested is shown in figure 5.7. It focuses on the connection between hardware and PAC.. Figure 5.7: Signal testing part of the system Figure 5.8 shows the two main logic of PAC that need to be tested in signal testing. One is to detect a sensor signal, it means when the sensor has detected a subject, the indicator in PAC should show right. The other logic is control logic, it means when PAC send a command to specific hardware components, they can respond right action.. Figure 5.8: Testing logic of software for signal testing Manual testing process As figure 5.9 shows, it is the manual testing process for signal testing. The tester needs import code to PAC and manually to observe the indicator light of PAC..

(52) Chapter 5. Case study and Results. 42. Figure 5.9: Manual testing process 5.3.2.2. Functional testing. For functional testing, we took STB & DBH parts for example, and these parts are typical parts that controlled by PAC. They are representative of the other parts of the printing plant. STB part contains machines and the main function is to transfer papers from A to E (figure 5.10). DBH part contains some machines that the main function of it is to press label (which printed our by the printer that shows the address and contact information of subscriber etc.) by a mechanical arm to the specific bundle of papers. As the figure 5.10 shows, papers have been produced and send to stacker A (Making bundle from per copy) and then bundles put out from stacker and transfer to conveyor B. Figure 5.11 shows the real machine of.

(53) Chapter 5. Case study and Results. 43. STB & DBH.. Figure 5.10: Simulation chart of STB & DBH. Figure 5.11: Real machine of STB & DBH Figure 5.12 shows the functional testing part of the system. It related to all components of the system..

(54) Chapter 5. Case study and Results. 44. Figure 5.12: Functional testing part of the system Figure 5.13 shows the main functional logic of software that should be tested for functional testing. 1.Queuing algorithm: When bundles transferring on the conveyor if the front bundle stop (pressing label or packing) the later bundles needed a stop to avoid the crash between bundles. 2. Control printer to print right label and press on to the right bundle(Label id = bundle id), when the bundle is removed or put on to conveyor manual should not influence the function.. Figure 5.13: Testing logic of software for functional testing Manual testing process Figure 5.14 shows the process of manual testing for functional testing..

(55) Chapter 5. Case study and Results. 45. Figure 5.14: Manual testing process for function. 5.3.3. Analysis for RQ2. As the collected data of manual testing data shows, there are many testing activities in a project for different parts. One or two testers need to implement all of them in a project and it involves many details. We conclude the challenges of manual testings bellow: Software testing challenge: 1. Many testings are no specific steps, mainly depends on the tester’s experience and no test case generation tool so the software testing coverage cannot be ensured, tester usually implements many repeatable and lots of testing to detect the problems. 2. Server program needs to offer information (like bundle id and label id) to PAC and Printer. This information is important for PAC and Printer to make.

(56) Chapter 5. Case study and Results. 46. right action. Software problem of the server is usually detected by observing the real machine response. Many observing works make the testing low effectiveness. Sometimes it needs at least 2 testers to test system together, one is simulation some activities of real machine (Like put papers on STB), another one is to observe the phenomenon. 3. Normally core logic of the program of PAC and server that can be reused in different projects. Random testing steps and operation sequences are hard to remember, and tester record data mostly by using mind or paper, very less useful testing data saved for later software testing or other function. And it easy to ignore some situations. 4. As the testing combine hardware. Sometimes the problems maybe not the logic of the code. It is also about the connection of the cable, devices may be damaged, the configuration of hardware, voltage for devices, Physical properties. Which sometimes tester will think it is the logic problem and change code. Which let it much harder to analyze the software problem. It is not easy for people to realize these problems. Testing process challenge: 1. If some problems happen, the code needs to be modified timely and imported to PAC and test, again and again, the process is tedious. 2. Sometimes it needs testing control the machine and sees the signal light at the same time, but the button may far away from the PAC, and need someone else to help. 3. For security reason, sometimes two machines cannot be running together, but people sometimes will forget this and give an output signal to make both machines running, it easy to make machine damaged. 4. Sometimes one testing activity wastes much paper. Some paper material is expensive. It is a waste resource and money.. 5.3.4. Model-based Testing of Software. This section we describe the model-based testing for software error detection. Based on the analysis of the mapping study, we have found a potential MBT method from [11] that could be used in the case company. Now we first use two examples (STB & DBH parts), which are highly representative of PAC based controlling system in the case company to describe the model-based testing method and how it implemented. And then discuss to what extent the method address.

(57) Chapter 5. Case study and Results. 47. the challenge of manual testing and limitations. Lastly based on the method, we proposed a combined method of model-based testing and simulation testing, which is a great possibility to address the challenges of manual testing. STB Example: As shown in figure 5.15, STB contain 5 sensors(S1,...,S5), 5 Holders (H1,...,H5), 1 conveyor (A0). The main function of STB is to transfer bundles. Bundles are pushed out from the stacker and transferred on the conveyor(A0). Sensors(S1,..., S5) are used to detect if there is a bundle at the specific position. To avoid confliction between bundles, holders(H1,..., H5) are used to lift up the bundle if there is a bundle in the previous position.. Figure 5.15: Sample STB DBH Example: As shown in figure 5.16, DBH contains 2 sensors(S6, S7), one arm (A1), one printer (A3), one roller (A2). The main function of DBH is to print and press label(L1) which show the address and copy number of bundles on the specific bundle(B1). When bundle put out from stacker and detected by sensor 1(S1), the printer will print correspond label and it is transferred by roller(A2) to position 2(S7), when the bundle transfer to the position4 (S4), the arm(A1) will press the label on bundle, as different bundle contain different number of copies, the Labe Id and Bundle Id must match correctly. Normally there is not only one bundle on the conveyor, so there are often two labels on position 1 and position 2 respectively, so printer(A3) could not print label if two positions are full..

No results found