Teknik och samhälle
Datavetenskap
Examensarbete
15 högskolepoäng, grundnivå
Comparative study of the testing tools Ghost Inspector
and Selenium
Jämförande studie av testverktygen Ghost Inspector och
Selenium
Dennis Kristensson
Examen: Kandidatexamen 180 hp Handledare: Magnus Krampell Huvudämne: Datavetenskap Andrabedömare: Ivan Kruzela Program: Systemutvecklare
Sammanfattning
Studien undersöker de automatiserade testverktygen: Ghost Inspector och Selenium. En limiterad litteraturstudie och två fallstudier görs, för att undersöka och utvärdera testverktygen. Detta för att kunna tillgodose riktlinjer som är användbara, vid val av ett verktyg för att utföra regressionstestning på webbapplikationer.
IStone använder Ghost Inspector som verktyg vid regressionstest på webb applikationer, eftersom det ger dem en effektiv konfigurering och underhålls-process. Selenium kräver mer ansträngning för konfigurering och underhåll, eftersom verktygen inte är en tjänst. Miljön i Ghost Inspector underlättar för iStone att exekvera test kontinuerligt och förser projektteamet med det senaste testresultatet.
Abstract
The study evaluates the automatic testing tools: Ghost Inspector and Selenium. and focuses on comparing them in terms of the set up and maintenance processes. The study includes a limited literature study and two case studies. The purpose is to evaluate the tools and provide guidelines for choosing a regression testing tool when testing the functionality of web applications.
IStone use Ghost Inspector as regressions testing tool on web applications, since it offers an effective set up and maintenance process. Selenium requires more effort to set up and maintain, since Selenium is not a service. The Ghost Inspector environment comforts iStone with the ability of executing test continuously and supplies the latest test results to the project team.
Keywords: automated testing, agile, case study, continuous integration, functional testing, Ghost Inspector, Jenkins, maintainability, regression testing, Selenium, test management.
Acknowledgment
I would like to thank iStone for the opportunity to write a study within the company. They have supported me with material and ideas and have encouraged me to make this study. It has been an interesting experience and I have gained much knowledge within short time. Many thanks to my supervisor Magnus Krampell for qualified guidance, valuable ideas and concrete feedback.
Contents
Sammanfattning ... ii Abstract ... iv Acknowledgment ... vi Glossary ... x 1 Introduction ... 1 1.1 Background ... 1 1.1.1 Testing ... 11.1.2 The company IStone ... 1
1.2 Purpose of Research ... 1
1.3 Question formulation ... 2
1.4 Scope and limitations ... 2
2 Theoretical background ... 3
2.1 Software Testing ... 3
2.2 Black box Testing ... 3
2.3 Regression Testing ... 3 2.4 Ghost Inspector ... 3 2.5 Selenium... 3 2.6 Jenkins ... 4 2.7 JUnit ... 4 3 Related work ... 5
3.1 Literature search criteria ... 5
3.2 Selected papers ... 6
3.2.1 Holmes et al. - Automating functional tests using Selenium ... 6
3.2.2 N. Seth et al. – ACI (automated Continuous Integration) using Jenkins: Key for successful embedded Software development ... 6
3.2.3 S. Bauersfeld et al. - A Java Library for Fully Automated GUI Robustness Testing ... 7
3.2.4 T. Parveen et al. - A Case Study in Test Management ... 7
3.2.5 H. Kaur et al. - Comparative study of automated testing tools: Selenium, Quick Test Professional and TestComplete ... 8
3.2.6 A. Bruns et al. - Web Application Tests with Selenium ... 9
3.2.7 R. A. Razak et al. - Agile testing with Selenium ... 9
4 Method ... 10
4.1 Method description ... 10
4.2 Fact-based case study ... 10
4.3 Practice-based case study ... 11
4.3.1 Set up Ghost Inspector and Selenium ... 11
4.3.2 Set up Jenkins ... 11
4.3.3 Recording tests ... 12
4.3.4 Replay tests ... 12
5.1 Fact-based Case Study ... 13 5.1.1 Overview ... 13 5.1.2 Recording plugin ... 14 5.1.3 Variables ... 14 5.1.4 Application support ... 15 5.1.5 Costs ... 15 5.2 Practice-Case Study... 16
5.2.1 Set up the environment ... 16
5.2.2 Jenkins ... 17 5.2.3 Recording tests ... 17 5.2.4 Replay 5 tests ... 17 5.2.5 Report ... 19 6 Analysis ... 21 7 Discussion ... 23 7.1 Method discussion ... 23 7.2 Discussion... 24
8 Conclusions and future work ... 25
8.1 Conclusions ... 25
8.2 Future work ... 25
References ... 26
Appendix ... 30
Glossary
Agile: An iterative software development methodology.
API: Application Programming Interface is used by applications for communication.
Cloud-service: Shared computer resources and data on demand.
Continuous integration: A development practice when developers integrates code into shared repositories daily.
CSS: Cascading style sheets is a language for styling HTML documents. CSV: Comma-separated values stored in lines as data records.
IDE: Integrated software environment is a software suite with tools for writing and testing code.
GUI: Graphical user interface is used for interacting with e.g. a computer or device.
HTML: Hypertext Markup Language is a language for building and structuring websites.
Silver bullet: A simple, perfect solution to a software problem. Slack: A message service mainly used for enterprise projects. Test suite: A container with a set of test cases for testing software.
XPath: XML Path Language is used to navigate through elements and attributes in an XML document.
Variable: A variable store data and can be used for different purposes, e.g. sign up credentials or check with an assertion that a value corresponds to another value.
1 Introduction
1.1 Background
1.1.1 Testing
Software testing is useful when assessing the quality of a software product. Testing provides information to the customer and shows if the product is built to the given requirements [1]. Software testing is in many organizations not prioritized. According to [2] companies should have a testing strategy for a more effective software testing.
Automated testing can be applied in many of the stages and environments in software development. It can be used for testing platforms, layers, interfaces, etc. A common use for automated testing is regression testing. Regression testing, reduce test time and ensures stable software versions in projects. Regression testing is effective when the development is iterative. The testing process includes organizing test cases, test artifacts, and test results. The process should be followed systematically, for successful results in the software project [3].
Agile development includes black box testing and regression testing, by executing automated scripts with tools e.g. Selenium or Ghost Inspector. Agile testing creates a time-efficient testing process, by incremental and iterative test new delivered functions [4].
Two tools for doing automatic regression testing are Ghost Inspector and Selenium. Ghost Inspector is delivered as a cloud-service with predefined functions for test automation [5]. Selenium is an open source testing tool and may be one of the most used regression testing tools. Test automation has benefits which support frequent testing with feedback, documentation and test reports. The process of doing, it automatically is appropriate if the graphical user interface not change continuously. This since more time needs to be spent on modifying tests. Test automation is suitable when working with Agile methodologies [6].
1.1.2 The company IStone
IStone implements and maintains comprehensive IT solutions. IStone has offices in 12 countries with 607 employees in total. The company provides IT solutions to areas such as business intelligence, e-commerce, marketing, etc. [7]. IStone wants an evaluation of the automatic software-testing tools: Ghost Inspector and Selenium.
1.2 Purpose of Research
The study provides IStone and others with results in relation to the set up and maintenance processes of the two evaluated tools. The purpose of this study is to evaluate the tools Ghost Inspector and Selenium. The results and conclusions are derived from a limited literature study and two case studies: fact-based and practice-based.
1.3 Question formulation
The tools Ghost Inspector and Selenium are evaluated in this study focusing on the set up and maintenance processes. The tools are both used for regression testing web applications. Ghost Inspector is a service and Selenium is an open source tool.
RQ: In the context of the regression testing tools: Ghost Inspector and Selenium, how do the tools differ regarding the set up and maintenance processes?
1.4 Scope and limitations
Ghost Inspector is chosen since it is currently used at iStone for regression testing. Selenium is chosen since it is one of the most used open source tool for testing [6]. Only the set up and maintenance process of the tools are evaluated.
2 Theoretical background
2.1 Software Testing
Software testing is used to assess quality of the software and to ensure that the developed features meet the given requirements. The software testing process consists of a collection of methods, e.g. black box and regression testing [2,8].
2.2 Black box Testing
Black box testing is a method to verify if the program meets the requirements. This is done without accessing the code. The most suitable persons to run black box tests are non-programmers. A programmer usually test “what they programmed it to do”, the non-programmer test realistic scenarios and is more inclined to find faults [8].
2.3 Regression Testing
Regression testing provides a stable and reliable software development process. Testers run tests on previous and updated code versions, to ensure, that new or modified features, has not affected features that previously worked. New code is included into the code base, once it has been verified and regression tested. The testing process helps to identify faults at an early stage of the development and is most effective when the testing is done automatically [9,10].
2.4 Ghost Inspector
Ghost Inspector was introduced 2014 by Justin Kleem as a regression testing tool. Ghost Inspector relieves developers from installing software, finding packages and provides a service with everything configured [11]. The tool is useful for creating web application tests through a methodical black-box way of working. Tests are built with the Ghost Inspector plugin, installed in the browser Google Chrome. Recorded tests are saved in an account at the Ghost Inspector website. After a test is executed, a video and screenshot are saved on the website to get a clear overview of the result [12,13].
2.5 Selenium
Selenium was introduced 2004 by Jason Huggins working at ThoughtWorks. Jason Huggins developed Selenium as a JavaScript library for doing test automation thought multiple browsers. Selenium is a framework that includes several tools, used for creating and executing automated test scripts, for regression testing of web sites. Selenium is an open source tool, that can be customized with plugins and support multiple browsers for executing tests on. The Selenium IDE is used for recording and executing tests via the browser Firefox. Recorded tests can be exported in several programming languages and executed with Selenium Remote Control and Selenium Standalone server [4,6].
2.6 Jenkins
Jenkins is a tool used for effective continuous integration, by automating the build and deploy process in a software project. It is configured via a web browser interface. Jenkins is an open source tool and can be customized with plugins. Jobs can be scheduled and configured in Jenkins for executing tests automatically on other services. Jenkins supports sending build status notifications and maintain reports, useful for the development team [14,15].
2.7 JUnit
JUnit is an open source framework from 2002 written by Erich Gamma and Kent Beck. JUnit is used to write and run unit tests in Java for testing individual parts of a code. It uses assertions to match tests against an expected result [16]. JUnit can be used for running Selenium tests and for creating test reports [6].
3 Related work
3.1 Literature search criteria
The databases ACM, Google Scholar and IEEE provides the study with computer science papers. The databases are used for searching on related keywords regarding automated testing. Papers and articles from Google are selected to complement missing materials. Searches in the presented databases are logged in the appendix for the chosen articles, to get traceability in the study. All the papers found in Chapter 3.2 are used in the study.
The specific key phrases are selected by reading papers about regression testing. The search strategy is made by first use regression testing and automated testing as keywords. If an interesting keyword appears in a paper, it is added to the next search. When a search is made, the result in the first page is evaluated. Every keyword is searched in all the presented databases. The peer-reviewed reports are prioritized.
The following keywords are used in the database searches: • Automated testing • Agile • Case study • Continuous integration • Functional testing • Ghost Inspector • Jenkins • Maintainability • Regression testing • Selenium • Test management
The following questions are inspired from [17] and considered when an interesting paper is found:
• Is it related to any other papers? • Is it valid and well written?
• What is the purpose of the paper? • What type of paper is this?
3.2 Selected papers
These papers are selected to present related work to software testing by following the method in Chapter 3.1.
3.2.1
Holmes et al. - Automating functional tests using Selenium
In this paper [18], the authors refer to Selenium as being close to the silver bullet. According to the authors, it is not difficult to write tests in Selenium. To be able to record tests a Selenium installation must be made. To keep tests organized, each test is grouped into suites, containing other tests with the same story. The authors are striving for an independent, self-contained testing solution. By including new tests into completed tests and reducing duplicated code, flexibility and maintainability are increased in the testing process.The following process is used in the study: • Before development, tests are written.
• When development is done, all tests are executed and discovered bugs are corrected.
• Then code is reduced for better maintainability. Conclusions of the study:
• Store all tests in one suite for easier managing and reducing test time. • If possible, write the test first and then develop, else write the test directly
after the new code.
• Look for unique identification of elements, by using the Selenium IDE • Exclude XPath expressions, since it increases the test time.
3.2.2
N. Seth et al. – ACI (automated Continuous Integration) using
Jenkins: Key for successful embedded Software development
In this paper [14], the authors propose a way of running a daily test execution when building software using Jenkins. The study is built on common project scenarios, with teams spread over different countries. For building and testing the environment Android Lollipop is used. When a build is successfully created, the results are stored and transferred for flashing the Android device. When the device is ready for testing, it is deployed to the customer. The framework JUnit runs tests automatically and helps to discover possible faults or issues before a release. This process is done two or three times daily. After a build is completed, the results of the test run are reported from Jenkins by e-mail to all team members.
Conclusions of the study:
3.2.3
S. Bauersfeld et al. - A Java Library for Fully Automated GUI
Robustness Testing
In this paper [19], the authors present a Java Library for generating automated GUI tests. The GUI (Graphical user interface) is a central link, connecting a user with the software for making interacting possible on for example a website. Automated tests are often written manually and need much effort to develop. For creating automated tests on a GUI, input chains are recorded for re-creating user interaction with the system. It is useful if the GUI not shifting constantly.
The authors attempt to find a way of building auto generated test cases during a test run. They produced a Java program called GUITest. It generates tests for advanced software applications without human interaction. The library does not require any input strings, manually generated models or patterns. This is an advantage that differs from other software libraries. The GUITest process starts with gathering information about the current state. It scans for positions and sizes of elements. The gathered information is used to build a hierarchical tree, containing all elements. The GUITest use the tree to detect default actions like simple clicks, keystrokes, etc.
To test GUITest in context, Microsoft Word is used together with test configuration files. The files are restored after each test to ensure same conditions and tests is done by generating random interactions. If the program crash, the time and interaction are logged.
3.2.4
T. Parveen et al. - A Case Study in Test Management
In this paper [2], the authors believe testing is not enough prioritized in most organizations and require more effort. Companies should have a testing standard for creating an effective management strategy.
The case study is made at a big Software company with offices spread over the world. At the company employees works with an agile development and executes tests after changes in the code. Communication between employees is made via emails containing the test results. The authors believe that this is not a reliable way to handle the result. They choose Test director which is a web-based testing for managing test data and share results within an organization. Test director makes it easier for different teams to get an overview of the performance of each test.
The research is focused on the testing processes within test management. The authors write about test experience and how to organize requirements, test cases and results within companies. The result shows how important software testing is for creating stable software.
3.2.5
H. Kaur et al. - Comparative study of automated testing tools:
Selenium, Quick Test Professional and TestComplete
In this paper [1], a comparison between different tools for testing web application is made. The purpose of the study is to investigate the tools usability. The following criteria are examined:
• Capability of generation of scripts • Costs
• Data driven testing • Easy to learn • Execution speed • Playback of the scripts • Recording Efficiency • Reusability
• Test results reports
The study includes three automated testing tools with the same core functionality. The authors create a list with tools limited to, requirements and chose the top three most suitable tools for the study. The tools are Selenium IDE, Quick Test Professional and TestComplete. A table with features of the chosen tools is made, to discover differences between the tools. Features for comparing the tools are:
• Application support • Database applications • Licensing costs
• Object oriented language support and scalability • Platform dependency
• Programming skills • Report generation
• Support for operating system/platforms • Usage
The conclusion from the study results in the fact, that each tool is useful within different circumstances. Choosing an appropriate tool should depend on the application, the budget and the way of working in terms of efficiency. The Quick Test Professional tool serve same functionality as Test Complete and is more useful for critical application testing. If Test Complete serves required demands, there is no purpose of using the Quick Test Professional since it is more expensive. Selenium is recommended as a cost-efficient testing tool.
3.2.6
A. Bruns et al. - Web Application Tests with Selenium
In this paper [20], the authors present how to use Selenium. Testing is a comprehensive process which can be endless, especially if it is done manually. The importance of doing it automatically has increased and tools like Selenium has appeared for acceptance test application. The acceptance test focuses on effects showed directly to the user. It tests if the functions work as supposed after added functionality. Acceptance test is time-efficient until the software is changed and tests need to be recorded again. The authors believe in crafting code after tests, for more flexibility.
Selenium is a framework for testing web applications. The Selenium core runs a JavaScript application. Selenium has created an own language called Selenece, based on a grouped HTML table consisting of commands, elements and values. The language Selenece may be unnecessary to learn, since Selenium supports converting tests through the Selenium Remote Control to C#, Java, Pearl, PHP, Python and Ruby.
According to the authors, Selenium contributes to coming closer the real user experience. If the developers expect changes, a good idea is to assign a unique ID to an element when tests are written. XPath expressions or CSS selects can be used if IDs not are preferable under development for identifying elements.
3.2.7
R. A. Razak et al. - Agile testing with Selenium
In this paper [4], the authors present Selenium and how it automates the testing process. It has high flexibility and allows operations for locating elements and comparing results towards the behavior of the application. A table is presented showing compatible frameworks using Selenium, e.g. Junit, TestNG, Ruby.
A benefit with Selenium is that it can test multiple browser platforms. The Selenium IDE is a tool for building test scripts in Firefox. It is easy to use for recording tests, containing user actions, which can be exported to different programming languages for execution. To use Selenium the authors implements the framework with Python. Jenkins is integrated into the Selenium testing process for creating an effective continuous integration process. The authors describe the process step by step how to set up the environment with Jenkins. Jenkins executes tests and reports the result. The conclusion is that Selenium has positive impact when testing automatically.
4 Method
4.1 Method description
This chapter describes the method used to answer the research question. A limited literature study is made, together with two case studies. One case study examines facts, another the practical process. The boundaries in the cases are the set up and maintenance processes, since those are most important for iStone. With inspiration from [1], the tools are compared and presented in tables.
4.2 Fact-based case study
To understand the concept automated testing, scientific papers are studied and reviewed. To be able to answer the research question, a limited literature study is made. The databases ACM, Google Scholar and IEEE provides the study with information. Papers from Google are selected to complement missing materials. Facts are collected in the literature study and used in the fact-based comparison, as the authors of the paper [1] did, when comparing Selenium and two other tools in a table. According to the authors of [20], the best source for finding information is in the documentation section on the tools website.
Following criteria cover the most important functionalities for examination, some of them are also used in the paper [1].
• Application support • Cost • Display resolution • Environment • Programming skills • Recording plugin • Report generation • Variable
4.3 Practice-based case study
The practice-based study includes the process of set up and maintain the environment, inspired from the paper [4] where the authors describe the process of set up and testing with Selenium.
In the case study [21], the authors based their cases on own experience when studying maintainability. This is made in this practice-based case study as well. The author of this study has the main responsibility for using Ghost Inspector software at iStone and is acting as an objective observer and performer in the study. Findings regarding setting up the environments, recording tests, replay the tests and creating test reports are discussed in the discussion Chapter 7.
Criteria used for comparing and reviewing the tools: • Set up process:
o Ghost Inspector o Selenium
o Jenkins • Maintenance:
o Record five tests
o Replay the recorded tests o Create test reports
4.3.1 Set up Ghost Inspector and Selenium
Table 1Ghost Inspector Selenium
To set up Ghost Inspector, the guide supplied on the Ghost Inspectors website [5] is followed.
A registration is made on the Ghost Inspectors website. The Ghost Inspector plugin is installed in a Chrome browser.
To set up Selenium a new Maven Java project is created in the editor Intellij IDEA, which is a tool for writing code. The tests are recorded with the Selenium IDE and exported into a Java file in JUnit format.
Selenium Remote Control 2.53.1 and Selenium Standalone Server 2.53.1 are used for executing tests.
4.3.2 Set up Jenkins
Ghost Inspector and Selenium are configured with Jenkins, which enables tests to be executed remotely. Jenkins is configured with a plugin developed to be used with Selenium [22]. Ghost Inspector is connected through its API. Jenkins sends a request to Ghost Inspector which execute all tests in a suite and returns the result [23].
4.3.3 Recording tests
Five test cases are used in the test runs selected from a project on iStone. Each test differs as much as possible to reduce duplicates and redundancy. These test cases are inspired from the project on iStone and can be found in the Appendix. The tools shall record and replay tests that can be found in the documentation. The Ghost Inspector plugin is used for recording tests in the browser Google Chrome. When tests are recorded, and exported to JUnit 4 (Java) format, the Selenium IDE is used with the browser Firefox.
To organize tests, inspiration is taken from the paper [18], where the authors carefully examining each recorded step to reduce duplicated code. In the study tests are grouped into suites. The studies [4,14] presents Jenkins for managing the test environments. Jenkins is configured to execute Ghost Inspector and Selenium tests for getting control over the environments.
4.3.4 Replay tests
When all tests are recorded, they are replayed for examination. Table 2
Ghost Inspector Selenium
Ghost Inspector tests are executed directly in the Ghost Inspector environment.
Selenium server and Selenium Remote are used for executing Selenium tests.
4.3.5 Create test reports
JUnit is used for creating reports from test runs in Selenium. It outputs a file in HTML format [16]. In Ghost Inspector are test results shown in each test suite.
5 Result
This chapter presents two case studies according to the method in Chapter 4. Presenting the tools next to each other make it easier to see the differences and investigate them individually.
5.1 Fact-based Case Study
5.1.1 Overview
Table 3
Features Ghost Inspector Selenium
Application support The Ghost Inspector plugin can only be installed in the browser, Google Chrome [5].
The Selenium IDE plugin can be installed in the browser Firefox [27].
The Selenium server and Selenium remote control can run on PC/MAC/UNIX [1].
Can be customized with open source plugins freely after requirements [6]. Display resolution Has support for display
resolutions, same as many phones, tablets
and computers are using [26].
Display resolutions can be set in the Selenium code. The Webdriver.Window Interface is used for setting a resolution in Java [29]. Environment Cloud-based service
accessible from
www.Ghostinspector.com [5].
Can run in a local and in a cloud environment with e.g. Browserstack [28].
Licensing cost Commercial [5]. Open source [1]. Programming skills Require skills in the
languages: HTML, CSS and JavaScript [24].
Require skills in one of the languages: Java, .Net, Pearl, Selense PHP, Phyton and Ruby [1,20].
Require skills in the languages: HTML, CSS and JavaScript [6].
Report Generation Can be done by e.g. Slack [25].
Video support and
screenshot comparing [25].
Can be done with the plugins JUnit Report or TestNG. [6].
5.1.2 Recording plugin
Table 4Ghost Inspector Selenium
The Ghost Inspector plugin for Chrome is used for recording tests. The plugin has basic functions as making recordings and assertions. Recorded tests can be replayed and modified in the Ghost Inspectors web-application. The test editing is done through the Ghost Inspectors visual editor [13]. A cogwheel is spinning during a test replay until the is test done and turns green or red, depending on success or failure.
The Selenium IDE is used with the browser Firefox for recording tests in Selenium [6]. Recorded test includes interactions, made by a user during a browser session. The Selenium IDE recognizes link clicks, values in input fields, options selected in dropdowns, etc. [27].
During a test replay, each step turns green or red, depending on success or failure [6].
5.1.3 Variables
Table 5Ghost Inspector Selenium
Variables can be managed in each test suite. Ghost Inspector provides the user with a visual editor, useful for structure and set up tests with variables supplying input. Ghost Inspector can also be used with variables uploaded as a CSV file [30].
Selenium can manage variables both in Selenium IDE and in Java. Examples of commands for storing variables in Selenium are: storeElementPresent, storeText and storeEval [1].
The user can choose how to set up variables in Selenium [27]. In this project, variables are stored in a Java file and can be accessed from all tests in the suite.
5.1.4 Application support
Table 6Ghost Inspector Selenium
The supported browsers to run tests in with Ghost Inspector are Firefox and PhantomJs.
Ghost Inspector has support with external services, e.g. Jenkins and the message service Slack. Each test suite can be executed from Jenkins. The Slack integration allows Ghost Inspector to post test results to a Slack channel [25].
Selenium supports testing with the browsers: Firefox, Google Chrome, Safari, Edge, PhantomJs, etc.
The Selenium IDE has support for converting recorded tests to code in the following languages: Java, .Net, Pearl, PHP, Phyton and Ruby [4]. Plugins and browser support can be installed. Available plugins on the Selenium website are e.g. screen shots on fail and test results [27].
5.1.5 Costs
Table 7Ghost Inspector Selenium
Ghost Inspector has four subscriptions to choose between: Free, Small, Medium and Large. These provide different numbers of monthly test runs, team members, screen sizes, length of results retention, etc. [26]. All subscriptions contain:
• API access.
• Full video of tests.
• Integration with services, e.g. Jenkins and Slack.
• Simple screenshot comparison.
Selenium is an open source project free to use. It does not require any license and can be used without any limitations [18].
The authors of [1] refer to Selenium as a cost-efficient testing tool.
Table 8. These are the different plans which Ghost Inspector offer (2017-03-14) [26].
Free Small Medium Large
Price monthly 0 USD 71 USD 179 USD 359 USD
Team member(s) 1 team
member 5 team members 20 team members 40 team members
Tests runs monthly
100 runs 10,000 runs 40,000 runs 100,000 runs
Length of result
5.2 Practice-Case Study
5.2.1 Set up the environment
Table 9Ghost Inspector Selenium
To learn Ghost Inspector, read the manual on the website. There is an extensive documentation with examples and videos. An account is required on the Ghost Inspectors website. The guide supplied on [5] with a video was followed.
Ghost Inspector gives possibilities for users without programming skills to be involved in the testing process [31]. 1. A registration is done and the Ghost Inspector plugin is installed into the Chrome browser.
2. After a successful login, a suite is created with a brief description.
Installations:
• Ghost Inspector plugin 3.1.5 • Google Chrome 57.0.2987.133
The learning process of Selenium is complex. Much documentations need to be read. The process starts with examining required software for the setup. The Selenium frameworks and plugins are downloaded an installed. The Selenium version 2.53.1 is used since the latest version 3.X does not have support for running the Selenium remote control directly [20].
Selenium requires skills in program-ming, since it is used in e.g. Java environment.
To set up Selenium a new Java Maven project is created in Intellij. The tests are recorded with Selenium IDE and exported into a Java file in JUnit 4 format. Installations: • Chrome driver 2.29 • Firefox 52.0.2 • Intellij IDEA 2016.3.4 • JUnit framework 4.12 • Selenium framework 3.2.0 • Selenium IDE 2.9.1
• Selenium remote control 2.53.1 • Selenium standalone server 2.53.1
5.2.2 Jenkins
Table 10
Ghost Inspector
Selenium
Jenkins is configured for executing test through the Ghost Inspectors API.
Jenkins is configured for executing test through the Selenium Standalone server.
The Jenkins set up process for executing tests could be made both in Selenium standalone server and through Ghost Inspectors API. Jenkins worked well with both tools and results were received correctly.
5.2.3 Recording tests
The following steps are done to record a test. Table 11
Ghost Inspector
Selenium
1. Enter the website.
2. Open the document containing the test steps.
3. Start recording with the Ghost Inspector plugin and follow the test case given from step two.
4. Make assertions by switching to assertion mode in the plugin to validate the appearance of an element.
5. Finish the test and enter a name for the test.
6. Store it in a specified suite and access it from the Ghost Inspector website.
1. Enter the website.
2. Open the document containing the test steps.
3. Start recording with the Selenium IDE plugin and follow the test case given from step two.
4. Make assertions by right click on the mouse. Alternatives, e.g. assert-ElementPresent and verifyText can be used to validate the appearance of an element or text.
5. Finish the test and enter a name for the test.
6. Store it in a suite and access it from the Selenium IDE.
7.
Exported the test into the Java JUnit format and import it into a Maven project.Ghost Inspector
Selenium
1. Open the suite containing all the recorded tests.
2. Click into the given test and click on the blue button “Run test”
3. The results are presented with a video and screenshot.
An advantage when working with Ghost Inspector is that screenshots and video recordings on test results are saved [25].
A disadvantage with Ghost Inspector is that if a failure occurs during a test run, the Ghost Inspector plugin does not save alternative possibilities for running the test.
When replaying tests in Ghost Inspector:
• Input fields were often not found. This was solved by using a delay on the executed step.
• The visual editor in Ghost Inspector does not have the function of setting test execution speed.
1. Open the Java project containing all the recorded tests.
2. Click into the given test and adjust execution speed.
3. Execute the test.
4. The results are presented in the Intellij IDE console.
An advantage when working with Selenium is that when a failure occurs the recorder save alternative possi-bilities for running the test.
A disadvantage with Selenium is that screenshots and video recordings on test results not are saved.
When replaying tests in Selenium: • Input fields were often not found.
This was solved by using a delay on the executed step.
• The test “Mega Menu”: Could not be created without the use of XPath. The authors of [18], did not recommend XPath as a good option when recording tests, since it increase test time.
5.2.5 Report
Ghost Inspector report
Ghost Inspector creates reports automatically after each test run. Tests can be accessed from the used account on Ghost Inspectors website. The result view is presented in the figure 1 below.
Ghost Inspector
1. Go to the Ghost Inspector website and click into the given suite and execute all tests.
2. After the test execution, the results are presented.
Selenium report
Test reports are created by the framework Junit and are outputted as an HTML file. The code was configured after the section “How do I use Ant to create HTML test reports” from [16].
Selenium
1. Open the Java project in Intellij IDEA containing all recorded tests. 2. Click into the given test and execute it.
3. A HTML file is created in the project source folder with the results.
6 Analysis
Examination of collected information indicates that Selenium puts higher requirements of the user than Ghost Inspector. Knowledge of how to set up and maintain the Selenium in a local environment is needed, while Ghost Inspector is cloud-based. The set up process in Ghost Inspector can easily be made by registering an account and install its plugin.
Basic programming knowledge in HTML, CSS and JavaScript is useful when using both tools, in order to understand the tests. The Selenium IDE has a visual editor, like Ghost Inspector, but to get the most out of Selenium, knowledge in programming is required. The programming part when using Selenium can be complicated when the solution needs to be customized. When deciding which tool to use, the knowledge in software programming of future users should be considered.
The Ghost Inspector documentation [5] is clear with instructions on how to use the tool and differs from the more detailed Selenium documentation. Selenium gives the user possibilities to read about different ways of using it [6]. The Ghost Inspector documentation is simpler, since the only way of using Ghost Inspector is with its visual editor and recording plugin. Selenium can be used in several ways by the Selenium IDE or combined with Selenium Server and Remote Control with customized code.
The set up process was easier to follow for Ghost Inspector than Selenium. Decisions about choosing frameworks, how to set up them and which environment to run them in had to be considered in Selenium.
Several computers can access the user account on the Ghost Inspector website at the same time to run tests and view the test results. These can also be reported to the external message service Slack. A plugin for Slack were not found for Selenium, instead reports were stored in the local Selenium environment. It was easy to understand the reports both in Ghost Inspector and Selenium, since the results consist of the status passed or failed. An advantage with Ghost Inspector is that each test result included a video and a screenshot of the test run which makes it easier to see what went wrong from a user perspective.
Jenkins can be configured with both tools and is good for viewing test results and executing different test suites. Jenkins provides Selenium with a web based interface and can be used to execute test remotely and manage test results. Junit reports were easy to implement in code by following the guide [16], in the section “How do I use Ant to create HTML test reports”. The Selenium reports were shared manually, this process should be considered if an automated process like Ghost Inspector has been needed.
A price plan is needed when using Ghost Inspector, since 100 tests monthly in the free plan may not be enough to run tests extensively. This aspect should be considered when choosing a tool, since a price plan cost the company both employee time and a monthly subscription. Selenium has no license cost, but takes more time to set up and maintain. In Ghost Inspector, a monthly subscription is paid. Since it takes longer time to set up a project with Selenium, the cost aspect needs to be combined with other criteria based on the project needs.
The Ghost Inspector plugin records and make assertions. The simplicity was useful when the functionality was tested in the fourth test. The third test was suitable for Selenium, since the test has more functionalities to be tested, which were easier to structure and control. When building tests with Selenium, Java code was written to count objects in results with the function size (), which returns the size of a list. A count function would be useful in the visual Ghost Inspector editor, when test cases data are collected and presented.
When test suites are divided e.g. in desktop, iPad or mobile views for testing displays, all variables need to be declared in the suites in Ghost Inspector. This is not time-efficient and creates redundancy, since each suite needs to have its own variables. In Selenium variables are easier to manage and can be stored in one Java file accessible from all tests.
To test display resolution in Ghost Inspector, a list with various display resolutions is presented. Selenium lets the user set a resolution in tests freely. In need of testing a precise resolution, Selenium can be useful, otherwise both tools provide the functionality of testing various displays in a good manner.
The Selenium IDE can manage tests during recordings and integrates the alternatives, e.g. storeElementPresent, storeText, storeEval in the Firefox browser. During recordings in Ghost Inspector, only the actions: “assert” and “record” can be selected and recorded tests cannot be continued. It results in a trial and error process with long waiting times, if recorded test needs to be modified. An advantage with the Selenium IDE were that recorded tests could be continued which makes the maintenance easier.
7 Discussion
7.1 Method discussion
The limited literature study supported the case studies with information. When searching on Ghost Inspector in the ACM, Google Scholar and IEEE databases, all facts could not be retrieved from peer-reviewed papers. Information was instead collected from the Ghost Inspector website to fill the literature gap. The method of complement peer-reviewed material with information from related websites were used by the authors of [18,19] as well.
The practice-based case study is inspired by the [21] paper, where the authors act both as observers and participants. This was the only way of doing the study on iStone, since the author of this paper has the main responsibility for using Ghost Inspector on iStone. The fact-based study in 5.1 is inspired from [1] comparing facts about the tools in tables. This helped to reveal facts that otherwise may have been missed in the research. The criteria in the tables are predefined in 4.2 and built on other studies. This study can be reproduced by following the defined criteria and method in Chapter 4, together with sources found in the reference list. The practice-based study in 5.2 is made by following instructions from the Ghost Inspector and Selenium websites and is inspired from papers [4,21]. An explanation is made of each step to prevent misunderstanding and make the work reproducible. The discovered differences in the fact-based and practice-based study were clear and easy to describe, which minimize the risk of the author being subjective.
The study focuses on providing an objective result, so iStone can choose a suitable tool for upcoming projects. The result in Chapter 5 covers both the set up and maintainability processes, as asked for in the research question. A literature study and the two case studies provide enough information to present a result to iStone and others, useful for choosing between Ghost Inspector and Selenium. The method is therefore successful since it answers the research question.
7.2 Discussion
RQ: In the context of the regression testing tools: Ghost Inspector and Selenium, how do the tools differ regarding the set up and maintenance processes?
The study indicates that Ghost Inspector is preferable when testing needs to be introduced within short time. Selenium fits users with programming skills and may be the tool to use when the testing process needs customization. Ghost Inspector fits projects in need of a fast cloud-based testing solution. The learning curve is longer when using Selenium than Ghost Inspector, since the set up and maintenance process is more complicated. Both tools are useful when working agile with continuous integration and when deciding which tool to choose, the requirements in the project shall be taken in consideration.
The process of repairing tests appeared with high frequency in the project when using both Selenium and Ghost Inspector. The authors of [13] recommend writing tests before the development. This seems to be a common problem within testing, since [18,19,20] refer to the process of maintaining tests as endless. Writing tests before development could not be applied in this research, since the tests and the code already were written. It would be interesting to evaluate the process when tests are written before the code, since the authors of [13,18] recommend doing the process in this way.
The study resulted in similar conclusion as the authors of [1] concluded from their study. There is no silver bullet in software testing. Each tool is useful within different circumstances. Choosing an appropriate tool should depend on the application, the budget and the way of working in terms of efficiency. The Selenium tool offers the same functionality as Ghost Inspector and is more useful for critical application testing. If Ghost Inspector serves required demands, there is no purpose of using Selenium since it requires more knowledge and has a longer set up and maintenance process.
8 Conclusions and future work
8.1 Conclusions
IStone use Ghost Inspector as regressions testing tool on web applications, since it offers an effective set up and maintenance process. Selenium requires more effort to set up and maintain, since Selenium is not a service. The Ghost Inspector environment comforts iStone with the ability of executing test continuously and supplies the latest test results to the project team through Slack.
8.2 Future work
This study focuses on comparing two tools in terms of facts, practical set up and maintenance processes. For future work, more tools could be studied. The study could also compare other aspects or focus on specific functions. It would also be interesting to compare the monthly license cost for setting up and maintain Selenium, with the cost of a full-time employee, doing the same job.
References
[1] H. Kaur and G. Gupta, “Comparative study of automated testing tools: selenium, quick test professional and testcomplete,” Harpreet kaur et al Int. Journal of Engineering Research and Applications ISSN, pp. 2248–9622, 2013. [2] T. Parveen, S. Tilley, and G. Gonzalez, “A case study in test management,” in Proceedings of the 45th annual southeast regional conference, 2007, pp. 82–87. [3] E Dustin, J Rashka, J Pau, “Automated Testing,” Automated software testing: introduction, management, and performance, Canada: Addison-Wesley, 1999, pp. 3,4,7.
[4] R. A. Razak and F. R. Fahrurazi, “Agile testing with Selenium,” in 2011 Malaysian Conference in Software Engineering, 2011, pp. 217–219.
[5] “Getting Started,” Ghost Inspector. [Online]. Available:
https://ghostinspector.com/docs/getting-started/. [Accessed: 02-Apr-2017]. [6] “Selenium-IDE — Selenium Documentation.” [Online]. Available:
http://www.seleniumhq.org/docs/02_selenium_ide.jsp. [Accessed: 26-Mar-2017]. [7] iStone, “iStone.” [Online]. Available: http://www.istone.com/en/. [Accessed: 24-Apr-2017].
[8] L. Williams, “A (partial) introduction to software engineering practices and methods,” NCSU CSC326 Course Pack, vol. 2009, no. 5, pp. 33–63, 2008. [9] W. E. Wong, J. R. Horgan, S. London, and H. Agrawal, “A study of effective regression testing in practice,” in Proceedings the Eighth International
Symposium on Software Reliability Engineering, 1997, pp. 264–274. [10] A. Tarhini, Z. Ismail, and N. Mansour, “Regression Testing Web
Applications,” in 2008 International Conference on Advanced Computer Theory and Engineering, 2008, pp. 902–906.
[11] “Why Did We Build Ghost Inspector?” Ghost Inspector. [Online]. Available: https://ghostinspector.com/blog/why-build-ghost-inspector/. [Accessed: 10-May-2017].
[12] “API Tests Level Up: Automated Browser Test Steps with Ghost Inspector,” Runscope Blog. [Online]. Available:
https://blog.runscope.com/posts/api-tests-[14] N. Seth and R. Khare, “ACI (automated Continuous Integration) using Jenkins: Key for successful embedded Software development,” in 2015 2nd International Conference on Recent Advances in Engineering Computational Sciences (RAECS), 2015, pp. 1–6.
[15] “Jenkins,” Jenkins. [Online]. Available: /index.html. [Accessed: 23-Apr-2017]. [16] “JUnit - Frequently Asked Questions.” [Online]. Available:
http://junit.org/junit4/faq.html. [Accessed: 08-May-2017].
[17] S. Keshav, “How to read a paper,” ACM SIGCOMM Computer Communication Review, vol. 37, no. 3, pp. 83–84, 2007.
[18] Holmes and M. Kellogg, “Automating functional tests using Selenium,” in AGILE 2006 (AGILE’06), 2006, p. 6 pp.-275.
[19] S. Bauersfeld and T. E. Vos, “Guitest: a java library for fully automated gui robustness testing,” in Proceedings of the 27th ieee/acm international conference on automated software engineering, 2012, pp. 330–333.
[20] A. Bruns, A. Kornstadt, and D. Wichmann, “Web Application Tests with Selenium,” IEEE Software, vol. 26, no. 5, pp. 88–91, Sep. 2009.
[21] W. Hordijk and R. Wieringa, “Surveying the factors that influence
maintainability: research design,” in ACM SIGSOFT Software Engineering Notes, 2005, vol. 30, pp. 385–388.
[22] “Selenium Plugin - Jenkins - Jenkins Wiki.” [Online]. Available:
https://wiki.jenkins-ci.org/display/JENKINS/Selenium+Plugin. [Accessed: 09-May-2017].
[23] “API Documentation,” Ghost Inspector. [Online]. Available: https://ghostinspector.com/docs/api/. [Accessed: 09-May-2017]. [24] “Test Editor,” Ghost Inspector. [Online]. Available:
https://ghostinspector.com/docs/test-editor/. [Accessed: 07-May-2017]. [25] “Build & Deployment Integration,” Ghost Inspector. [Online]. Available: https://ghostinspector.com/docs/integration/. [Accessed: 27-Feb-2017].
[26] “Run Java Selenium tests in the cloud. 1100+ desktop and mobile browsers.” [Online]. Available: https://www.browserstack.com/automate/java. [Accessed: 07-May-2017].
[27] “Pricing & Plans,” Ghost Inspector. [Online]. Available: https://ghostinspector.com/pricing/. [Accessed: 13-Mar-2017].
[28] “WebDriver.Window.” [Online]. Available:
https://seleniumhq.github.io/selenium/docs/api/java/org/openqa/selenium/W ebDriver.Window.html. [Accessed: 07-May-2017].
[29] “Downloads.” [Online]. Available: http://www.seleniumhq.org/download/. [Accessed: 26-Mar-2017].
[30] “Creating and Using Variables,” Ghost Inspector. [Online]. Available: https://ghostinspector.com/docs/variables/. [Accessed: 14-Mar-2017]. [31] “Changelog,” Ghost Inspector. [Online]. Available:
Appendix
Test cases
1. Registration of an account
Preconditions: GIVEN that you are on the Login page Steps:
• WHEN you enter a First name, Last name, an unregistered email address, a password longer than 6 characters
AND click on Register
THEN an account should be registered and you should be redirected to the My Account page
• WHEN you enter a First name, Last name, an unregistered email address, a non-matching email address in the confirmation box, a password longer than 6 characters
AND click on register
THEN an error message per incorrect field should appear to inform you about the non-matching email address
• WHEN you do not enter anything, and click on Register
THEN an error message per incorrect field should appear to inform you about the missing inputs
2. Navigation in the mega menu
Preconditions: GIVEN that the mega menu is available Steps:
• WHEN you click on a top-level category
THEN the mega menu is opened with the second level categories connected to the top-level category
• WHEN you click on a second level category
THEN the second level category page is opened with the third level categories connected to the second level category
• WHEN you click on a third level category
THEN the third level category page is opened with the products connected to the third level category
3. Check-out test
Preconditions: GIVEN that you are on the checkout page Steps:
• WHEN you buy product(s)
THEN the order confirmation page should be displayed • WHEN you buy product(s) with Klarna
THEN the order should be created with the correct information • WHEN you buy product(s) with voucher + Klarna
THEN the order should be created with the correct information • WHEN you buy product(s) as an anonymous customer
THEN the order should be created with the correct information • WHEN you buy product(s) as a logged in customer
THEN the order should be created with the correct information
4. Search for product, category, names and item number Preconditions: GIVEN that the search box is available Steps:
• WHEN you perform a search query for a product category
THEN you should get realistic search results according to the search parameters
• WHEN you perform a search query for a product name
THEN you should get realistic search results according to the search parameters
• WHEN you perform a search query for an item number
THEN you should get realistic search results according to the search parameters
5. Stock level in stores
Preconditions: GIVEN that you are on a product page AND on a site with stores Steps:
• WHEN you click on the Stock Level in Store button with a size selected THEN a list with all available stores should be displayed
• WHEN you click on the Stock level in Store button without a size selected THEN a message should be displayed prompting the user to select a size • WHEN you click on the Show all stores with product in stock
![Table 8. These are the different plans which Ghost Inspector offer (2017-03-14) [26].](https://thumb-eu.123doks.com/thumbv2/5dokorg/4181629.90911/27.918.117.795.87.967/table-different-plans-ghost-inspector-offer.webp)
