Teaching software testing in a modern development environment

(1)

INOM

EXAMENSARBETE DATATEKNIK, GRUNDNIVÅ, 15 HP

STOCKHOLM SVERIGE 2019,

Teaching software testing in a

modern development environment

Lära ut mjukvarutestning i en modern utvecklingsmiljö

ANDERS GAWELL ANTON KALLIN

KTH

SKOLAN FÖR ELEKTROTEKNIK OCH DATAVETENSKAP

(2)

(3)

i

Abstract

All developers understand the benefits of testing their code to ensure its functionality. Today’s market is moving further towards design principles where testing is a central or driving force during development. This puts a certain pressure on academia to supply these skills to their students.

Recently the course II1302 - Projects and project methods at the Royal In- stitute of Technology in Kista made a concerted effort to introduce the students of the course to these modern concepts. This thesis investigates how areas of testing can effectively be introduced to the students in the course, utilizing a tailored example that takes the area of testing into particular consideration and how to automate it via DevOps-tools provided by a cloud-based service.

Further, it also makes an attempt to provide additional material to be used for teaching testing in conjunction with the example provided.

The case study covers the development of an example application, meant to mirror a typical student project. It also covers how this was used for teaching the students about the testing areas considered. The covered testing areas include unit testing, integration testing and UI testing. With these given testing areas, the application and an associated learning module was developed for each area in question. Relevant standards, strategies and approaches was also identified for each of these areas.

The thesis also presents important properties to take into consideration when developing similar examples in the future, based on the experiences ob- tained during the study. These include needs such as understandable by inex- perienced students, applicability outside the course, adherence to established standards, tools that are simple to use and an architectural structure that al- lows for testing.

Some improvements are also recommended: the students would benefit from learning software testing from an early stage of their studies. The content of the learning modules should also be brought to the students earlier in the course, so it can be applied in their projects at an early stage as well.

Further research is also recommended to evaluate the suitability of using other cloud-based environments instead, and to evaluate the applicability of the learning modules for students of varying disabilities.

Keywords: Teaching, DevOps, Test automation, Node.js, Unit testing, Integration testing, UI testing, Continuous Delivery, Continuous Integration, Student project, Cloud services.

(4)

Sammanfattning

Alla utvecklare förstår fördelarna med att testa kod för att garantera dess funk- tionalitet. Dagens industri går i en riktning där testning spelar en central del av design under utveckling av mjukvara. Denna tendens lägger en viss press på högskolan att lära ut dessa erfarenheter till dess studenter.

På senare tid har kursen II1302 - Projekt och projektmetoder på Kungliga Tekniska Högskolan i Kista tagit en stor ansats för att introducera sina studenter inför dessa moderna koncept. Denna uppsats undersöker hur testningsområdet effektivt kan introduceras till studenterna inom denna kurs, genom att utnyttja ett egengjort exempel som har området i fokus, samt att automatisera detta via DevOps-verktyg tillhandahållna av molnbaserade tjänster. Dessutom görs även en ansats för att tilldela ytterligare material som kan användas för att lära ut testning av mjukvara i samband med det givna exemplet.

Fallstudien omfattar utvecklingen av en exempelapplikation, som var av- sedd att likna ett typiskt studentprojekt. Den täcker även hur denna användes för att lära ut de betraktade testningsområdena till studenterna. De täckta om- rådena av testning inkluderar enhetstestning, integrationstestning och testning av användargränssnitt. Med dessa givna testningsområden utvecklades både applikationen och dess associerade lärmoduler för vardera testningområde i fråga. Relevanta standarder, strategier och metoder var också identifierade för vardera av dessa områden.

Denna uppsats presenterar även ett antal viktiga egenskaper att hålla i åtan- ke vid utveckling av liknande exempel i framtiden, baserat på erfarenheterna från studien. Detta inkluderar behov som tillgänglighet för mindre erfarna stu- denter, applicerbarhet utanför själva kursen, tillämpning av etablerade stan- darder, utnyttjande av lättanvända verktyg och en arkitektur som tillåter test- ning.

Några förbättringar föreslås även: studenterna skulle gynnas av att lära sig om mjukvarutestning i ett tidigt skede av sina studier. Innehållet i lärmodulerna bör även presenteras för studenterna tidigare i kursen för att kunna appliceras i deras projekt.

Ytterligare forskning rekommenderas även för att utvärdera andra lämpliga molnbaserade miljöer, samt för att utvärdera tillämpbarheten av lärmodulerna hos studenter med inlärningssvårigheter.

Nyckelord: Undervisning, DevOps, testautomatisering, Node.js, enhets- testing, integrationstestning, UI-testning, Continuous Delivery, Continuous In- tegration, studentprojekt, molntjänster.

(5)

Chapter 1 Introduction

An increased pace in competition amongst businesses, along with a steadily changing need in the market has influenced the way enterprises operate in the IT business[1][2]. Besides not being able to have customers wait for months for a new release anymore, customers also expect to be able to engage with the developers via continuous feedback [2].

This drive towards agile development and pushing the development-side into the same space as the operations-side, i.e. DevOps, brings distinct require- ments to the development process. An important part of this process revolves around continuously testing the software delivered[2]. In order to optimize this, the need for automation has also become an essential part of agile development[3].

While academia is working to meet the market demands of theses compe- tences, there are gaps in knowledge to be filled here, specifically in the area of testing. How can academia hope to match the market demand for compe- tence of developing tested code, as well as apply this in a modern development environment which also demands its automation?

1.1 Background

The course Projects and project methods, II1302 is held for second year stu- dents in the Degree Programme in Computer Engineering, TIDAB, and the Degree Programme in Electronics and Computer Engineering, TIEDB, at the Royal Institute of Technology (or KTH). One of the course’s intended learning outcomes is to “Apply an appropriate project processes appropriate to the art Information Technology (IT)”[4]. The participating students are working in project groups of 3 to 5 people. Each project group is assigned an IT project

1

(10)

where each project group member is expected to contribute both in the software development process and project management.

All students are also assigned at least one formal role. For these, a student has the main responsibility of project management, stakeholder representation, software development, software architecture, or software testing. The students are trained in these roles at seminars during the course, as well as how to apply their knowledge in their respective project group. In addition to the seminars, the course material is hosted on Canvas, the learning management system used by KTH.

From spring 2019, it is recommended that the student project is being hosted on the cloud as well as be designed as an Internet of Things appli- cation, to incentivize learning about these modern concepts. It is because of this that the students are being introduced to DevOps and its methods during the course, via both the cloud service itself, and by designing applications that take Internet of Things into account.

1.2 Problem

Since the projects are hosted on a cloud service, new demands of the students have emerged. The course’s ambition is that the development process is at least partly automated, and that the project groups use some of the DevOps facilities provided by the cloud platform. Another problem is that the TIDAB and TIEDB programmes have no course on the subject of testing [5]. This means that no prior experience of testing can be expected from the students.

Because of the new requirements and the fact that the students have very little experience in software testing, the teaching of the students in charge of testing has to be enhanced and adapted to these circumstances. For this, new learning material needs to be developed. The learning material will be a Canvas module consisting of the following:

• Written material

• Video material

• Seminars with the students in charge of testing

• An example application demonstrating how to implement testing in a project similar to those in the course.

(11)

CHAPTER 1. INTRODUCTION 3

1.2.1 Research Questions

1. How to create an example application suitable for learning about modern development environments, and testing?

2. How to use the example application to teach about development environments, and testing?

1.3 Purpose

The purpose is to examine how to assist students in increasing their knowledge about software testing, automation of software tests, and how to use these in a DevOps context.

This thesis is primarily in the interest of KTH itself, to be able to improve the quality of the education provided by the institution. It is also beneficial for the students themselves in several ways, to get them better prepared for working in large IT projects, and to develop software of higher production quality.

For employers in the IT industry it would reduce the need of introduction for newly graduated employees. These new engineers can also in many cases contribute with new knowledge about modern software development practices.

1.4 Goal

The goal is to create an example application and associated learning modules for teaching in the course II1302. Each learning module should cover differ- ent important aspects of testing as part of software development. The example application is aimed to cover these relevant aspects taught in the learning modules.

The concepts taught and the examples provided should be relevant to the course II1302, and applicable to the students studies.

Also of importance is that the material should be at an adequate difficulty level for the students. Again, testing is only part of the course for the students, and the students cannot be expected to have extended knowledge in this.

This thesis addresses these problems directly by identifying relevant areas of knowledge, creating an application and associated learning modules that can be used to learn about these relevant concepts.

(12)

1.5 Delimitations

This thesis is a case study, which was conducted within the following primary constraints:

• The study will focus on the requirements for the course Projects and Project Methods at KTH.

• The example application will utilize a MERN-like stack via JavaScript.

– CouchDB in particular will be used instead of MongoDB.

– Express.

– React framework.

– Node.

• The project will be a built-in implementation, using IBM’s Cloud Foundry service.

• The project will use the DevOps tools provided IBM Cloud’s service specifically.

– The items must be included in IBM:s Lite plan, so as to not spend any money on the service.

• The study is only concerned with the following areas of software testing:

– Unit testing.

– Integration testing.

– User Interface testing.

1.6 Document Outline

Background Theory describes the fields and terms that the thesis concerns.

This is to prepare the reader for the remainder of the report.

Methodology details how the thesis was conducted, as well as how the prob- lem and issues were approached. The purpose behind this is to provide transparency behind how the studies in the thesis were conducted.

Results describes the complete process of development, and summarizes the result achieved.

(13)

CHAPTER 1. INTRODUCTION 5

Discussion & Conclusion provides summarized answers to the research ques- tions, and contains a discussion regarding the results arrived at, and the methods used to produce these.

(14)

(15)

Chapter 2 Background Theory

2.1 The MERN-stack

All web applications are built upon multiple technologies, referred to as a

“stack”, as popularized by the LAMP stack¹. The MEAN-stack was an early alternative that emphasized a shift towards single page applications, that used lightweight calls to only update parts of a website, rather than the whole page.

The stack was composed of MongoDB for persistent storage, Express as a web- server built on Node, Angular for the front-end framework, and Node for the server-side runtime environment.[6] Besides the appeal of shifting towards single page applications, it also allowed for a single underlying language across the entire web application, i.e. JavaScript[7].

Since the MEAN stack was first introduced, React has since been launched as an alternative front-end technology, hence the name MERN stack.

The difference between these is that while Angular is an MVC framework, while React is merely a library for building a user interface.[6]

2.2 The Internet of Things

The Internet of Things, or IoT, is not a concept with a single clear definition², since as it became more universal, so too did the definition evolve to more general terms[8].

One of the earliest definitions of IoT defines the term as a concept where a

1LAMP is an acronym for Linux, Apache, MySQL and PHP, which are the technologies that form the bundle, or stack.

2See reference [8] for a more complete collection of definitions of Internet of Things.

7

(16)

variety of things and objects in the environment are able to interact and coop- erate to reach a common goal. These objects can, for example, be RFID-tags, sensors, mobile phones, among other things[9].

Another definition provided by the IEEE Internet Initiative makes a distinc- tion between small and large environment scenarios, in terms of complexity and management of the “things”. For the smallest scenario, an IoT is said to have “things” of actuating and programmable capabilities, where their states can be collected or changed from anywhere at any time[10]. In a larger environments, the “things” in question are characterized as being owned by different administrative or managing domains without explicit relationships, with the issue of scalability becoming a dominant point[10].

While both, and maybe even most, of these definitions vary in terms of word, they do include several common elements. As defined by Hassan, Khan, and Madani [8], an IoT is made of uniquely identifiable objects, that can both be physical or virtual. Further, these objects can capture their surrounding context (via sensors) and send this data over the Internet, or even change their own state, or the state of their surroundings. Also, all of the above, should ideally function without much, or any at all, human intervention.

2.3 Cloud Computing

2.3.1 Platform as a Service (PaaS)

Platform as a Service (or just PaaS) is an abstracted and integrated cloud- based computing environment that supports the development, running, and management of applications. Application components may exist in a cloud environment or may integrate with applications managed in private clouds or in data centers[11]. Working in a Paas cloud environment, programmers can focus on providing proper source code, since operating system, middleware and runtime usually is provided by the service[12]. Some popular PaaS solu- tions are AWS Elastic Beanstalk, Google App Engine, and Cloud Foundry.

2.4 DevOps

The term DevOps is a portmanteau of the words Development and Opera- tions [13][1]. It describes several practices to simplify cooperation between those departments.

(17)

CHAPTER 2. BACKGROUND THEORY 9

2.4.1 Development

Dev area refers to the development of an application or service, via say a soft- ware company. Besides the coding of a product, it also entails the planning of how problems are supposed to be solved, in what ways they’re aiming to test them and building the code itself. For this to be possible, it usually includes some automation, via say testing frameworks for unit and integration testing, to allow for such continuous integration.

2.4.2 Operations

The Ops area is concerned with the actual deployment of an application or service, to maybe a customer. Not only does software have to be deployed to a given end-point, say a company server, but it also presumably requires some operation on the user end. This operation is also monitored in some capac- ity to provide some feedback how the end users are using the software. This monitoring can range from anything from a simple crash report to extensively detailed user statistics via trackers. The important part here is that some feed- back about the software is fed back to the Dev area so some improvement can be made.

2.4.3 Important DevOps Concepts Continuous Integration

The principle of continuous integration revolves around integration not occurring in a single phase of development, but throughout the entire process.

Having developers work in independent branches for long periods of time con- sequently increases the likelihood of introducing breaking changes when the branches are merged. In large change-sets it becomes hard to identify what’s caused the error to occur in the first place. Continuous integration demands smaller change-sets on more frequent periods to alleviate these issues, and systems that implement this usually runs a series of automated tests before merging code[3].

Continuous Testing

To continuously integrate, there’s also a demand for ongoing testing and verification of the code. This naturally results in shorter phases for testing, and also gives a certain quality guarantee very early in development. Automated testing

(18)

is the primary way to allow for this, but it may also simulate an environment similar to the production environment being fed various use cases[13].

Continuous Delivery

Closely related to continuous integration, continuous delivery is what allows for continuous integration of software. It is a set of general software engineering principles that allows any changes to a new release to be deployed after the tests for integration have passed[3].

Continuous Deployment

Linked closely to continuous delivery, continuous deployment refers to putting that delivery into production. Essentially, continuous delivery ensures that a new change can be deployed, while continuous deployment ensures that they are deployed into production[3][14].

Continuous Monitoring & Feedback

These concepts revolves around the feedback Ops returns to Dev. Continuous monitoring provides metrics to operations and various stakeholders at different stages of the delivery of an application[13]. Continuous feedback refers to how the end user is using the software, which can be either raw data or elaborate feedback[13].

2.5 Software Testing

2.5.1 Testing Areas Considered Unit Testing

To test a completed unit it must be proven to satisfy its intended purpose. These kinds of tests include some documentation, an objective and an expected result, and are written by the programmer and are executed on a completed unit (or module).[15] The units under test are isolated during unit testing. To ensure isolation, external units are simulated. These simulated units are referred to as mocks.[16] In essence, the modules are verified that they do perform the task expected of it. In unit testing it is desirable that all possible scenarios for the unit are tested.

(19)

Figure 2.1: The test automation pyramid, with tools used in this thesis catego- rized by layer.

Integration Testing

When individual units have been verified for their intended behaviour, these tests are held to ensure their behaviour when combined. The purpose behind this is to identify potential failures when integrating different components, since this is not guaranteed when they work as intended when isolated.[15]

UI Testing

As the user provides a system with input, they expect changes in the interface provided. The testing process for this area verify that the system is responding to the user in an expected manner.[17]

Acceptance Testing

This testing is carried out by the customer to verify that the software matches the customer’s expectations prior to acceptance.[15]

2.5.2 Testing Models The Test Pyramid

The test pyramid[18] (see figure 2.1) is a guideline model describing how to set up and distribute automated tests. As defined by Martin Fowler, software developer and co-author of the agile manifesto[19], the model suggests that

(20)

Figure 2.2: The V-model illustrated.

more lower level unit tests should run through an application than higher level GUI-tests[20].

Going upwards in the pyramid, the cost and time consumption a test requires increases. The size of each section reflects the proportions of tests that should be written. The model indicates that most of the written tests should be unit tests, less tests should be integration tests, and the least amount of tests should be UI tests [20][21].

Note that the names and amount of layers in the pyramid can vary from source to source. Some call what’s identified as Integration in figure 2.1 as the API layer[22], or the Acceptance or Service layer[20][23]. Others even identify several intermediary layers like a Component layer, or even separate layers like Integration and API as different types of tests[22]. Usually this difference comes down to the focus of the application being developed, but regardless of the naming convention, the principle in the pyramid remains the same. Tests down at the lower levels of functionality are more cost-effective than ones at the higher levels.

The V-Model

The V-Model is a representation of the development process, and how the phases of developing code relates to testing it[24]. This model is illustrated in figure 2.2, where the downward slope represents verification and planning and the upwards represents validation via testing. At each level a verification phase corresponds to a validation phase. The horizontal distance between phases represents the time required for successful validation, and the vertical level between phases represents the difference in functionality.

For example, the unit tests validate the detailed design of individual components. These unit tests are of a lower level of functionality than the integra-

(21)

tion tests.

(22)

(23)

Chapter 3 Methodology

3.1 Research Methodologies

3.1.1 Qualitative & Quantitative Methods

The choice between a qualitative or a quantitative research approach depends on if a project is numerical in nature or not. Quantitative research requires measurable data to verify the hypothesis in question, and as such requires a quantifiable nature[25]. This kind of research also carries an additional requirement of a non-vague hypothesis that’s answerable. Qualitative research concerns understanding of meanings and behaviours to reach tentative hypoth- esizes and theories[25]. Despite this added vagueness however, the research still may answer the hypothesis sufficiently.

3.1.2 Inductive & Deductive Approaches

The research approach is what establishes what is true and false[25]. There are primarily two methods to determine this, which are inductive and deductive (barring the abductive approach, which uses both of the aforementioned approaches).

Inductive reasoning formulates theories and propositions with alternate explanations from observed patterns. Commonly data is collected via a qualitative method and afterwards analyzed to arrive at these explanations. These propositions are outcomes based on the behaviour observed, but does carry the added demand that there must be enough data to be able to derive an explanation of the behaviour in question[25].

Deductive reasoning builds upon the testing of verified theories, to either

15

(24)

verify or falsify a hypothesis. Usually this is done by rigorously testing these against huge data sets, which results in an outcome that’s a generalization based on the collected data (i.e. does the proposition hold for our data?)[25].

3.1.3 The Case Study

Case studies are a research strategy that involves using an empirical investiga- tion of a phenomenon in a real life context.[25] In essence, it is research into a specified area by observing the outcome of the case in question. A case study can be based on either quantitative or qualitative evidence, as well as a mix of both[25].

3.1.4 Bunge’s Method

Mario Bunge defines the general scientific strategy as the following sequence of ten steps:[26]

1. First, a problem has to be identified.

2. Afterwards a choice of approach must be decided upon. This includes methods included, the goal to be striving towards and background knowledge.

3. With an approach given, the problem can be be formulated with this chosen approach. Essentially, the initial raw problem is defined into a problem that is more defined.

4. With the problem defined, more relevant information can be gathered in preparation for solving the problem.

5. With sufficient knowledge and a well defined problem, a research plan can be designed.

6. Now that a plan has been established, it’s time to implement it. The plan that’s been decided upon must be kept to during implementation.

7. At this point a given approach has been provided as well as an implementation of it. All that is left to do is to check the outcome of the implementation.

8. Now, with a result, the candidate solution can be evaluated with regards to the background knowledge acquired.

(25)

CHAPTER 3. METHODOLOGY 17

9. If the solution is satisfactory, the background knowledge can be extended or revised.

10. If the result isn’t satisfactory or found acceptable, the research process must restart. This can either include replicating the same process or by modifying a significant component, like the method or hypothesis.

3.1.5 Applied Approach

This thesis has a number of understated factors that are crucial for the choice of approach to answer the research questions. Firstly, the thesis is less concerned with pure data, rather than an interpretation of a description, since it aims to provide an answer to the “Hows” in the research questions. Because of this qualitative methods were used to allow room for descriptions, as well as the research questions themselves. Secondly, the research questions do not have an underlying hypothesis, besides the assumption that the problem can be solved. Since conclusions are instead derived from the experiences garnered during the research process itself, an inductive approach was also deemed appropriate.

For these reasons, this thesis is a qualitative case study. The approach of using a case study was decided upon as the answers to the research questions aren’t necessarily universal.

3.2 Research Method

The problem is defined from the research questions defined in 1.2.1, with the delimitations in 1.5 and course goals in regard. The research questions first need to be divided into sub-questions:

1. How to create an example application suitable for learning about modern development environments, and testing?

(a) Which properties should the application have?

(b) Which tools and technologies should the application use?

2. How to use the example application to teach about development en- vironments, and testing?

(a) What content should be presented?

(b) How should the content of the learning modules be presented?

(26)

Figure 3.1: Outline of the research method

To answer the research questions, and its sub-questions, the research process goes through the following steps shown in 3.1.

3.2.1 Literature Study

Information about the concerned areas is gathered in regard of the questions in the previous step. Suitable methods and established standards needed to be taught to the students, are identified. The resources in the literature study is mainly scientific articles, books, technical documentation, and video walkthroughs.

(27)

3.2.2 Problem analysis

The questions from the problem statement are delimited and further clarified using the information gathered in the literature study. If further questions need to be added, they are also defined here. Hypotheses are drawn from the questions. Requirements for evaluation of the hypotheses are also defined.

3.2.3 Development

In the case study, the plan set up in the previous stage is implemented. The plan is strictly followed. The planned functionality is implemented for the example application, with the requirements from the planning stage in regard.

The written material is created and a video walkthrough is recorded. The video and the written material is published on Canvas, and the changes to the example application are pushed to the GitHub repository. A seminar is being held for the students, where information gathered from the literature study is presented, and put together and delimited in regards of the planning of the case study.

3.2.4 Gathering Result

The resulting modules and the example application are gathered. The authors’

observations from the development and creation of the example application and the learning modules are discussed and documented. Feedback from An- ders Sjögren, and the students is gathered and documented as well.

3.2.5 Evaluation

The hypotheses from the planning stage are evaluated against the gathered result and documentation from the previous stage. The gathered results and the evaluation of the hypotheses are used to answer the research sub-questions, which in turn are used to answer the research questions.

3.3 Project Method

To answer the research questions the case study takes an iterative approach based on Bunge’s definition of the scientific method. The process is illustrated in figure 3.2, and goes through three distinct main phases:

(28)

Figure 3.2: Workflow of the development of the application and its associated learning modules. The gray elements indicate indicate how the method relates to Bunge’s method.

(29)

Figure 3.3: Development sub-phase, which details the development of the application and its associated learning modules.

(30)

Pre-study This phase encapsulates the definition of the work before a plan has been set. Before the research plan is produced and adhered to, the study remained in this state. This conforms to steps one to five in the sequence described in section 3.1.4.

Method is a sub-phase that encapsulates the gathering of relevant informa- tion and the definition of the research plan. It is part of the pre-study as the work hasn’t started yet, but requires that the problem itself has been defined. This pertains to steps four and five described in Bunge’s method. At the end of this phase we define a hypothesis for the research questions and its sub-questions.

Implementation This relates to the implementation of the provided plan from the pre-study phase. The requirements imposed by each iteration, i.e.

testing field, are prioritized and planned before development commences.

Development This sub-phase encapsulates the work of each iteration, which is illustrated in figure 3.3. It includes the development of the example application and the learning module, which is done by setting up tests against the requirements put forth beforehand, and developing the application around these.

The application is validated by putting it through these pre-made tests, and development continues if they don’t pass. If they do pass, development shifts to the learning module, by documenting the results in a lab compendium, and expanding this to include the relevant knowledge needed to understand the example and concerned testing field. In addition to this, a video walkthrough is recorded as well. If there is time available to extend the example, development on the app is resumed and the sub-phase restarts. Otherwise, the module is presented at the oral seminar and feedback received from the students are documented as well for future work. The oral seminars are a final deadline and signifies the end of one iteration and the start of a new one.

Each iteration goes through the steps included in this sub-phase at least once to construct a result, evaluate it and revision it if necessary. This also pertains to steps six to ten described in section 3.1.4.

Conclusion This details the discussion and conclusions drawn regarding the project and its suitability. In this phase a thorough evaluation of the final result is done, along with the process itself. Also, more thorough feedback from

(31)

the students of II1302 is also acquired to give a different perspective on the process. The final result is presented for peer-review and a final revision is made before submission.

3.3.1 Development Iterations

The development of the test module and the example application is divided into four iterations:

1. Unit testing the Server.

2. Integration testing the Server.

3. Acceptance testing the Graphic interface.

4. Unit testing the Client.

In each of these iterations, the rigid structure described in the implementation phase is always followed, and is based on the MoSCoW method as well as Bunge’s method. An iteration has a single field in software testing to implement into the example application.

Throughout the iterations, the functionality of the application is expanded upon. In each iteration, tools pertinent to the area at hand are considered, such as frameworks and other technologies, to form a conclusion on what’s most suitable for the user-group.

Each iteration also allows for feedback from the user-group by allowing them to watch the example take form successfully and how the testing area in question is applied to it. This feedback primarily comes from the oral seminar with the students, where the testing area is explained, the expanded functionality of the application is described and how the area is implemented into the application.

3.3.2 Project Requirements The MoSCoW method

The MoScoW method is a technique to manage project needs by ordering the requirements based on their priority to the project as a whole.[27]. These priorities are divided into four main categories:

Must These requirements are critical to the project and must be met. In a sense, they are the core functionality that the project lives and dies on.

(32)

Figure 3.4: The project management triangle

Should These requirements are important, but not critical to the project. Since they aren’t entirely central to the project they have the possibility to be absent when the project reaches its conclusion, but they still represent a large chunk of what is expected of the project.

Could These requirements are of less importance, but will still be valuable to the project. Not necessarily neglected, but the project can survive without it.

Won’t Not in the scope of the project, but might be valuable in later works.

These are important to note, to ensure that the project delivers what’s expected of it, and more importantly what it will not.

Application of Requirements

According to Eklund [28] the Project Management triangle (fig 3.4) divides the project constraints in three categories:

• Time

• Cost

• Quality

At least one of the constraints needs to be flexible. In this project, the time constraint is fixed: the project needs to be finished in 10 weeks. The Cost constraint is also fixed: The two authors of this thesis has 40 hours per person and week to work on the project. This means that the quality constraint needs to be held flexible. Therefore some project requirements may not be fulfilled.

(33)

To ensure that the requirements are prioritized correctly, this project uses the MoSCoW method for requirement priority management (see section 3.3.2):

Must

• Deliver a suitable minimal example application with a DevOps environ- ment and IoT in mind.

• Provide a study on testing modules and the example application for a student project (this report).

• Provide the students with a document detailing the testing area in question and how it is implemented in the example application.

• Deliver a video walkthrough explaining the implementation of each testing in more detail.

• Host a seminar walking through the testing area and the implementation, to allow for instant feedback from the user group.

Should

• Include a set of frameworks and/or libraries suitable for each testing field.

• Detail how the frameworks and libraries used in question interacts with each other.

Could

• Deliver a quiz testing the students knowledge for each testing area.

Won’t

• Deliver testing examples that assumes anything other than JavaScript being used.

3.3.3 Focus Group

A group consisting of the head of testing for each project group will function as a focus group. Members of the focus group will be interviewed regarding their group’s individual implementation of testing, and how they perceive said implementation. The information collected from the interviews will be a part of the evaluation of the testing methodologies and frameworks.

(34)

3.4 Technologies Used

IBM Cloud: The full stack cloud platform used by the course in question.

Provides the DevOps environment in particular via toolchains.

Toolchain: A set of tools used for development, deployment, and operation tasks[29]. Provides an interface for reading changes in the repository, along with a delivery pipeline.

GitHub: Hosting service for version control of code. Used to host the source code of the example application, along with being a catalyst for the toolchain via acting on new versions of source code being pushed to the repository.

Node: JavaScript run-time environment, allowing JavaScript to be run outside of a browser. Used as the basis for the example application, as well as for server-side functionality.

Express: Node framework used for building web applications. Provides var- ious utilities for server-side functionality, like allowing HTTP requests to be handled via API:s. Used for the example application’s back-end, by having an application script act as a webserver.

Jest: A testing framework for Node. It provides a test runner, a built-in asser- tion library, and code coverage functionality.

React: JavaScript library used for building user interfaces and managing client- side states. Used for the example application’s front-end.

(35)

Chapter 4 Results

4.1 Example Application

The example of an IoT application selected, was a punchclock, and it was continuously used as an example in the course.

The system was intended to have two different types of users, employees and administrators. Employees use an IoT device to allow employees to reg- ister their punch-in and -out times to which are then stored as timestamps in a non-relational database. The administrators however has a web interface that renders the registered (i.e. “punched”) timestamps of the employees. Ad- ministrators can either review the total worked hours of all employees, or the registered timestamps of a single employee, by navigating the interface.

As the application wasn’t intended for commercial use, the focus on extended functionality is not a primary concern, hence administrators only being able to filter the registered timestamps for single employees. Issues that do not pertain to this functionality or the issue of testing were not considered during development, as the end result was intended to be as lightweight as possible.

In essence, the application is merely meant to reflect a typical student project, while also providing enough functionality to allow to be tested.

4.1.1 Web Application

The web application runs in a Node runtime environment, with an Express ap- plication as the web server [30]. The server provides the user interface through a React front-end. Application data is fetched from the Cloudant database, and is provided to the React front-end via a HTTP API. The server part of the application is implemented in a layered architecture. The IoT application, which

27

(36)

Figure 4.1: Structure of the Punch clock example.

will continuously be used as an example in the course, will be a punch clock.

This system will use an IoT device to allow employees to register their check- in and check-out time, which are then stored in a database. This system will also have a web interface to allow employers to review the working periods of their employees. This approach allows for a web application that incorpo- rates a typical stack of software while also incorporating IoT devices into the structure (Fig 4.1).

4.1.2 DevOps toolchain

The DevOps tools used for the example application is built upon IBM Cloud’s cloud-native toolchain[31]. The toolchain uses integrations for Github, the Eclipse Orion Web IDE, the Slack team collaboration service, and a deploy- ment pipeline.

The deployment pipeline (fig 4.2) consists of the five different stages:

Build The source code is fetched from the GitHub repository. The build script in the package.json file is run, and dependencies are installed. The build artifacts are compressed, and passed to subsequent stages.

Unit and integration tests Unpacks the artifacts from the Build stage. The unit tests are run. If all tests passes, the next stage is run.

Deploy to test Runs a script passing the build to a Cloud foundry instance designated for running a test version of the application.

GUI tests Unpacks the artifacts from the Build stage. Runs GUI tests against the test application. If all tests passes, the final stage is run.

(37)

CHAPTER 4. RESULTS 29

Figure 4.2: The deployment pipeline for the Example application

(38)

Deploy to production Pushes the build to a new Cloud foundry instance. This instance runs the actual fully tested application.

One of the goals was to make the cloud services fit within the free Lite plan. In the Lite plan there is a limited amount of memory usage allowed. Therefore, the test application is stopped when the GUI testing is finished, right before that deployment of the production version of the application. This does not affect the performance or quality of the application. Since the test applica- tion is stopped in the beginning of the Deploy to production stage, the test application could still be used for manual acceptance testing, if the Deploy to production is triggered manually.

4.2 Course Modules

Each of the considered testing fields was designated a course module. The contents of these modules are primarily composed of three components, along with an oral seminar. These components includes a lab compendium, a video walkthrough and the slides used during the seminar.

The seminar and lab compendium serve the same purpose, which is to introduce the area in question, provide some examples, and show an implementation of the idea through the example application. While the lab compendium is intended to convey the same information as the oral presentation, albeit in written form, it does provide more opportunities for elaboration.

The video walkthrough also serves to introduce the testing area considered, via examples and the settled implementation of the area for the example application. The walkthrough also visually show how the environment was set up on the DevOps environment used, to allow the user group to retrace the steps taken to allow the testing to occur.

While the presentations at the seminars were not recorded, they do still follow a set flow, via the presentation slides used. As the slides are an integral part of the presentations as well, they were also included in the modules to allow a different mean of divulging the information given during the seminar.

4.2.1 Unit Testing

For the first course module the students are given an introduction to the concept of automated software testing, starting with the individual units. To this end, a description of the application from both perspectives were given, along with a settled structure and functionality requirements, to establish the context behind

(39)

the example. As an additional primer of the application structure, the MERN stack, as well as the components included, was introduced to further motivate the application structure.

The process of setting up automated tests was also elaborated upon by in- troducing the Test Pyramid, and stressing the importance of lower level testing over high level tests. The design of unit tests was also discussed, as the argu- ment type directly affects the range of cases a unit tests should include. To this end, the examples given provided instances of “bad data” as well, to illustrate the further size of data to test against.

With the necessary background established, the server-side logic was presented, along with the chosen representative cases used to test it. This devel- oped functionality corresponds to the Calculations unit in figure 4.1, which includes a functions that takes a list of timestamps in the form of JSON objects and determines the total time worked from that list. To test this, an ordinary case of several timestamps was provided, along with several different cases of bad data. For the ordinary case, the test merely succeeded if the calculation re- turned the correct result. For the cases of bad data, the function was expected to throw errors as a result to prove the expected behaviour of the function.

These cases included punch-outs occurring before punch-ins, timestamps that go outside the boundaries of the 24-hour clock, and a timestamp including a non-representative type of the clocked time. These bad data cases exemplify illegal data values and constructs, that go outside the boundaries of the function’s purpose, which should be guarded against. Other examples of bad data covered, but not exemplified include cases of no data, undefined values and cases of incomplete or overflowing data structures.

Finally, the incorporation of automating the unit tests via IBM Cloud:s DevOps Toolchain was described, more specifically via the environments im- plementation of a delivery pipeline. As the stages in the pipeline are primarily composed of BASH scripts, a stage could be included that both resolved the Node dependencies required, along with running the tests. A stage deploying the application to a production environment, provided by the cloud service, was also included.

4.2.2 Integration Testing

For the second module the students were introduced to integration testing, along with the choice of integration strategies. As the application and MERN structure had already been presented, the seminar moved directly towards this next level of testing, in accordance with the Testing Pyramid.

(40)

Since the design of the tests didn’t differ much from the unit tests, besides the importance of testing results rather than implementation, the students were instead introduced to some strategies associated with integration testing. The strategies of Big Bang, Top-Down, Bottom-Up and Sandwich were all introduced, along with their respective strengths and weaknesses. Alongside these, the use of drivers and stubs in each of the integration strategies was also made apparent.

As the module was primarily focused on testing of internal components, external components and libraries was considered very lightly. In fact to push the performance of the tests, the students were encouraged to hide the interac- tion against external systems, like file systems and databases, during testing via mocks. This reason behind this was to emphasize the importance of keeping the tests as lightweight as possible, but also to keep track of what’s important to test. The students were however given advice on how to test against external components, by only testing via the code that interacts directly with these components (again, keep the tests as isolated and small as possible). An example of an external integration test was also given by testing the HTTP endpoints through SuperTest[32], a testing library for Node HTTP servers.

4.2.3 User Interface Testing

For the third module the students were introduced to the concept of GUI testing and the tools used to automate this. Again, since the seminars assume that the students took part in the previous ones, the seminar moved straight along to the final level of the Testing Pyramid.

As a start a brief overview of the different types of GUI tests were provided. Specifically, the distinctions between functional testing, visual testing, acceptance tests and automated GUI tests was highlighted and their applicability. While the focus of the module was on scripting automated GUI tests, alternatives like manual tests and record/playback automation was also as an alternative. The reason for this was to further illustrate the strengths and weaknesses of these approaches, specifically compared to scripted tests.

With this background the Selenium framework itself was presented to illustrate scripted tests. Utilizing the framework’s official JavaScript API, user input was simulated by selecting HTML elements via CSS selectors and utilizing relevant functions provided by the API to “click” these. Since the students had been encouraged to use JavaScript from the start of the course, only the most common JavaScript implementations of Selenium was presented and used during the seminar.

(41)

Besides this, students were also informed about several best practices when testing GUI, as well as common pitfalls when testing the GUI. The importance of testing against “bad input” was especially stressed, along with a short recap of malicious input via common attack techniques such as SQL injection[33]

and Cross-site scripting[33].

The video walkthrough, besides covering the aforementioned areas, also provided an example of how to create Selenium tests from Use Cases, using Jest. The incorporation of Selenium’s JavaScript API and Jest was also covered here, as Selenium itself doesn’t facilitate assertions when comparing during tests (not a problem since Jest could fill this gap instead).

4.2.4 React & Front-end Unit testing

The fourth module covers the concept of front-end frameworks. For this module no seminar was held, nor was a compendium made. Instead the relevant tools was merely linked to along with their own tutorials. The main focus of this module revolved around the React framework and how to create unit tests for a React module. Common tools for layout, building, and type checking were also referred to.

The reason behind the lack of a seminar came down to the fact that very few groups had implemented React into their applications. Also at this point in time in the course, the students were in the concluding phases of their projects, so the knowledge imparted wouldn’t be able to be applied. Because of this, it was decided that React would only be mentioned, as another seminar wouldn’t serve much of a purpose for the projects.

(42)

(43)

Chapter 5 Discussion & Conclusion

5.1 Course Observations

As the student groups were subjected to the learning modules, several diffi- culties were encountered by them during the progress of the course. The following observations are the results of informal sessions with the students and their projects, and are not the result of a quantitative analysis. Despite this, the concerns raised are applicable to the course’s learning modules. The fourteen groups in question consisted of three to four members each.

• Unit tests were sparely used universally. The student groups simply didn’t prioritize testing to the same extent as the rest of the course’s contents.

• While most of the student projects were written in entirely in JavaScript using a Node back-end, at least three groups didn’t. These resorted to the languages Python, PHP, and Scala. As such, they had to resort to testing frameworks outside the authors recommendations.

• Some groups used back-end frameworks for their projects. Frameworks used were Flask, Play, and Laravel.

• Many of the projects featured architectures that complicated implementation of testing. These flat unstructured projects didn’t facilitate unit- or integration testing, as they lacked components to test. For these, automated UI tests were simply the only resort.

• Several student groups preferred using relational databases, such as DB2, over the non-relational alternative Cloudant.

35

(44)

• Very few groups used React, or any front-end framework at all. As such, the final course module was fairly inapplicable for their projects.

5.2 Answering the Research Questions

5.2.1 RQ1: How to create an example application suit- able for learning about modern development en- vironments, and testing?

Which properties should the application have?

The application should be similar to the group projects in the course Our example application is built on the MERN -like stack, consisting of Node.js, Express, React, and Cloudant, as this stack was considered to be the standard for this course. A majority of the student projects used Node.js and Express.

The choice of database and front-end solution did however differ more. A few groups used React, but other groups were using server-side rendered pages, using very little application logic on the client. Many groups also had difficul- ties working with Cloudant, and chose to work with IBM:s DB2, a relational SQL database.

The application should be structured in a way that makes the code testable To make the example application easy to unit test, we chose a modular approach. The application server follows an object oriented paradigm, and uses a layered Model-View-Controller (MVC) architecture. The Express front-end was mainly used for handling HTTP requests, and making calls to a controller object. In the first seminar, one of the students asked about testing of the HTTP interface. In this student’s case, their application was heavily coupled, and the only provided interface for testing was the HTTP endpoints, which directly led to the introduction of API testing in the second seminar.

The application should meet standards and best practices Despite the small nature of the application itself, there are still several standards in ef- fect. The application uses a layered MVC-like architecture[34][35], and ad- heres to object oriented standards such as separation of concerns [36], low coupling[35], and high cohesion[35].

(45)

CHAPTER 5. DISCUSSION & CONCLUSION 37

The application should be easy to understand for a second year student at the TIDAB or TIEDB programme In the example application the focus lies on teaching rather than functionality. The module for time calculation, for example, is using integers as a time unit. The reason behind this decision was solely to avoid distraction from the main functionality.

Which tools and technologies should the application use?

The tools should be simple to use One of the main reasons that the Jest framework was chosen was its simplicity. An installation of Jest includes a testing framework, a test runner, and code coverage tools. Since Jest follows the Convention over configuration, very little or no configuration is required.

An Express server can be run with very few lines of code. The example application uses a MERN -like stack, and the entire application is written in JavaScript.

The application should use technologies that are useful outside of the scope of II1302 React is a very popular framework [37] and is widely used by companies as Facebook, Instagram, Netflix, and the New York Times. [38]

Also worth mentioning is that one can create Native mobile applications for iOS and Android with React syntax by using React Native [39]. The Jest framework is one of the most popular testing frameworks [40][41], and is as well as React developed by Facebook [42]. In a survey conducted by Katalon and KMS in 2018, 84% of software testers answered that they were using, or have used Selenium [43]. WebDriverJs is the official JavaScript implementation of Selenium [44], and was therefore chosen for GUI testing of the example application.

5.2.2 RQ2: How to use the example application to teach about development environments, and testing?

What content should be presented?

Theoretical background The students need a proper introduction of the subjects covered, to learn not only how, but also why, and when the testing types are being used. In each seminar the students are introduced to the type of testing covered, and which problems that are being addressed. The introduction also covers the part of the example application that is concerned in

(46)

the particular seminar. The students are also taught how the covered test types relates to known models, such as the test pyramid and the V-model.

Tools used The Learning modules themselves cover all the tools used in the example application, along with other similar tools intended for different frameworks.

In the first module the students are introduced to the MERN stack, Jest, and the basics of test automation.

The second module introduces few new tools, but does include an intro- duction to API testing using Supertest.

The third module focuses on the Selenium framework, as well as intro- duces the students to WebDriverJs, and other popular GUI testing tools for JavaScript.

The fourth module provides links to information about React and other front-end frameworks. Links are also provided to some of the most common tools used for building and testing a client-side application, such as Webpack, Babel, and Enzyme.

Testing strategies The relevant strategies within each of the covered fields of testing needs to be presented in order for the students to understand how to apply the tests in their specific projects. The learning module mentions the most common strategies used in each of the covered test types.

How should the content of the learning modules be pre- sented?

Seminars At the seminars an oral presentation is given to the students. An overview of the course module is presented. The visual material mostly consisted of slides, but some live coding examples and demonstrations of relevant web pages were also shown during the seminar. This also served as an oppor- tunity for the students to ask questions directly to the teachers. Demonstrations of various strategies in automation of tests, both locally and in the cloud were also given.

Source code Source code showing how to implement different testing strategies should be provided, to give the students a concrete view of the theory taught in the modules. In the learning modules code examples are provided, not only in the example application, but also in the compendiums and video walkthroughs, which are focused on implementing the taught material as code.

(47)

CHAPTER 5. DISCUSSION & CONCLUSION 39

Code examples are also shown in the seminars, in the form of live demos.

The source code for the example application and the recommended DevOps toolchain are also available in public Github repositories.

Lab compendium The written compendium was both intended as a substi- tute for the ones who couldn’t attend the seminar, as well as a reminder of the many concepts introduced and their connection during the seminar. More importantly, the written compendium was a chance for the students to have ready-made explained examples available, to be able to apply them in their own projects.

Video walk-through The video walkthrough was intended as a different alternative to the compendiums and seminars. The spoken nature allowed for a less formal explanation of the concepts, and the recording of a desktop environment could highlight the meaning behind individual lines of code more directly.

5.3 Portability

5.3.1 The example application

The application is designed with a Node runtime provided by a PaaS in mind.

This does not however bind the application to IBM Cloud, since similar Node runtimes are provided by several cloud service providers. While the example application is not designed for any specific back-end framework, it should be easy to implement in most of the MVC frameworks due to the simple im- plementation of that design pattern in the application. The use of Cloudant does bind the application to IBM Cloud to a certain degree, since Cloudant is an IBM service, but the Node client library for Cloudant is a wrapper for Nano, which is a library for Apache’s CouchDB. All the commands used in the application are Nano commands, except for those used for establishing the database connection. One configuration file, manifest.yml, contains the IBM Cloud-specific settings. The filename is quite generic and may conflict with configuration files for other cloud services.

5.3.2 The DevOps toolchain

The source code for the toolchain is written according to Open Toolchain’s YAML standard, and needs to be modified if another cloud solution is used.

(48)

Most of the tasks performed in the deployment pipeline, however, are implemented as bash scripts, and should easily be ported to any cloud service running a Linux environment. The deployment scripts are specific to Cloud Foundry, and needs to be modified if another deployment method is used. The memory optimization made in the deployment pipeline, is done with the Lite plan in mind. This does not bind the toolchain to IBM Cloud per se, but might be unnecessary if another cloud service is used, or if the conditions for the Lite plan are changed.

5.4 Reliability

In qualitative research, the subject of “reliability” refers to the extent that a measurement will yield the same results for repeated attempts [45][25]. In essence, will the same result be produced, if the same process is repeated by another party at another time?

5.4.1 User Group

There is a significant difference of knowledge and experience amongst the students themselves, which is difficult to account for. Besides this, there are also big variations in the student projects regarding architecture, functionality, programming languages used, and the choice of tools and frameworks. In this project the student interactions has been mainly with a group of test leaders, consisting of fourteen students in total. This small group of students makes it difficult to achieve a representative selection, as even the students in charge of testing for their respective projects, may not be raising the concerns of their peers. Studies on future courses needs to be done to improve the reliability.

5.5 Validity

Besides the concern of reliability, the subject of validity is also crucial to es- tablishing a degree of objectivity in qualitative studies[45]. “Validity” refers to the extent that a given procedure has provided a correct answer. Essentially, in the context of the research questions, has the research method answered the questions posed?

The creation of the example application itself, while threadbare, does provide an example suitable for teaching about testing. While the application is