Tools Integration Challenges Faced During DevOps Implementation

(1)

Master of Science in Software Engineering June 2021

Dharma Teja Bonda

Vishnuvardhan Reddy Ailuri

Faculty of Computing, Blekinge Institute of Technology, 371 79 Karlskrona, Sweden

(2)

This thesis is submitted to the Faculty of Computing at Blekinge Institute of Technology in partial fulfilment of the requirements for the degree of Master of Science in Software Engineering.

The thesis is equivalent to 20 weeks of full time studies.

The authors declare that they are the sole authors of this thesis and that they have not used any sources other than those listed in the bibliography and identified as references. They further declare that they have not submitted this thesis at any other institution to obtain a degree.

Contact Information:

Author(s):

Dharma Teja Bonda

E-mail: dhbo18@student.bth.se Vishnuvardhan Reddy Ailuri E-mail: viai18@student.bth.se

University advisor:

Dr. Ahmad Nauman Ghazi

Department of Software Engineering

Faculty of Computing Internet : www.bth.se

Blekinge Institute of Technology Phone : +46 455 38 50 00 SE–371 79 Karlskrona, Sweden Fax : +46 455 38 50 57

(3)

Abstract

Background. Since the conception of DevOps several tools have come into exis- tence that ease its implementation. A variety of these tools are used to implement a standard DevOps pipeline. The process of selection of these tools and the interac- tions between them is called tool integration.

Objectives. This thesis aims to address and solve the various tool integration challenges faced during the DevOps implementation. The primary goal is to identify the tool integration challenges that occur while implementing a standard DevOps pipeline with the help of feedback from experienced DevOps practitioners. After identifying these challenges, the possible mitigation strategies to these challenges are devised by analyzing the responses received from a large population of DevOps practitioners.

Methods. Survey is chosen as the research methods for this thesis. Two data collection methods have used for implementing the survey namely interviews and survey questionnaire. Major tool integration Challenges are identified from the an- alyzed interview data. Using these challenges as inputs a survey questionnaire is prepared for gaining insights on occurrences of these challenges in large population of DevOps practitioners. The main aim of the survey questionnaire is to find out the mitigation strategies to these challenges by analysing the survey responses.

Results. The tool integration challenges were identified by conducting semi-structured interviews with eight DevOps practitioners. Seven common challenges have been identified from the interview data using Immersion approach. A survey questionnaire prepared by using these challenges received 79 responses in total, out of which 62 are considered. Using Thematic Analysis the mitigation strategies for these challenges have been extracted from the responses of the survey.

Conclusions. Most of the tool integration challenges reported by the interview participants seem to be associated with CI tools when they are integrated and used together with other DevOps tools. Most of the mitigation strategies are aimed towards the "Tool" theme among the various themes found in thematic analysis of survey data.

Keywords: DevOps, Tools, Integration, Challenges, Automation

i

(4)

(5)

Acknowledgments

Firstly we would like to express our deepest gratitude to our supervisor, Dr. Ah- mad Nauman Ghazi. This thesis work would not not have been possible without his guidance and continuous support. We thank him for providing us with his valuable feedback and constant support which has helped us greatly to stay on the right track.

Secondly we would like to thank all the practitioners who took part in the interviews and the surveys conducted during this study. Without their valuable inputs and experiences this thesis couldn’t have been made possible.

Finally this thesis could not have been possible without the love and support from our family and friends. They provided us with their constant support and motivation during this pandemic which were crucial for us to complete this thesis. We would like to dedicate our success to all these people. Thank you Everyone!

iii

(6)

(7)

Contents

Abstract i

Acknowledgments iii

1 Introduction 1

1.1 Aims And Objectives . . . . 2

1.1.1 Aim . . . . 2

1.1.2 Objectives . . . . 3

1.2 Research questions . . . . 3

1.3 Outline . . . . 4

2 Background 5 2.1 DevOps . . . . 5

2.1.1 Definitions . . . . 5

2.1.2 Fundamental characteristics of DevOps . . . . 7

2.1.3 Achieving Automation Through Pipelines . . . . 7

2.1.4 DevOps Tools . . . . 8

3 Related Work 11 4 Method 13 4.1 Research Method Selection . . . 13

4.2 Research Design . . . 14

4.3 Data Collection Methods . . . 15

4.3.1 Interviews . . . 15

4.3.2 Survey Questionnaire . . . 20

4.4 Data Analysis . . . 22

4.4.1 Data analysis of Interviews . . . 22

4.4.2 Data Analysis of Survey Data . . . 23

4.5 Validity Threats . . . 24

4.5.1 Survey . . . 24

5 Interview Results and Analysis 27 5.1 Interview Responses . . . 27

5.2 Interview Analysis . . . 29 v

(8)

6 Survey Results and Analysis 33

6.1 Survey results . . . 33

6.1.1 Demographics . . . 33

6.1.2 Challenges and validation . . . 37

6.1.3 Mitigation . . . 37

6.1.4 Survey Analysis . . . 38

7 Discussion 43 7.1 RQ-1: What are the tool integration challenges identified during De- vOps implementation? . . . 43

7.2 RQ-2: What are the possible mitigation strategies devised for overcoming these tool integration challenges? . . . 45

8 Conclusions and Future Work 49 8.1 Conclusion . . . 49

8.2 Future Work . . . 50

A Survey Questionnaire 55

vi

(9)

List of Figures

4.1 Research Design . . . 14

6.1 Q-1.1 Distribution of Respondents’ ages . . . 34

6.2 Q-1.2 Gender of Respondents . . . 35

6.3 Q-1.4 Roles of Respondents . . . 35

6.4 Q-1.5 Total Work Experience . . . 36

6.5 Q-1.6 Experience in the field of DevOps . . . 36

6.6 Q-1.7 Size of the Organization . . . 37

6.7 Thematic map for challenges 1,2 and 3 . . . 39

6.8 Thematic map for challenges 4,5 and 6 . . . 41

6.9 Thematic map for challenge 7 . . . 42

A.1 Survey Questionnaire 1 . . . 55

vii

(10)

(11)

List of Tables

2.1 DevOps capabilities and enablers [1] . . . . 6

4.1 Interview Questions . . . 17

4.2 Description of semi-structured interviewees . . . 19

4.3 Survey Questionnaire Design . . . 22

5.1 Comments of Participants in the interview . . . 27

5.2 Tool Integration Challenges found from the interviews . . . 30

6.1 Themes . . . 38

7.1 Challenges and their mitigation strategies. . . 46

ix

(12)

(13)

Chapter 1

Introduction

DevOps practices are concerned with the collaboration between software developers and operational teams in order to improve the efficiency of system development process [2]. DevOps practices focus mainly on sharing the responsibilities amongst the team members and to automate the release process with the help of DevOps tools.

The DevOps tools are used in every phase of software development for various purposes such as Continuous Integration and Continuous Testing (tools like Jenkins, Travis, Codeship etc.), Cloud management and application deployment (tools like Puppet, Ansible, Chef, Terraform etc.), monitoring and management of recorded data (tools like AWS Cloud Watch, New Relic etc.), Version control management (tools like Github, Mercurial, Bitbucket, etc.), Database management (tools like MongoDB, MYSQL etc.), Team Communication (tools like Slack, Hipchat etc.) [3].

These tools are used for implementing DevOps and the process that involves the selection of these tools and how well these tools can be used together for constructing a standard pipeline is known as Tool integration. Hence proper tool integration could help business organizations and various companies to devise better strategies and take rational decisions [3, 4].

Tools are a vital part of the DevOps approach. They are the necessary aspects required for the DevOps approach to achieve automation [4]. They are closely linked to various DevOps practices such as automation, monitoring or integrated Configu- ration, and help to initiate these practices [5]. A set or a combination of these tools used for delivering, developing and managing the software applications throughout their entire development lifecycle is known as a DevOps Toolchain [6]. Toolchains are normally coordinated by a DevOps organization. DevOps tools are defined as tools used for achieving goals such as helping with collaborations of people from different divisions, facilitating continuous delivery and maintaining the reliability of software [7].

In DevOps tool integration is considered to be one of the most important aspect as it reduces considerable amount of time and work required for manually execut- ing, reporting and monitoring data [2]. Various DevOps tools are used to construct a standard pipeline for making the implementation of software development easy.

There are plenty of DevOps tools present for constructing a standard pipeline but the main challenge arises when we have to select few appropriate tools from the total

1

(14)

2 Chapter 1. Introduction available tools for implementing each of the DevOps phases [8]. It is essential for every DevOps organisation to choose the correct tools [9].

Organizations are met with difficulties in selecting the right tools due to the associated risks of tool maturity. When selecting tools for automation, the risks of replacing them also need to be taken into account. Therefore it is advised to use a setup where tools can be substituted and interchanged in order to solve this problem [10]. In DevOps, understanding the correlation of tools and concepts allows practitioners to take informed tooling decisions and thereby ensures that the technical, managerial, and organizational decisions are well aligned with each other [7]. Since DevOps depends heavily on automation tools selecting a proper toolset becomes a major challenges for DevOps practitioners. It is not possible for any individual or organisation to master all the DevOps tools due to the fact that a large number of them are available in the market [4, 11] and also due to their ever changing nature [12].

Challenges like these hinder the implementation of DevOps processes and makes the organisation skeptical about adopting the DevOps approach. Resolving these problems could lead to better implementation of DevOps in organisations which could further lead to an accelerated system development process. Various tools are used to support the different aspects of the development process. The selection of the software tool depends primarily on the product functionality and customer/user needs.

Problem Statement

Even after the process of selection, integration of these tools can sometimes prove to be problematic, thus increasing the difficulty to seamlessly integrate and maintain the pipelines [3]. In a deployment pipeline, the integration of tools must occur and cannot be avoided. Integrating several different tools for this purpose is a significant challenge [10, 13–15] because of the lack of compatibility between those tools [14, 15]

and their different levels of maturity [10, 16, 17]. The need for using several different DevOps tools for building a pipeline may lead the practitioners into a problematic situation where several team members or multiple teams use different tools in the same pipeline. This notion has been termed as being an anti-pattern for the very foundation of DevOps [10]. Hence the DevOps practitioners working with multiple tools may encounter these challenges in their daily work. This problem is our primary motivation for conducting this study. Our study focuses on finding and mitigating real-world tool integration challenges personally faced by different DevOps practitioners working in the industry.

1.1 Aims And Objectives

1.1.1 Aim

The main aim of this research is to address and find workarounds for the tools integration problems faced by DevOps practitioners during the DevOps implementation.

(15)

1.2. Research questions 3 For accomplishing this we have chosen to conduct interviews followed by a survey questionnaire to know more about the tool integration process. Interviews consist of questions that inquire about the experiences or difficulties faced by the different practitioners of DevOps while integrating tools during DevOps implementation. Us- ing the found challenges as input a survey questionnaire is prepared for the purpose of finding the mitigation strategies to these challenges. From the responses received in the survey we intend to figure out which of these challenges pose more of a threat to the DevOps practitioners along with the tool categories which cause these problems.

1.1.2 Objectives

The main objectives of this study are :

1. Identifying the tools integration challenges faced by DevOps practitioners working in industry.

2. Identification of possible mitigation strategies for overcoming these challenges in order to ease the construction of standard pipeline.

1.2 Research questions

The following research questions will be answered by our research in order to fulfill our aims and objectives:

• RQ1: What are the tool integration challenges identified during De- vOps implementation?

Motivation: A large number of challenges related to DevOps implementation have already been identified but the existing information of problems faced while integration of different tools and technologies is very limited and are not easily available because they vary between various organizations. We answer this research question by receiving input about the challenges from DevOps practitioners by conducting semi-structured interviews.

• RQ2: What are the possible mitigation strategies devised for overcoming these tool integration challenges?

Motivation: Different organizations implement DevOps in different ways but the aspect of tools integration and facing its challenges is common in all of them. Devising mitigation strategies to overcome these challenges could ease the process of DevOps implementation and make it more efficient. We answer this research question after taking the practitioners experience and suggestions about dealing with such challenges from the above mentioned RQ1. A survey

(16)

4 Chapter 1. Introduction questionnaire is implemented based on the challenges given by the practitioners to know more about their frequency of occurrence, source of cause as well methods to handle the mentioned challenge.

1.3 Outline

This thesis is divided into 8 chapters. Chapter 1 gives the introduction, aims and objectives of the study as well as the research questions which are to be answered.

Chapter 2 presents the background work performed for this study. Chapter 3 de- scribes the related work and similar studies. Chapter 4 discusses about the research methodology used in this study. Chapter 5 discusses about the interview results and interview analysis. Chapter 6 discusses the survey results. Chapter 7 presents the discussion section of the study and finally Chapter 8 concludes the study with conclusion and future work.

(17)

Chapter 2

Background

2.1 DevOps

2.1.1 Definitions

DevOps is a set of practices designed to reduce the time between a change to the system and a change to normal production, while at the same time ensuring high quality [18]. It is defined as an approach in which communication is enhanced in order to improve the quality of software and increase the frequency of releases in software organizations [19]. The DevOps approach facilitates a quicker, cheaper and simpler delivery of company value to the final customers of the company. This value may be releases, updates or enhancements to products more frequently [20]. DevOps reflects a multidisciplinary and collective initiative of the organization for automating the continuous delivery of new software updates all while ensuring the correctness and reliability of these new updates. DevOps provides the underlying infrastructure with smooth performance, availability, and stability of the applications, which are first developed, then tested and finally put into production.

DevOps provides several practices, tools and principles that help to ease the process of software development by improving collaboration between the teams and thereby bridges the gap between development and operations team, which is considered as one of the biggest challenges faced by any organisation [1, 21].

DevOps can also be defined as the set of capabilities that are tailored to fre- quent releases and deployment, development processes such as ongoing integration and testing, and continuous release and deployment. DevOps also involves having a collection of cultural enablers like trust, respect, shared responsibilities etc., which aim to break the working gap between development and operations teams. Finally, it provides a range of technical enablers required to achieve the above-mentioned capabilities, such as tools to automate the development, testing and delivery activities [1]. The summary of the capabilities and enablers given by Smeds et al. are presented in the below table 2.1.

5

(18)

6 Chapter 2. Background Table 2.1: DevOps capabilities and enablers [1]

Capabilities

Continuous planning

Collaborative and continuous development Continuous integration and testing

Continuous release and deployment

Continuous infrastructure monitoring and optimization Continuous user behaviour monitoring and feedback Service failure recovery without delay

Cultural Enablers

Shared goals, definition of success, incentives

Shared ways of working, responsibility, collective ownership Shared values, respect and trust

Constant, effortless communication Continuous experimentation and learning

Technological Enablers

Build automation Test automation

Deployment automation Monitoring automation Recovery automation Infrastructure automation

Configuration management for code and infrastructure

In the above table 2.1, according to Smeds et al., the term "continuous" refers to delivering small increments without delay whereas "automation" refers to providing proper tool support without manual intervention. The technological enablers are implemented based on the choice of the tool, configuration of the tool and design of the tool. The authors also state that DevOps isn’t effective without these cultural and technological enablers.

Apart from these terms some commonly used continuous practices implemented in DevOps as mentioned in Table 2.1 are:

• Continuous Integration(CI) : It consists of steps to manage changes to the source code of the developing software. Usually these modifications are merged with the remaining code of the software. Code inspection tools are also used to verify possible defects or measures with the help of quality metrics [22].

Unit testing is also performed for similar reasons. The code then needs to be built and tested with a number of acceptance tests. These actions are normally automated using suitable tools, but the continuous integration emphasizes that they are routinely carried out so that developers can get rapid feedback when the code changes are incorrect [23].

• Continuous Delivery (CD): It is the next logical stage following CI. It includes steps to build and automatically install a new version of the developed software within an environment, which may or may not be production environment [22, 23]. For example it can also be used for the purpose of testing [24]. Continuous delivery allows new applications to be released quickly and consistently, often many times a day.

(19)

2.1. DevOps 7

• Continuous Deployment (CDE): Similar to CD, it has to do with being able to install new releases quickly, except this time the new releases are directly installed in end users production environment [22, 23]. This indicates that a software company can never implement CDE without CD.

2.1.2 Fundamental characteristics of DevOps

According to Lwakatare the fundamental characteristics of DevOps are [25]:

• Culture: Devops promotes respect, mutual responsibility and a focus on open communication between the teams which carry out operations and development activities.

• Automation: Using suitable tools to automate tasks in the software development process which includes the integration of source codes, testing, delivery and deployment, and also to track changes to the software artifacts (configuration management).

• Measurement: The performance measurements of both operations and developers are focused on the same market value-driven measures (only software improvements that were integrated, tested, built up and deployed add value for the software user) [26, 27]. Significant metrics therefore have to relate to user needs and they must be the same for the operations and the development in order to motivate each and every role to align its objectives and also to cooperate more effectively.

• Collaboration: Improved collaboration is essential to facilitating successful practice of DevOps, so developers and operators can perform a range of software development tasks together such as writing deployment and running test scripts, as well as addressing emerging issues together.

• Monitoring: In order to avoid issues in future updates, anyone involved in the software project should also be involved in closely monitoring the production environments.

2.1.3 Achieving Automation Through Pipelines

Automation enables DevOps developers, operators, testers and other stakeholders to automate tasks performed while creating and deploying software [4, 28]. Automating tasks, such as integration, testing, building and delivering the software, eliminates delays and risks, as such activities are time consuming and error prone if performed manually [26]. In particular, producing a new release can be a complex task for many applications. It may also require fixing any unintended bugs, so that the new release can run properly. Such operations are difficult to carry out manually. Man- ual delivery and deployment of a new product version is error-prone and can lead to delays and increased costs [28]. These issues mostly impact the operations team, hence raising the likelihood of conflict between them and the development team [26].

For all these purposes, DevOps depends heavily on automation.

(20)

8 Chapter 2. Background

The deployment pipeline is one of the key elements of DevOps automation. It is also known as delivery pipeline [26]. Modifications to the source code have to go through this mechanism in order to be rendered available to end users [28]. It serves the purpose of bringing the developed software from version control to production environment. DevOps facilitates in optimizing the deployment pipeline by automating each and every phase of the pipeline to prevent delays throughout the development process [8]. As a result of this, DevOps brings about continuous practices like CI, CD, CDE and continuous monitoring [1]. Through the automation of the deployment pipeline by DevOps, the pipeline provides regular feedback in order to identify and easily correct possible errors. In addition, each release is constantly monitored by operators for expected performance, stability, security or other problems [25, 26, 29].

This will help the team to adapt to these challenges more effectively and prevent them in the future.

2.1.4 DevOps Tools

Many different tools offer the aspect of automation for the development processes such as source code management [30], build, testing and deployment. The term

"automation" stands for convenient tool support [1]. DevOps uses many of them readily, not only to eliminate manual labour and possible mistakes, but also as a way of receiving continuous input during the development process so that problems can be identified and resolved quickly. Secondly these tools facilitate improved communication and collaboration between the teams [31, 32]. Project automation software existed before the introduction of DevOps, but many of these software were renamed DevOps tools after DevOps came in [26].

From the related work that we have studied the DevOps tools are generally broadly divided into these categories:

1. Source Code Management (SCM) Tools: These tools are used to track changes in the evolving source code while in the software development process and enables the developers to work together by sharing their code and work from different locations. Version control tools are responsible for managing changes in the code and therefore regulate how the developers should collaborate during the coding phase. SCM tools are helpful for developers/teams who are working from multiple locations and have the need to contribute their code to a common place. GitHub is the most popular open source free-mium tool that supports the functionality of distributed systems. Other popular SCM tools: Mercurial, Bitbucket, Subversion etc.

2. Build Tools: Build tools are used to execute a program or a set of programs from the source codes of a particular software product. These tools convert the source code into machine understandable instructions as per the specific computer architecture involved. Examples of build tools are: Maven, Ant, Gradle etc.

(21)

2.1. DevOps 9 3. Continuous Integration (CI) Tools: These tools are helpful in automating the integration and merging of code and submits this code to common source code location of the currently developing software. This is done to ease the upcoming processes of building and testing. If any mishap occurs during the CI process, these tools immediately alert the developers by providing them with the relevant feedback. Widely used CI tools are Jenkins, TeamCity, Travis CI etc.

4. Configuration management tools: These tools track the changes to artifacts and are responsible for maintenance of different versions of several different artifacts. Hence their primary functionality is to establish and maintain the consistency of software products over their entire lifecycle. For example:

while implementing a new feature requested by the user, the developers have to track it during the whole of its journey i.e from being a requirement to a fully implemented feature in the final product. These tools are therefore used to track any related changes in the source code and also are responsible for implementing the tests written for that feature. Chef, Ansible and Puppet are few of the commonly used configuration management tools.

5. Cloud Tools: Cloud tools are used for integrating deployment as well as to collaborate in order to aid the DevOps practices. Since they support the DevOps practices, these tools are frequently used by the DevOps practitioners.

Popular cloud services are : Microsoft Azure, Amazon Web Services etc.

6. Testing Tools: Testing tools provide a testing as a service which plays an important role in the creation of automated DevOps pipeline. These tools help in the verification of quality of the software product and often involves combining the continuous testing aspect with automation. These tools are primarily used for improving the collaboration aspect along with maintaining the quality of the software product. Popular testing tools are Selenium, Cucumber etc.

7. Containers: A container is a standard software unit that bundles code with all its dependencies so that the application runs from one computing environment to another easily and reliably. Containers are used in the development of platforms and for the deployment of applications in the infrastructure. They help reduce the time gap between development and production. Containers are easy to maintain and deploy and hence are very useful for DevOps practitioners.

Popular container technologies are Docker, Kubernetes etc.

8. Deployment Tools: The tools are responsible for automated deployment of new releases of software. Whenever new changes are made to the software, these changes are run through a series of tests, which on passing allow the newer version of the software to be deployed automatically without manual effort. Jenkins, Ansible, Capistrano are few popular deployment tools.

9. Database Tools: These tools are responsible for storing and handling the data, procedures, schemas, metadata etc., and perform other database admin- istrative tasks. In addition to this Databases as Code (DbaC) can also be considered to be similar to source code, i.e. it also goes through the processes

(22)

10 Chapter 2. Background of Continuous Integration and Continuous Deployment. Examples include DB- Maestro, MongoDB etc.

10. Monitoring Tools: These tools are crucial for observing and solving the main infrastructure problems that might cause an hindrance to business such as RAM, Memory space, CPU load etc. These tools are used greatly helpful in maintaining cloud applications. Examples include Nagios, NewRelic, Cacti etc.

11. Collaboration Tools: These tools provide a means of communication and collaboration amongst the teams. In DevOps trust, communication and collaboration are key ingredients which help the teams to realise their roles, ideas and goals. Jira, Slack are few popular tools used for the purpose of collaboration.

(23)

Chapter 3

Related Work

Tool support is a necessary part of DevOps’s automation aspect [4]. The research study by Ebert et al., [4] has determined that the aspect of selection of tools is very important as the tools have to be suitable to the working environment(s) of the organisation(s). The authors provide a brief description of recent DevOps tools and technologies and how they are used in the industry. This study helped us to understand the significance of various DevOps tools and most importantly how the practitioners use various tools for performing different tasks in the industry.

The selection of appropriate tools for particular projects involves facing a lot of challenges [10]. Building a pipeline for serving particular purposes by integrating various tools has been determined to be a difficult task, since it involves identification of a group of tools that are applicable to the respective industry as well as choosing those tools from a wide variety of DevOps tools that are available. Compa- nies and organizations might have many constraints while choosing these tools such as checking whether the tool is easy to implement, cost effective, compatibility to the environment, maturity of the tool, etc. This study helped us to understand how different practitioners choose the DevOps tools according to their project requirement by making a judgement on the total effectiveness of the tools that are being considered.

A study by Amaradri et al., suggests that the process of tool integration required for the construction of pipeline can be quite complicated [13]. Companies generally use tools for implementing DevOps as they are helpful for mitigating challenges or solving the problems faced by the companies. Another challenge is encountered when adding new tools, i.e. it is important to verify if the new tools can be combined with the already existing tools. This causes changes in the workflow, to which the DevOps practices have already been adapted. This study helped us to understand the various challenges and their solutions faced by the industrial organisations in adopting De- vOps, in particular we focused on the DevOps tools problems faced by the DevOps teams during continuous integration deployment and testing. We referred to these challenges for framing the questions for the interviews as well as for the survey.

However, there seems to be a misconception that a single tool is enough to automate the entire delivery pipeline [8]. Typically, a variety of tools need to be used and integrated in order to achieve continuous pipeline. This creates the challenge of selecting apt tools for implementing the practices of DevOps.

11

(24)

12 Chapter 3. Related Work

The lack of expertise in the selection of the appropriate tools gives the developers and operators a challenge as it makes it difficult to implement such tools seamlessly in their production processes. This primarily occurs when the development and operations teams use different sets of tools [3]. Integration of several different tools used for implementing the same pipeline can be very challenging according to several authors [3, 5, 33, 34]. These studies have helped us in understanding the tool integration problems faced by the DevOps teams and how the synergy between the developers and the operations teams is crucial for implementing DevOps.

Even within the operations team, it is not always possible to use the same resources.

In particular, if a product’s software can run in different production environments, their specifications may be different and might require different delivery tools. This contributes towards the process being more complex in nature [1].

Based on the related work described earlier, we find that many studies were performed on tool-related challenges in DevOps. In contrast, there are very few studies conducted on the tools integration challenges while implementing DevOps. There- fore, when applying the DevOps approach, we try to recognize the challenges faced by the DevOps practitioners during the tools integration process, which is necessary for creating a typical DevOps pipeline and identify the approaches that can be pursued to mitigate these challenges. Studying the previous studies related to our research topic has helped us to understand the common challenges involved when using multiple tools while implementing DevOps. This knowledge has helped to create relevant questions for the interviews as well as for the survey questionnaire. Several challenges have been found from the interview and a survey questionnaire has been prepared using them. The responses obtained from the survey questionnaire helped us to identify the possible mitigation strategies. Some of these mitigation strategies have been confirmed to work by the survey respondents while others are probable solutions that could have a possibility of solving these challenges. Hence the identified solutions can possibly mitigate these tool integration challenges.

(25)

Chapter 4

Method

4.1 Research Method Selection

There are different research methods applicable to the field of software engineering but in general case studies [35], surveys [36] and experiments [37] are widely accepted empirical methods. We have chosen survey as our research method for this study.

Interviews and survey questionnaire are used as data extractions methods for implementing this survey method. Interviews are first conducted to identify the tool integration challenges faced by DevOps practitioners. The results of the interview are to answer RQ1. Based on the results obtained from the interviews we conduct a survey questionnaire to know about the insights of practitioners and their experiences while facing these challenges as well the mitigation strategies employed by them to resolve these challenges. The results obtained from the survey answer RQ2.

We decided to go with the interviews as they help us to record genuine challenges faced by relevant DevOps practitioners and these challenges help us to gain a insight on common tool integration problems faced by majority of them. A literature review was not opted for finding the challenges as there are very limited number of studies conducted on this topic and thus it is difficult to conduct a concrete literature review.

We felt that conducting interviews could fetch us genuine tool integration challenges faced by different DevOps practitioners in the real world. A survey is chosen as it is the most appropriate method to find out which of these challenges found in the interviews occur more frequently in a large sample population and how the participants could mitigate these challenges.

Survey is one such research method which is useful in collecting quantitative data from a wide ranging population [38]. The survey is selected as our main research method because it has a broader scope for the population as a result of which it collects vast amounts of significant data. A survey is generally used to define, compare and explain knowledge, actions and characteristics of the population by gathering valuable information or data [36]. We have chosen survey to answer both our research questions. Specifically we have used two data collection methods for the survey. Semi-structured interviews are the first data collection method which are used for answering RQ1 whereas a survey questionnaire is the second data collection method used for the purpose of answering RQ2.

The following are the reasons for not selecting experiment or case study as our research method:

13

(26)

14 Chapter 4. Method

• An Experiment is not suitable for this research because the factors that cause tool integration challenges are not known beforehand and thereby its implementation is halted [37]. Also results obtained from an experiment are sometimes not relevant as they are produced often in a unrealistic and controlled environment [39].

• Interviews conducted within a case study can also be used to find out the challenges in RQ1 but they would also require to work on a definite case afterwards.

This is particularly difficult for this study as DevOps is a new and evolving concept and it would take a lot of time and effort. For this research we felt that gathering larger number of inputs from relevant population could be beneficial in drawing an apt interpretation of the challenges found in the interviews and therefore a survey is much suitable for this purpose.

4.2 Research Design

This section explains the research structure and the approach taken to conduct the research is described in the following figure 4.1 .

Figure 4.1: Research Design

(27)

4.3. Data Collection Methods 15

4.3 Data Collection Methods

We have chosen two data collection methods for the survey research method namely semi-structured interviews and survey questionnaire. These data collection methods are used to answer research questions RQ1 and RQ2 respectively. Detailed structure and implementation of these data extraction methods are given below:

4.3.1 Interviews

Interviews have been selected as the appropriate data collection method for finding the tool integration challenges faced by DevOps practitioners during the implementation of DevOps. In interviews, the goal is to gather rich information about different elements, such as attributes, preferences, behavior, experiences, and perceptions of small number of respondents [40, 41]. In qualitative research, interviews are considered as the most effective data collection method [41].

Generally, a variety of questions which help to answer the research questions are devised and posed to the chosen respondents of interviews. These questions can be of two types: Open-ended or Close-ended. Open-ended questions allow a wide variety of responses from the respondents, while in the case of closed-ended questions, the respondents are able to select only from a small selection of answers. Interviews can further be categorized as unstructured interviews, semi-structured interviews and fully structured interviews [35].

In an unstructured interview, the interviewer formulates the interview questions on the basis of his general interests, and the discussion during the interview progresses forward on the basis of the interest of both the respondent and the researcher. Con- sequently, this style of interview primarily involves open-ended questions. When compared to unstructured interviews, in a fully structured interview, all questions are set in advance and are raised in the same order as formulated in the plan. There- fore, these questions are limited to closed-ended questions [35].

In a semi-structured interview, all questions are pre-set, similar to fully structured interviews, but they do not need to be answered in the same order as planned, as the interviewer has the freedom to decide on the order based on the progress of the conversation in the interview, while at the same time ensuring that all questions have been addressed. Semi-structured interviews primarily provide the interviewer with the opportunity to improve and investigate the field of study and encourages researchers to adjust questions appropriately. These interviews therefore contain both open-ended and closed-ended questions [35].

A questionnaire was prepared for conducting the interviews and it was modified according to the feedback received from supervisor. Research question RQ1 is answered by conducting interviews. The tool integration challenges obtained after analysing the data collected from interviews are used as inputs for making a survey.

Hence interviews also indirectly facilitate the answering of research question RQ2.

(28)

Participation Selection

DevOps practitioners from the operational and development teams of various software companies that implement the DevOps practices as a prerequisite are chosen as the participants of this research. These practitioners may use various DevOps tools during different phases of development as a part of their work.

Interview Design

We have chosen semi-structured interviews as the data collection method for our research. Semi-structured interviews are flexible as they allow the researcher to improve and explore the aspect under study as a result the researcher is able to change the questions according to the current context.

A set of open and closed questions are prepared prior to the interviews. This questionnaire reviewed by our supervisor and necessary changes were made as mentioned in the feedback from our supervisor. The questions are regarding the personal experiences and challenges faced by the industrial practitioners while integrating various DevOps tools. Semi-structured interviews are then conducted on the DevOps practitioners to get a clear understanding about the different tool integration related challenges and their mitigation techniques.

These interviews are conducted online via Skype, Zoom or over a phone call.

The participants are then asked for the permission to record their answers given to the questions in the questionnaire and in-case they disagree with this request, their answers are instead noted down by hand. Qualitative data is then collected from the interviews as they provide a deeper and richer understanding. The collected data from these interviews is further analysed.

The audio of all the participants in every interview are recorded in a convenient audio format. These recordings were later transcribed into text for the purpose of data analysis. The following guidelines have been followed while conducting interviews [35]:

• The purpose of the interview must be addressed by the interviewer and the respondents are made clear on how the data obtained from the interview will be used.

• After this, a series of introductory questions about the respondent’s role, background, etc., have to be posed to the respondents.

• If the interview involves personal/sensitive questions, caution must be taken and the anonymity/confidentiality of the respondents must be assured.

• Major findings can be outlined by the interviewer at the conclusion of the interview in order to eliminate misunderstandings and to gain feedback/further clarification from the respondents if necessary.

(29)

4.3. Data Collection Methods 17

• It is recommended that interview sessions are recorded in an acceptable audio or video format, so that they can be later transcribed into text to make sure that all the information is captured.

• Interviewees should be told before hand that the interview audio is being recorded.

• The interviewee must be informed of the interview and schedule the required date and time for the interview. The appointments must be arranged as early as possible to be flexible for adjustments.

Table 4.1 consists of the questions asked during the interviews along with their contexts. In this table some of the questions are mentioned to answer RQ2 along with RQ1. This is because the responses from the DevOps practitioners to these questions have helped us to know what to expect from the responses that are to be received through a survey. This is the reason why we have mentioned that some of the questions also help in answering RQ2 but their primary goal is answering RQ1.

Table 4.1: Interview Questions

ID Interview Question Context of the ques-

tion

RQ’s helped An- swered Q1 Could you please state your background -

your name, role and work experience? General background of the interviewee.

Q2 What is DevOps to you and your organi-

zation? Regarding the views of

Interviewee on DevOps andhow DevOps practices af- fect their work in the industry.

Q3 What tools do you use as part of DevOps? The tools they use to construct a standard De- vOps pipeline.

RQ1

Q4 What is the need of tool integration in De-

vOps as per you or your organization? The views of the interviewees on the importance of tool

integration and proper tool selection.

RQ1

Q5 What problems do you face during tool integration while working with multiple tools?

The challenges faced by the interviewees while integrating

various tools when constructing a DevOps pipeline.

RQ1

(30)

Q6 How do you go about solving those prob-

lems? The strategies used by

the interviewees to solve the challenges

encountered in the above question.

RQ1,RQ2

Q7 Which of those problems are you able to

successfully solve? The amount of these

problems that they can solve in-order to

resolve the issues.

RQ1,RQ2

Q8 What about the problems that you are not able to solve? How do you deal with them?

The amount of the problems that go unsolved and the strategies

they employ to deal with them.

RQ1,RQ2

Q9 Do you have any intuition why could the above problems be occurring and how could they be fixed/or avoided?

Factors or causes due to which the problems occur in the first place

as per the interviewee.

After describing these causes we ask them about the mitigation strategies that they have employed to fix/avoid the

situation?

RQ1,RQ2

Q10 Any other comments you would like to add or suggestions/improvements to the tools you would like to suggest?

Certain features or improvements that the interviewees would like to seein the already existing tools or emerging new tools.

RQ1,RQ2

After the questionnaire was finalized, we went ahead and interviewed 8 DevOps practitioners belonging to different companies. These interviews were conducted by either phone call or over video conferencing apps like Skype or Zoom. The interviewees roles, medium of the interview, duration etc., are presented clearly in Table 4.2. From the table 4.2 it can be clearly seen that participants with experiences ranging from more than one year to more than five years have been contacted for this interview. This is done so that the study is relevant with the DevOps practitioners who use the current DevOps tools and technologies in the current scenario. We decided to analyze the current scenario as DevOps tools continuously keep evolving and changing, therefore it is important to stay relevant with the current standards.

Another reason can also be the influx of large number of newer DevOps tools that are becoming available in the market so the older/deprecated tools might no longer be valid or viable. Different DevOps practitioners use different tools according to

(31)

4.3. Data Collection Methods 19 their industry sector/environment. Even though the tools are different, they all can be categorized within the 12 categories of Devops tools mentioned in section 2.1.4.

Therefore the tool categories of the DevOps tools along with the name of the specific tools used by different practitioners that cause the problems have been focused and discussed during the interviews. It was important for us to collect diverse information from different practitioners with different experiences as we had to conduct a survey by using the responses from interviews as inputs. This was because we were hoping to find interesting results from the survey.The main motivation was to see how the in-puts gathered from interview a small group of diverse practitioners could be represented by a larger population of DevOps practitioners across the world.

Table 4.2: Description of semi-structured interviewees

Interviewee Role Duration Experience Interview Medium

1 DevOps Engineer 30 mins 2+ Years Skype

2 DevOps Engineer 25 mins 1+ Years Phone Call

5 DevOps Manager 25 mins 5+ Years Zoom

6 DevOps Engineer 28 mins 1+ Years Zoom

8 DevOps Manager 30 mins 4+ Years Zoom

Other Research Activities

We have also performed several other activities in addition to conducting the actual interview. These are:

• Scheduling with the participants Interview participants were carefully chosen on the basis of their role, experience, company/organization. For the interviews only participants with at least 1 year of experience in the role of developer or operations and those who are employed in a DevOps implementing company or organisation, ideally multi-national, were chosen.

• Background information collection The background information of participants and their organisation are collected before the interviews. This is done to verify if all the questions would be relevant for them to answer.

• Studying Interview guides Papers written by several researchers, which contain the guidelines for conducting interviews are studied and we tried our best to adhere to these guidelines.

• Transcribing Data After conducting the interviews the collected audio recordings are transcribed. It takes about 8 hours to transcribe one hour of audio recording according to [42]

• Summary Writing After transcribing the audio recordings, the transcribed data is then summarized which makes it easier to fit the data to answer the

(32)

20 Chapter 4. Method research questions. It takes about 3-4 hours to summarize one hour of audio recording according to [42]

These activities have been performed right after we gathered responses from all the participants. The reason for not performing these activities after each interview is because these activities are time consuming and require a lot of effort. So we felt that assigning time and effort for all these tasks after conducting each interview is the most optimum choice.

4.3.2 Survey Questionnaire

By following the guidelines given by Singer et al. [43] we conduct the survey in the following six steps: objectives, sampling and population, Design of the survey, Validation of Survey, Data collection and Data analysis [44].

Defining the survey objectives

The survey objective is established clearly to attain the expected results of the study [45]. This survey aims to find out the various tool integration challenges faced by the DevOps practitioners as well as how the practitioners deal with such challenges.

This survey’s importance is to help us to understand the difficulties experienced by different DevOps practitioners while integrating certain categories of DevOps tools and also to know how certain measures can be taken to overcome these problems.

Population and Sampling

The target population for this survey are DevOps practitioners working in the industry such as DevOps engineers, DevOps managers, DevOps analysts, System admin- istrators and CICD engineers from various organisations. They have been selected as the target audience because generally they work with a lot of DevOps tools and have the best insights on challenges that occur in the DevOps implementation process.

The sampling method is the process of selection of a subset of respondents from the target population [46]. In order to minimize the time and effort needed to complete the analysis, we use a non-probabilistic convenience sampling method. There are many sampling methods available such as Simple Random sampling, Cluster sampling, Systematic sampling, Stratified random sampling, Multistage sampling - in case of probabilistic sampling methods whereas the non probabilistic sampling methods are Quota sampling, Dimensional sampling, Convenience sampling, Purpo- sive sampling and Snowball sampling [47]. The time period available for us to conduct the survey study resulted in selecting non-probabilistic convenience sampling technique. This technique is effective when the targeted audience is well defined, but contacting them within a given timeline is difficult [48]. Convenience sampling was used to send out the online questionnaire to interested participants in the first round who were approached using social media. Then Snowball sampling technique is used in the second round to maximize the size of the collected data. This was done by requesting Participants In the first round of the survey to refer possible participants for the second round.

(33)

4.3. Data Collection Methods 21 Survey Design

The survey questionnaire is designed in this step. For implementing this step an online questionnaire is created using Google forms. The questionnaire would consist of both close-ended and open-ended questions. For answering the close-ended questions the respondents have to choose from the given options where-as the open-ended questions have to be usually answered in a sentence or a paragraph. The survey is structured so that the whole population is represented by the sample and the questionnaire is designed to receive results in a fair period of time. For implementing this step an online questionnaire is created using the popular survey tool Google forms.

The survey is designed in such a way that it satisfies all the above objectives.

The survey questionnaire consist of both close-ended and open-ended questions [48]. For answering the close-ended questions the respondents have to choose from the given options where-as the open-ended questions have to be usually answered in a sentence or a paragraph. Various scales such as Likert scale and multiple options have been used for closed-ended questions and the collected data is ordinal in nature. Some of the questions are open-ended and they help us to understand the justification or underlying opinion of the respondent on the subject being discussed in the question.

Table 4.3 shows the implemented survey questionnaire design. The actual Survey Questionnaire can be seen from figures A.1-A.8 in the appendix section.

Evaluating the Questionnaire

The survey questionnaire is assessed on the basis of the fact that all the survey objectives are covered. The survey questionnaire was first designed and then modified as per the changes suggested by our supervisor. The questions in the survey are constructed in a way that they are easy to understand and are written in simple understandable language.

The questionnaire is divided into three section namely: Background, Cate- gories of tools causing integration challenges and their occurrence and Mitigation strategies employed to deal with these challenges. The first section i.e. Background involves the questions related to Demographics and work background of the respondent. This assists in determining the quality of the survey results. It is also used to assess the credibility of the responses to the survey given by the respondents. The second section involves establishing the objectives of the survey and collect valuable information from the respondent regarding the categories of De- vOps tools that cause tool integration challenges obtained from the interviews along with their experiences while using these tools. The final section involves questions regarding the mitigation strategies to these challenges employed by the respondents and concludes the survey with the insights of respondent. The survey questionnaire has 19 questions in total and the average time taken to complete the survey is about 13 minutes.