Efficient parallel installation of software collections on a PaaS

(1)

Efficient parallel installation of software collections on a PaaS

Alexander Boraie

Computer Science and Engineering, master's level 2021

Luleå University of Technology

Department of Computer Science, Electrical and Space Engineering

(2)

Abstract

This master thesis analyses and investigates how to speed up the deployment of a suite of services to a Platform as a Service. The project use IBM’s Cloud Pak for Applications together with Red Hats OpenShift to provide insights on important factors that influences the deployment process. In this thesis, a modification was done on the installer such that the deployment instructions were sent in parallel instead of sequentially. Except for the parallel suggestion, this thesis also investigates different options on how constraints could be applied to the CPU and what the consequences are. At the end of this report, the reader will also see how the deployment times are affected by cluster scaling.

An implementation of the parallel deployment showed that the installation time of Cloud Pak for Applications could be decreased. It was also shown that the CPU was not utilized fully and that there exists significant CPU saturation during the deployment. The evaluation of the scaling analysis showed that, in regards of this thesis, it is more beneficial both timewise and cost-wise to scale horizontally rather than vertically.

(3)

Preface

I want to express my deepest gratitude to Nils Bergquist and Peter Forsberg at IBM Svenska for mentoring me in this master thesis. Throughout this thesis they provided me with insights, knowledge, guidance and much more. I also want to thank my internal supervisor, Olov Schel´en for his guidance and support. Lastly, I want to thank my good friend and former colleague Anders Lagerkvist, for his insights, discussions, and encouragements during this thesis.

(4)

1 Introduction

For reasons such as scalability, availability, and efficiency the cloud infrastructure has become a necessity for many of today’s business. One of IBM’s ways to help with the cloud transformation is by offering IBM Cloud Paks. Cloud Paks are collections of services and it is built upon the Kubernetes-based platform OpenShift [22].

Since containers gained popularity it has changed the way we look at application development, deployment, and maintenance. Containers are based on OS-level virtualization and together with the current state of cloud computing it has brought an evolution on how businesses handle their IT.

The popularity of containerized software and container orchestration is still increasing, and all major cloud providers are now offering alternatives, either Kubernetes directly or platforms built around it.

The maturity of containers and the different technologies around it has been progressing throughout the years and extensive studies have been conducted to ensure that performance, isolation, and availability expectations are met.

Research about virtualization techniques have been made to ensure that minimal overhead is introduced and compared to the previous de-facto standard Virtual Machines, containers offer many lucrative alternatives. Since management of containers without an orchestration platform would be complex, it is often a necessity to include management tooling whenever a solution is built upon containers. Thus, the popular orchestration platform Kubernetes has been forced to follow the increased needs and requirements the community has.

Even though the maturity-level for both containers and Kubernetes is high it is still equally important that it is used correctly. When large software solutions are built/deployed on Kubernetes it is of high importance that all components interact correctly such that containers and Kubernetes can deliver intended benefits.

The focus of this master thesis project is to identify and analyze the process when a large software collection is deployed on a Platform as a Service (PaaS). The chosen platform for this thesis is Red Hat’s OpenShift Container Platform (OCP) which uses Kubernetes as an orchestration tool.

The chosen software collection is IBM Cloud Pak for Application. In this thesis modifications are done to the original installer such that the installation instructions are sent in parallel to the cluster. Together with the new installer, suggestions for concepts and methodologies are highlighted and tested to provide insights in mechanics that could affect the computing resources during the installation.

1.1 Background

IBM provides suits called Cloud Paks, these are containerized software solutions which intends to help moving business applications to the cloud. Currently there are six different suits available, namely: Applications, Data, Integration, Automation, Multicloud Management and Security. These Cloud Paks were initially developed independently of each other and therefore a cohesive deployment model has not yet been defined. The fact that different businesses have different needs there is a demand for multiple solutions. However, each Cloud Pak still need to cover a lot of features to allow a smooth transformation to the cloud.

Since a lot of research and optimizations have been made on containerization techniques and orchestration platforms it can be expected that organizations and businesses enforce the best practices

(7)

and state of the art solutions when utilizing these technologies. However, how do they ensure that these technologies are optimally used when the product is original and innovative?

1.2 Motivation

Today, almost everything is optimized. This is especially true in the IT industry. Optimization has very few drawbacks and can bring benefits for both vendors and the customers. Yet, optimization may not be the priority when a product is developed since other factors such as time-to-market, sustainability, or robustness are often more important.

This master thesis does not only give insights to IBM, but also to users, administrators, and developers. On IBM’s behalf optimization will not only provide a more professional feeling from the product but also saving resources on servers in the cloud, independent on if it is bare metal or virtual servers. It is also important to remember that a small saving of resources for a single installation may not be very thrilling it can be of great significance if that small saving happens every time, especially for a company with IBM’s size. From the developers and cluster administrator’s perspective this project will provide insights and an analysis on what happens when a suite is deployed on Kubernetes, that is something that is not trivial today. Except that, it also provides a method and a work-process for how to analyze a clusters behavior. Not to forget, from the user’s perspective the most obvious benefit is that the installation takes a shorter time.

It should also be emphasized that even if this thesis focuses mainly on the installation, many of the outcomes are also applicable to the scenario where batches of software are deployed to a PaaS.

Thus, even if the installation often is a one-time process, updates of multiple software in a cluster is a more often occurring event and therefore extends the scope of the aforementioned benefits.

Another important aspect of this thesis is that if a system needs to be critically re-installed due to for example system failures, upgrade failures and so on. It can be of high importance that the installation is as fast as possible to minimize the downtime the businesses experiences.

This thesis is also important in the aspect that it applies a well-known computing technique, parallelism, into an existing solution and shows the consequences.

1.3 Problem Definition

The problem definition for this master thesis is: How can the installation of a software collection on a PaaS be done in an optimal fashion? The software suite will be analyzed to provide insights on how it is delivered to OpenShift and how the packaging is affecting the installation in regards of time and system resources. It will also be investigated if a modification of the installer allows for a parallel installation process to OpenShift instead of the existing sequential installation process. To make such a modification, knowledge about both the current installation architecture and internal dependencies in the installer is needed. This thesis also covers the task of identifying bottlenecks and improvements that could benefit the installation. Thus, to solve the main problem it is required to find state of the art mechanics in container-orchestration tools and apply them to this case. These mechanics will then be tested and evaluated to see if the installation time and usage of system resources could be reduced. Another important factor that also is investigated is how vertical and horizontal scaling of a cluster affects the installation

(8)

1.4 Delimitations

To get an exact model for how the system resources behaves would be very hard. There is a lot of factors involved in a Kubernetes setup and the variables that the Cloud Pak infrastructure brings is not negligible either. In this project, selected factors have been chosen to be tested, analyzed, and implemented but that is far from all factors that possibly could improve the installation time further. Another important aspect is that Kubernetes is a project under very fast development.

Thus, new features are frequently released.

An important factor regarding the tests and CPU analysis done in this thesis is that the environment is located in the cloud. This means that the environment is depending on the servers on IBM cloud and factors such as load on the host machine is not available and must be assumed to be consistent.

An alternative approach to this would be that OpenShift was deployed on a local isolated machine, however, the capacity and flexibility would be both costly and hard to maintain. At the beginning of the project efforts were made to setup OpenShift locally with options such as Code-Ready Containers and OpenShift with the help of Vagrant to create worker nodes. Even if the Cloud Pak installation worked in these environments there was a lack of consistency and the computing resources were not able to meet the requirements by this thesis.

Cloud Paks are constantly under development and a new version of Cloud Pak for Applications has been released since this master thesis started. This project has centered on Cloud Pak for Applications version 4.2.1 and any new features in new version has been overlooked.

Other delimitations are that some other resources or bottlenecks may not have been analyzed, for example I/O and network throughput. It should be mentioned that data for these are included in the OpenShift monitoring stack and did not show any sign of problems or possibilities for improvement at a first glance, however, these are still factors that possibly could be tuned and optimized to minimize the installation time of Cloud Pak for Applications.

1.5 Thesis Structure

Section 2 cover the basics but also important concepts in containers, Kubernetes, OpenShift and features in Linux. It also addresses how the installation of Cloud Pak for Applications is done and presents the internal dependencies inside the installer.

Section 3 describes the modified installer for Cloud Pak for Application. This implementation is a parallel version of the original installer and the section also covers decisions that were made for the new installer. Section 3 will also define methods for how the CPU availability could be modified.

Except the implementation and methods defined here, a developed tool that gathers statistics from the cluster and generates graphs for CPU usage for both pods and nodes is described.

Section 4 evaluates the new parallel installer. The parallel installer will also be tested with the four different CPU constraints methods from the previous section. This section also displays how the installation times and CPU resources are behaving when the cluster is scaled both horizontally and vertically. Throughout this section results and comparisons are made between the different configurations.

Section 5 reflects over the presented results as well as what could have been done differently and problems that occurred during this thesis. Section 6 describes the outcome as well as the impact of this thesis.

(9)

2 Theory

2.1 Containers

In short, containers are executable software units that can include application code, libraries, and dependencies[7]. Containers are virtualized on OS-level while Virtual Machines (VMs) are, on the contrary, virtualized on hardware level. This also means that VMs are running a guest operating system (with the help of a hypervisor) on top of the host operating system while containers run the software unit on top of a container runtime. This is further illustrated in the figure below:

App 1 App 2

Bins/Libs Bins/Libs1

Guest OS Guest OS

Hypervisor

Host OS

Infrastructure Virtual Machines

App 1 App 2

Bins/Libs Bins/Libs1

Container Runtime

Host OS

Infrastructure Containers

Figure 1: Comparison between Virtual Machines and containers.

As seen in figure 1, VMs are virtualized with the help of a hypervisor. The hypervisor will virtualize the physical hardware to the guest OS and this will, of course, introduce overhead. Containers, on the contrary, are using the container runtime which will be discussed further for the purpose of this master thesis. Containers can exist in two different states: a resting state and a running state.

The resting state is also known as a container image. The container images are, according to an article on Red Hat, defined by the Open Container Initiative[37]. The running state is essentially a container running. From an end-user’s view, it is the container engine that runs the container.

However, the container engine actually does not run the container, but it has many important responsibilities like handles user input (either from the end-user or a container orchestrator) or

(10)

handling the container image. The actual component that runs the container is the container runtime. To understand container runtimes, it is first important to notice that containers today are heavily built upon standards. There are many different standards depending on the container stack. According to a presentation by Scott McCarty there are three major organizations that are contributing to the container standards, these are[35]:

• Open Containers Initiative (OCI)

• Container Runtime Interface (CRI)

• Container Network Interface (CNI)

As described by McCarty, the containers workflow from image to process is:

1. With a tool (that is OCI compliant) the user can create a container image. This container image consists of a metadata JSON file and tar files that represents the different image layers.

2. A container engine will prepare the metadata for the container runtime and it will also unpack the tar files from step 1 into a root file system.

3. The container runtime (that also is OCI compliant) will consume the files from step 2 and instruct the kernel to run the container.

The creation of a container is according to McCarty created with the clone() (create a child process) system call[34][5]. The process that originates from the clone() system call will be setup in a kernel namespace. Namespaces are a feature of Linux that partitions resources for processes. For example, namespaces can partition hostnames, IPs, and PIDs to different processes. The container runtime will also initiate the cgroups, SELinux etc.

Containers has become a somewhat de-facto standard and according to the paper Virtualization vs Containerization to support PaaS a common way for PaaS vendors to ensure isolation, lifecycle management and fair CPU sharing a setup where applications runs in containers that then runs on top of VMs are used[6]. According to the same paper, the authors state that the performance is better for containers compared to VMs. This would mainly be because VMs need to translate the machine instructions from Guest to Host OS in contrast to containers where the performance can be compared to the underlying Host OS. An article from VMware confirms this but it also mentions other aspects to the container vs VM debate[44]. Some of the negative aspects of containers according to VMware are:

• Containers need to run on the same OS (if specific Linux or distro features are used).

• May be less secure than VM.

• Containers are relatively new. It is still evolving.

However, in the paper An Updated Performance Comparison of Virtual Machines and Linux Con- tainers the authors investigate the overhead introduced for both VMs and containers[9]. The authors discuss that Docker features such as NAT and layered images will come with a performance cost. They also claim that by using Docker with the default settings it may actually be as slow as KVM (Kernel Virtual Machine)[31]. The authors also claim that there is a trade-off between ease of management and performance (as in the case with Docker and the respective features). It should be noted that the result from the paper is that, in general, containers almost have an identical

(11)

performance as native computing. But nevertheless, there exists cases where the container engine setup needs to be configured and analyzed.

In summary, it seems that containers are a big part of the future and it is taking a direction to ensure that containers are future proof.

2.2 Kubernetes

Kubernetes (K8s) is an orchestration system for containers, that means that Kubernetes is used to manage containerized workload and services[29]. According to Kubernetes website these are some of the key-features of K8s:

• Service Discovery and load balancing - K8s is able to distribute traffic to containers that are capable to handle it, for example, if two containers are running the same application, K8s is able to redirect traffic to the container that has the most resources available.

• Storage orchestration - Essentially helps with storage configuration such as mounting.

• Automated rollouts and rollback - K8s helps with maintaining a desired state. Thus, a desired state is given to K8s and K8s will try to achieve that state all by itself.

• Automatic bin packing - By specifying resource limits to K8s, K8s will by itself place containers on the most fitting nodes.

• Self-healing - K8s both monitors and control that the containers are healthy. For example, if a container dies/shutdown, K8s will try to restart it.

• Secret and configuration management - K8s use secrets and configmaps to reliable store sensitive information and to easy configuration management.

To be able understand the analysis of the master thesis project a understanding of how Kubernetes operates is necessary. The following section describes shortly the different Kubernetes components and the different terminology used in Kubernetes:

2.2.1 Terminology and Components in Kubernetes

The following are important terminology that is used for Kubernetes:

• Nodes - A node is a machine that is able to run the containerized software. It is common by the Cloud vendors to provide a virtual machine as a node.

• Cluster - A cluster is the collection of all Kubernetes components, including all the worker nodes.

• Pods - Pods represents the containerized process running on a node. A node can have several pods on them. A pod can contain multiple containers.

• Control Plane - is a name of the collection for the components that makes decisions, de- tecting or responding to anything related to the cluster.

To understand how Kubernetes works and operates the following components are shortly described here, for further information visit Kubernetes website[29]:

(12)

• kube-apiserver - exposes the Kubernetes API to the outside world. The kube-apiserver are responsible for the communication between Kubernetes objects and external users. This is achieved with a REST API. For example, it is via the kube-apiserver that a user can request information about the state of a running pod.

• etcd - is a key value store [8]. It is used by Kubernetes as a distributed database and thus, K8s can store cluster configurations in etcd.

• kube-scheduler - is a scheduler which decide at which nodes the pods should be placed at.

• kube-controller-manager - runs the controller process. The controller process is checking that the desired states are met and if not, make the respective changes to meet the desired state.

• cloud-controller-manager - is used to allow Kubernetes to be compatible with a cloud provider. For example, the cloud-controller-manager may be used for setting up new routing on a cloud provider.

• kubelet - is an agent which makes sure that containers are running on pods. Thus, there exists a kubelet on each node.

• kube-proxy - is used to expose the applications as network services. There exists a kube- proxy on each node in the cluster.

2.3 Operators

In 2006 a blog post from CoreOS introduced Operators[38]. Operators were meant to ease the packaging, management, and deployment of applications in Kubernetes[3]. This is done by taking Kubernetes declarative model and apply it directly to applications. Operators use Kubernetes Custom Resources (CR) to manage the application, CRs enables extension of the Kubernetes API, thus new endpoints are created which are coupled directly to an application specific object[27]. To monitor and maintaining the application (as defined by the CR) operators also adds a control plane component which understand how to interact with Kubernetes in such a way that the desired state is achieved.

Operator Lifecycle Manager (OLM) is a tool (that is included on OpenShift on IBM Cloud) that ease the installation and management of operators in a Kubernetes cluster[10]. OLM also installs and manages operators’ dependencies if needed. One of the most noticeable features from OLM in OpenShift is the feature that cluster users can select to install Operators from catalogs.

2.4 CRI-O

According to CRI-O’s website CRI-O is an implementation of the Kubernetes CRI[4]. CRI stands for Container Runtime Interface and is a plugin interface to allow kubelet to use different container runtimes[1]. Before CRI existed, container runtimes were implemented with the help of internal interfaces in kubelet[26]. Thus, maintaining and integrating new container runtimes into Kubernetes required deep knowledge of both kubelet and Kubernetes.

To provide monitoring for each container a program named conmon is used[2]. Since CRI-O is built upon CRI and OCI compliant runtimes, conmon can monitor any of the runtimes used by

(13)

CRI-O. CRI-O makes it possible for Kubernetes to use any OCI compliant container runtime. The documentation on the CRI-O website describes how a pod gets up and running:

1. Kubernetes tells kubelet to launch a pod.

2. Kubelet forward the request to CRI-O via the CRI.

3. CRI-O pulls the image from a registry.

4. The image is unpacked into a “Copy-on-Write” filesystem (this is the containers root filesystem).

5. CRI-O generates OCI runtime specification in a JSON format. The JSON file tells how to run the container.

6. CRI-O launches the runtime.

7. All containers are monitored by a conmon process.

8. Networking are setup with CNI (CRI-O allows any CNI plugin to be used).

2.5 Control Groups

Control groups (cgroups) is a kernel feature which helps with allocating resources for processes[16][32].

cgroups can allocate resources such as CPU time, memory, network among different process. By using cgroups there is a possibility to control the process in regards of prioritization, managing, monitoring, and denying resources. This is a core feature for containerization since this allows isolation, but it also allows the processes to run more efficiently.

The cgroups feature allows processes to be organized into different groups and for those groups set specific properties (controls, limits etc.). The processes are structured into hierarchical groups and it can exist multiple cgroups on the same Linux system. Resource controllers (subsystems) are the components that handles the resource asset for the processes and represents, for example, the amount of time a process can use the CPU. Each subsystem (for example like CPU, memory, I/O) has its own tree (hierarchy) and each process on the Linux system belongs to one tree node in each hierarchy.

For example, if an application, such as Firefox is in the root position in a hierarchy and its child- processes are children in the tree. Each level in the tree can have different limits defined. If this hierarchy is attached to the memory resource controller it will limit how much memory Firefox can utilize. Because it is a hierarchic structure, properties from the tree can be effective in the sub-hierarchy as well.

In Linux, the first process created is the systemd init process. Thus, all other processes are child processes of that. There exists three unit types (provided by systemd) that are used for resource control. One of these unit types is the Scope unit type[12]. The Scope unit represent processes that are created with the fork() function. This is important since containers are created with the clone() (which is similar to the fork()) function and thus, has this unit type.

The most important subsystem for this master thesis is the CPU subsystem. The CPU subsystem has the responsibility to schedule and delegate CPU resources to cgroups[13]. The CPU resources is (in the case of OpenShift on IBM Cloud) using the Completely Fair Scheduler (CFS) to schedule

(14)

CPU resources[25]. CFS will schedule CPU time based on tunable parameters for cgroups. The following list describes the important parameters for this project:

• cpu.cfs period us – The value for this parameter specifies how often the scheduler should reallocate CPU to a cgroup. The cfs period us value is in microseconds and combined with the cfs quota us determines how much CPU resources tasks in a cgroup can utilize.

• cpu.cfs quota us – The value specifies for how long time tasks in this cgroup can run in a specific period (the period specified by cfs period us). If tasks in the cgroup already exceeded the amount of CPU time allowed by the cfs period us and cfs quota us they are denied any more CPU time until the new period starts. The default value for cfs quota us is -1 and that indicates that there is no CPU time restriction for that cgroup.

To illustrate how cfs period us and cfs quota us interacts the following example is given: If tasks in a cgroup is allowed to use the CPU for 0.5 seconds out of every 1 sec the cfs period us is set to 1000000 and cfs quota us is set to 500000. However, if the tasks want to use 0.6 seconds, they will get 0.5 seconds of CPU time and then have to wait 0.5 seconds to get access to the remaining 0.1 seconds.

Another important parameter from the CPU subsystem is the cpu.stat parameter. This is a read- only parameter which consist of values for how many periods that have passed by, how many times the tasks have been throttled (denied access) and for how long time the tasks has been throttled.

• cpu.share – is a parameter that is also used by the CPU subsystem. The parameter is used to specify a relative share of CPU time for tasks in a cgroup. This is best illustrated with an example: If two different cgroups have a cpu.share of 1000 they will get the same amount of access to the CPU. If another cgroup have a cpu.share of 2000, the tasks in that cgroup will have access to twice as much CPU as the previous mentioned cgroups. Because of the flexible nature of cpu.share it is hard to predict CPU access throughout the whole system since CPU access with shares depends on a variety of factors such as how many cgroups that are on the system or how much CPU other cgroups are utilizing at a specific moment.

2.6 OpenShift

2.6.1 OpenShift Scheduler

A scheduler is a process that tells which node a new pod so be located at. OCP contains a default scheduler which follows these three steps to decide what node the pod should be placed on[17]:

1. Available nodes are first filtered to see what nodes can actually run the pod depending on the requirements the pod has.

2. The available nodes from the first step is passed through some grading functions which grades the nodes depending on different priorities. The score from each grading function is multiplied by a weight that is either a default value or decided by the cluster admin.

3. The best scored node is selected. If there is a tie between nodes, one is chosen by random.

After the OpenShift scheduler has decided what node the pod should be placed on it uses the master API to create the binding.

(15)

2.6.2 OpenShift Templates

Templates are YAML (YAML Ain’t Markup Language) files which contains configurations parameters. For example, when Tekton is installed there exists a specific template which tells OpenShift what thresholds are used, how long to wait before retrying, how much CPU the pod can use, how much memory the pod can use and so on. The template files are used with the OpenShift CLI tool oc and to apply the YAML file to a cluster the command ”oc create -f template.yaml” is used.

Currently in the template files the only thing that are automated are secrets and images registry URLs. Thus, CPU requirements and memory requirement etc. are hard coded.

2.6.3 Resources

A cluster admin can set constrains on the amount of computing resources that are being used in a project. Developers can set constrains on pods and containers. To set constrains quotas and limit ranges are used.

The object ResourceQuota sets constrains on the total computing resources per project. It can also limit the number of objects that are created.

The following properties can be constrained with the help of quotas:

• cpu - The sum of all CPU from all ”non-terminal” pods are not allowed to exceed this.

• memory- The sum of all memory request from all the ”non-terminal” pods are not allowed to exceed this.

For quotas there exists both limits and requests[39]. Limits are defined as the maximum capacity a specific object can use. For example, if a container sets its CPU limit to 2 it means that the container at maximum can request 2 cores. If, however, the container tries to exceed the limit, the system will throttle the container. If a memory limit is exceeded the container is terminated and depending on the restart policy, it may be restarted.

Request are defined as a minimum capacity for an object. For example, if a container sets its CPU request to 2 it means that the container at minimum can request 2 cores. However, if no request is specified the container is allowed to use all the cores at the node. If a pod request exceeds the amount of available resources on the biggest node, the pod will forever be unschedulable.

The components that make sure that the limits are not exceeded is the kubelet and the container runtime. According to the Kubernetes documentations the runtime is the decider of how the restrictions are implemented, and thus, different runtimes have different ways to make sure that limits are enforced[30]. The Kubernetes documentation gives two examples of how restrictions are implemented. The first method is that the system takes actions whenever it detects a violation of the limits. The second method is that the system makes sure that limits are never exceeded. Thus, one way is proactive, and the other method is reactive. When kubelet is launching a container, it forwards the limits to the container runtime. If an object only specifies either a limit or a request Kubernetes will automatically assign a corresponding request or limit to match the specification (and therefore assign a guaranteed QoS, section 2.6.4) to that object).

There are three different computing resources that can be specified, these are CPU, memory, and huge pages.

(16)

A CPU unit in Kubernetes is defined to be 1 vCPU/Core for cloud environment and 1 hyper thread on bare metal. CPU units can be specified in either whole units or as fractions. CPU requests and limits are an absolute unit which means that if a container requests 1 unit of CPU it will request the same amount of CPU independent of the architectural design of the CPU (such as number of cores etc.).

2.6.4 Overcommitted State

If pods on a node together have a resource limit higher than what is available on the node, it is said to be in an overcommitted state[19]. Whenever a node is in an overcommitted state there is a possibility that pods will request more resources than what is available and thus, the node must have a way to prioritize the pods. This is done with Quality-of-Service (QoS) classes. These classes are:

• BestEffort - This is active when there are no limit or request specified. For CPUs, the container is allowed to use all CPU cores but with minimum priority. BestEffort has the lowest priority.

• Burstable - Is active when limit and request are set but they don’t have the same value.

If the request resource is exceeded the corresponding containers is either killed or throttled.

However, BestEffort containers are handled before any Burstable containers are affected.

Burstable is the middle priority.

• Guaranteed - Is active when limit and request are set and they have equal value. Objects in this class will only be affected if the limit resource value is exceeded. Guaranteed is seen as the top priority.

2.7 OpenShift on IBM Cloud

On IBM Cloud it is possible to install OpenShift via the IBM Cloud catalog. The instance created from catalog is called IBM Red Hat OpenShift Kubernetes Service (ROKS). According to the IBM Cloud documentation for ROKS the OpenShift cluster hosted on IBM Cloud uses CRI-O as container runtime[24]. However, as described in section 2.4, CRI-O is an implementation of the CRI and thus it uses an underlying OCI compliant runtime. ROKS is a great option for this thesis since it is easy to create a new cluster, it deploys new clusters fast and it is easy to scale. As will be seen later in the report, ROKS also comes with some pre-deployed software which can be utilized throughout this master thesis.

2.8 Cloud Pak for Applications

As previously stated, this master thesis will focus on Cloud Pak for Application. Cloud Pak for Applications focuses mainly on three things:

1. Support development of new cloud-native applications.

2. Receive new value from already existing applications. Modernize the applications.

3. Optimize core applications (meanwhile the transformation to cloud is happening).

(17)

The installation of Cloud Pak for Applications can be done via the CLI tool provided by IBM[23].

The CLI tool is a Docker container which consists of several components. Since Docker allows execution of a command to a container it is possible to use the docker exec command to run an instance of /bin/bash and look through all the files and also get full control of what is happening in the installation phase. This allows identification of what techniques are in use but also if there are any third-part software. In the Cloud Pak for Applications installation container Ansible is installed.

2.9 Ansible

Throughout the whole installation Ansible Playbooks is used[21][14]. Ansible is a tool striving to handle automation for IT environments and a key feature is Ansible Playbooks. Playbooks allows instructions to be bundled in a YAML file. The instructions describe for the host running Ansible what to do and how. Instruction has a great potential and can for example run shell commands, use Ansible modules (which are often written in Python) and so on. Ansible Playbooks also offers conditional execution which is frequently used throughout the installation of Cloud Pak for Applications. Ansible can delegate tasks to different hosts, however, in the context of Cloud Pak for Applications only one host is used, more specifically, the container running Ansible (localhost).

2.9.1 Ansible Installation Process

When the installation is starting a main file is executed. This main file is an Ansible Playbook which includes all the components that is needed to set up the Cloud Pak for Applications environment. As of today, there are eight main components/dependencies that are installed. The main installation file also includes some other tasks such as login, checking for the OpenShift registry, get active subscriptions and so on. For each of the eight components that are being installed the following general process is done:

1. Switch to the specific project.

2. Create a pull secret by using the environments variables.

3. Install the specific operator.

4. Check if the specific operator is available after installation, retry x times if not ready.

There may be some steps in between the ones listed above such as querying a specific route, adding something in between and so on. But the major steps are listed above. The main dependencies and operators that are installed for Cloud Pak for Applications are:

• Catalog

• Servicemesh

• Kabanero

• Tekton

• Landing

• Signing-manager

(18)

• Transformation Advisor

• Application Navigator

2.10 Installation Process Details

As previously described, the installation of Cloud Pak for Applications is done with the help of a Docker container together with Ansible Playbooks. This section describes installation details that are of importance to this thesis.

2.10.1 Requirements

IBM provides some recommendations and requirements to install Cloud Pak for Applications, these are:

• OpenShift 4.4 or 4.5

• Docker

• Git

The recommended minimum capacity for the cluster is:

Software Nodes Memory (GB/node) CPU (cores/node) Disk (GB/node) Accelerator

for Teams, Transfor- mation Advisor,

and Ap-

plication Navigator

3 16 4 20

Transformation Advisor needs a CouchDB instance and thus a Persistent Volume Claim (PVC) is needed for that. When using a ROKS instance from IBM Cloud an IBM Block Storage service is created as well. This Block storage can be used to provision storage for the CouchDB automatically.

To utilize the Block Storage service a PVC needs to be created before the installation starts and in the configuration file for Transformation Advisor the spec.couchdb.persistence.existingClaim property needs to be specified. The PVC needs to be created in a namespace named: ta.

2.10.2 Installation Structure

The installation container image can be pulled from an entitled registry hosted by IBM. As an employee at IBM, you will get a key for this registry and thereby also have access to the installation container.

Before the installation can start, the container image needs to be pulled down from the entitled registry, then it is required to accept a license and then a process for extracting the installation configuration files to a local host can be started. These configuration files consist of the following:

(19)

• config.yaml - This is the main configuration file. This is where the public subdomain of the cluster is set but also specification for if Transformation Advisor should be used and Github settings etc.

• kabanero.yaml - This file contains information dealing with the Kabanero instance.

• transadv.yaml - This file contains information dealing with the Transformation Advisor instance.

To start the actual installation the container includes a bash file named main.sh that will initiate the Ansible Playbook.

2.10.3 Installation Container Structure

Since it is a Ansible Playbooks based installation the following files in the installation container are Ansible related:

• ansible.cfg - This file contains settings for Ansible. Some examples of settings are location of inventory file, plugins location and stdout callback.

• inventory - This file specifies the managed hosts. In this specific case, the only managed host is the localhost.

In the installation container there exists a directory named playbook. This directory contains multiple YAML files which contains instructions for Ansible. Except for the instruction files there exists different folders that corresponds to different roles. Each role stands for a different component, for example a role can be catalog and another can be for Tekton. In each of these role-directories it exists both other instruction templates but also scripts that are meant to be executed locally. These scripts can be anything from checking a version or to get subscription info from the cluster. The recommended entry point to start the installation is through a file named main.sh. By sending an argument ”install” to the script an ”ansible-playbook install.yaml” command is executed. When the Ansible playbook is started it goes from the top of the YAML file to the ending in a serial fashion. The following section describes the Ansible Playbook installation flow:

2.10.4 Initial Tasks

These tasks are done at the beginning of the installation and are focused more on setting up the correct environment before the actual installation begins.

1. OpenShift login - The absolutely first task is to login to the OpenShift cluster. This task consists of the following sub-tasks: login to cluster, get cluster URL, get cluster version, get cluster administrator role, and check and save authorization endpoint.

2. Get external access to the cluster - This task simply uses the OpenShift CLI tool to query information regarding the Ingress Operator.

3. Check entitled registry variables - This task executes a script that checks if entitlement keys are set as environment variables in the running container.

4. Get existing operators - A python script is executed that queries the cluster for all operator subscriptions. This subscription is then categorized into five different objects: Red Hat-

(20)

operators, certified-operators, community-operators, ibm-cp-applications-catalog and ibm-cp- applications-subscriptions.

5. Check for existing subscriptions - In this task a check is done to see if there exists a subscription for Elasticsearch, Jaeger, Kiali and OpenShift Service Mesh Subscription. All these checks are done by using a shell script with the corresponding name. The name is then used together with a OpenShift CLI command.

6. Check for existing subscriptions (kabanero) - This has the same structure as task five but in this case OpenShift subscriptions are checked, these are: OpenShift Serverless, OpenShift Pipelines, Appsody and Open Liberty.

2.10.5 Installing Dependencies and Operators

The remaining part of the installation is generally done as described by section 2.9.1. For a complete reference of the installation process see appendix A

2.10.6 Resource quotas

Since not all operators have a defined resource quota some operators have the ability to use all memory and all CPU cores. The following table specifies the resources that has a resource quota set and are thereby limited by their values:

Component Request CPU Request Memory Limits CPU Limits Memory

cp4a catalog 10m 50Mi 50m 100Mi

cp4a console 10m 50Mi 20m 100Mi

Landing 50m 200Mi 500m 400Mi

mf-operator 400m 500Mi 750m 600Mi

Security Signing Manager 50m 100Mi 100m 300Mi

CouchDB (TA) 500m 1Gi 16000m 8Gi

Ta 500m 1Gi 16000m 4Gi

TA UI 500m 1Gi 16000m 4Gi

TA Operator 400m 500Mi 750m 600Mi

Table 1: The defined quotas from the original installation.

2.10.7 Internal Dependencies

By analysing the Ansible Playbook, the internal dependencies between the different components can be visualized. The following figure illustrates the relationships and dependencies:

(21)

Login to OpenShift servicemesh - Check Kabanero - Check Domain Check Registry Catalog - Check Subscription Install

Catalog ServiceMesh Kabanero Operator

Tekton Landing

Signing-manager TA-Operator appnav

Kabanero Transformation

Advisor Dependencies

Figure 2: Dependencies among the different components in the Cloud Pak for Applications installation container.

As seen by figure 2, it can be identified that Kabanero is a dependency for three different components and that the Transformation Advisor Operator is an dependency for the Transformation Advisor instance.

2.11 Monitoring

OpenShift comes packed together with a monitoring stack[20]. This stack consists of the following components:

• Prometheus: is a well-known system that is used for monitoring and alerting. Prometheus uses a scraping technique to gather metrics and it is well integrated with Kubernetes[43].

• Grafana: is a analytic and visualization tool that queries metrics (from Prometheus in this case). It offers an easy-to-use plugin system.[11]

• Alertmanagar: is an application that handles and manages alerts.[40]

• cluster-monitoring-operator (CMO): manages the prometheus-operator, Grafana, kube-state- metrics and node-exporter instances. It monitors all the monitoring components and makes sure they are up to date.

• prometheus-operator (PO): manages the Prometheus and Alertmanager instance.

• kube-state-metrics: A service that listens to the Kubernetes API. It then generates metrics for the state of the Kubernetes objects. The focus of kube-state-metrics is to generate new metrics from the Kubernetes object’s state without any modification to the object. An important note

(22)

is that the metrics generated by kube-state-metrics may be different from the ones shown by kubectl. To export the metrics, kube-state-metrics exposes an endpoint named /metrics.[28]

• node-exporter: is a system developed for Prometheus with the purpose to expose hardware and OS metrics[41].

It should be noted that the Grafana instance that comes with OpenShift is read-only and to build customized dashboards a community version of Grafana needs to be installed on the cluster, this was done according to a guide on IBM Developers[36].

(23)

3 Implementation

This section covers the implementation of a modified installer which sends the installation instructions in parallel. To provide graphs and data for the user this section also describes a program which gathers data from the cluster and visualizes it via graphs. Four different methods and concepts are developed in this section as well for controlling the CPU resources in the cluster.

3.1 Time Measurements

To record the execution time of the installation the bash command time was used[33]. The time command output three different times, these are real, user and sys. The real time is the time from start of call until the end and can be viewed as the actual elapsed time from start until finish. The user time is CPU time spent in user mode and sys time is CPU time spent in kernel mode.

3.2 Standard Deviation

To describe the variability of the installation time the following formula for standard deviation was used

σ =r P |x − µ|²

N (1)

where µ is the mean installation times and N is the number of installations.

3.3 Parallel Container Installation

As described by section 2.9 the installation is done with the help of a Docker container which makes the main focus of this implementation to run the installation in parallel on different containers.

3.3.1 Implementation Decisions

Ansible provides different strategies that can be set to tell how Ansible should execute the instructions. The interesting strategy in this project is the free strategy[15]. The free strategy is designed to make the execution as fast as possible by letting all hosts do all tasks, but they do not have to wait for each other. This allows managed hosts that are more powerful to continue with the next task even if another managed host is stuck on a task. However, there is still a major problem with this strategy. It does not allow one managed host to be responsible for a specific task since all hosts must run all the same tasks. Thus, the free strategy is not a suitable option.

A major bottleneck of the implementation is the polling for a OpenShift resource to be ready and is thus blocking the execution flow. A solution to this could be to utilize asynchronous (async) execution. Even though asynchronous execution is not real parallelism it could still enhance the performance. Ansible supports asynchronous mode for playbooks. This means that a task can be set to async and whenever that task is started, Ansible will continue to the next task immediately.

The async task can then be checked later in the playbook if it is done. The installation container is, as described in section 2.10.3, built upon roles. This means that async cannot be applied on the existing layer since it can only be applied directly on tasks. Thus, async mode would be both time consuming and would likely not provide an optimal solution since it still tells one managed host that they need to run all tasks.

(24)

3.3.2 Parallel Playbooks

To provide a simple and yet effective solution to run the installation in parallel an idea is to simply run Ansible Playbooks on different hosts. As described in section 2.10.7 the original Ansible Playbook has internal dependencies. Thus, the respective components with dependencies are places in separate containers. That is, one container will be responsible for the Kabanero Operator and the corresponding child-components. Another container will be responsible for the Transformation Advisor related instances. The third container will be responsible for the other components that does not have any dependencies. This architecture is further illustrated in the figure below:

Figure 3: Separate containers installation structure.

Each column in figure 3 represents a separate installation container. It is evident from the figure that all containers have the Install and Post-Install role in common. This is mainly due to simplifications and the two roles are assumed to have a minimum effect on the performance.

3.4 Metric Collection

To make an analysis of the system and the installation a tool was needed to collect metrics from the OpenShift cluster. The collection was done in two different ways.

3.4.1 Python Program

A Python program was developed that utilized the OpenShift API together with the OpenShift CLI tool. From the OpenShift API the program requested Pod status, this was done by API calls to the /api/v1/pods endpoint[18]. This endpoint returns a JSON object for each of the pods in the cluster. In these JSON objects, data regarding where to pod was located (at which node), it also included information such as status and labels etc.

(25)

To retrieve the information regarding CPU and memory usage the OpenShift CLI tool was used.

The command ”oc adm top nodes” and ”oc adm top pods –all-namespaces” built the foundation for the Python program. To read values from a table would be both inefficient and tedious. Therefore the pyplot framework was used to provide graphs. The tool can provide four different graphs: CPU utilization per pod, CPU utilization per node, limit utilization per pod and limit utilization per node.

3.4.2 Prometheus and Grafana

Since the Python program is not able to provide graphs for more complicated data sources an alternative implementation was done to produce graphs. This method was to utilize the Prometheus and Grafana instances as described by section 2.11. The Prometheus queries used are described below:

Information regarding the queries is taken from the node-mixin GitHub since there are no direct documentation regarding the metrics[42].

• node cpu utilisation:rate1m - This metric is defined to be the percentage that the CPU is not in idle mode.

• instance:node num cpu:sum - Defined as the number of CPUs per node.

• instance:node load1 per cpu:ratio - 1min average run queue length divided by number of CPUs

• (instance:node cpu utilisation:rate1m* instance:node num cpu:sum) / scalar(sum(instance:node num cpu:sum)) - CPU Utilization

• instance:node load1 per cpu:ratio / scalar(count(instance:node load1 per cpu:ratio)) – CPU Saturation

3.5 CPU Constraints

As can be seen later in the results both the original installation and the parallel installation suf- fers from CPU saturation. This section tries to minimize the CPU saturation with the provided conditions that the installation container has as described by section 2.10.

To further investigate what can be the cause for the high CPU saturation and how to minimize it an analysis of how different resource constraint techniques can be applied was made (see section 2.6.3).

The different scenarios analyzed are:

1. No resource quotas.

2. Scaled down original limits.

3. Guaranteed Quality of Service.

4. Using ResourceQuotas (namespace constraints).

To ensure a level of consistency, three installations are made on each of the test-cases and an average installation time is used for comparisons.

(26)

3.5.1 Method For Limits Collections

To get all the requests and limits that are active in the cluster a combination of the bash commands:

$ oc adm top pods and

$ oc get pod -A

are used. The first command is similar to the Linux command top. The second command get pod information with the A flag which corresponds to all namespaces.

3.5.2 Method for Gathering cgroup and QoS Information

This subsection describes the process for how verification of QoS classes, requests and limits information were gathered.

The focus in this example will be on the Transformation Advisor server pod since it is installed by the Transformation Advisor Operator and no explicit limits or requests are defined in the installation container for this pod.

To find out which node the pod is located at the following command was first executed:

$oc describe pod ta -5 dg2c4 - server -8695 dc9746 - kvkrd

which returns the host IP for the pod. This IP can then be mapped to a specific node. Since the node is now known, all of the following command are executed on the Linux machine where the node is located.

To find the containerID for Transformation Advisor the following command was executed:

$ ps ax | grep POD_

7 1 4 3 0 ? Ssl 0 : 0 0 / usr / bin / c o n m o n - s - c

a c 5 0 c 3 3 4 f 5 1 b 7 6 c 4 7 c 2 6 3 1 4 3 6 d 6 d b e 4 2 e f 0 3 b f f f 4 4 5 a 2 3 f 7 9 5 3 0 1 8 6 f 1 a 1 5 6 1 0 0 - n k 8 s _ P O D _ t a -5 dg2c4 - server - 8 6 9 5 dc9746 - k v k r d _ t a _ 8 5 c 0 a 3 1 5 -6120 -4 ae4 - ad7e -6 e 7 b 6 5 5 c 9 b 0 4 _ 0 ...

The only process found is the one created from conmon. Since conmon takes a containerID as an argument the containerID can be noted and the rest of the stdout can be disregarded. It is also worth mentioning that the PID returned by the ps command is the PID for the conmon process and NOT the container.

Since the previous command revealed that runc was the runtime the following command can be used to get the PID for the container:

$ runc ps

a c 5 0 c 3 3 4 f 5 1 b 7 6 c 4 7 c 2 6 3 1 4 3 6 d 6 d b e 4 2 e f 0 3 b f f f 4 4 5 a 2 3 f 7 9 5 3 0 1 8 6 f 1 a 1 5 6 1 0 0

UID PID P P I D C S T I M E TTY T I M E CMD

r o o t 7 1 4 6 3 7 1 4 3 0 0 0 3 : 5 3 ? 0 0 : 0 0 : 0 0 / p a u s e

(27)

By printing out the cgroups hierarchy for that PID we can get the cpuacct and cpu scope subsystems:

$ cat / proc /71463/ cgroup ...

3: cpuacct , cpu :/ k u b e p o d s . s l i c e / k u b e p o d s - b u r s t a b l e . s l i c e / k u b e p o d s - b u r s t a b l e - p o d 8 5 c 0 a 3 1 5 _ 6 1 2 0 _ 4 a e 4 _ a d 7 e _ 6 e 7 b 6 5 5 c 9 b 0 4 . s l i c e / crio - a c 5 0 c 3 3 4 f 5 1 b 7 6 c 4 7 c 2 6 3 1 4 3 6 d 6 d b e 4 2 e f 0 3 b f f f 4 4 5 a 2 3 f 7 9 5 3 0 1 8 6 f 1 a 1 5 6 1 0 0 . s c o p e

...

It can been seen from the output that this container has a Quality of Service class of Burstable.

To see the cfs quota and cfs period the following command is executed:

$ cat / sys /fs/ cgroup /cpu , cpuacct / kubepods . slice / kubepods - burstable . s l i c e / k u b e p o d s - b u r s t a b l e - p o d 8 5 c 0 a 3 1 5 _ 6 1 2 0 _ 4 a e 4 _ a d 7 e _ 6 e 7 b 6 5 5 c 9 b 0 4 . s l i c e / cpu . c f s _ q u o t a _ u s

1 6 0 0 0 0 0 and

$ cat / sys /fs/ cgroup /cpu , cpuacct / kubepods . slice / kubepods - burstable . s l i c e / k u b e p o d s - b u r s t a b l e - p o d 8 5 c 0 a 3 1 5 _ 6 1 2 0 _ 4 a e 4 _ a d 7 e _ 6 e 7 b 6 5 5 c 9 b 0 4 . s l i c e / cpu . c f s _ p e r i o d _ u s

1 0 0 0 0 0

3.5.3 Test-case 1 - No Constraints

By removing the limits from a YAML files in the installation container Kubernetes will not be told set any constraints on the pods created by the deployment of Cloud Pak for Application. Thus, the CPU may behave differently, and installation time may therefore also differ. By removing the limits the consequence should be that, as described in section 2.6.4 all pods will be have a BestEffort Quality of Service class, thus all pods will be able to use all the CPU cores but they have minimum priority.

3.5.4 Test-case 2 - Downscaled Limits

To make an effort that there will be no overcommitment the request and limits from table 1 will be scaled down such that the summation of all limits will be the same value as the size of the CPU on one node. In the original installation the limitations have a summation of 50170 millicores. If each component will get the same percentage of CPU cores as in the original installation some components will get as little as 4millicores, this is of course not applicable and therefore some modification has been made.

In the following table the requests and limits are defined for this test:

(28)

Operator Request CPU Request Memory Limits CPU Limits Memory

cp4a catalog 10m 50Mi 50m 100Mi

cp4a console 10m 50Mi 20m 100Mi

Landing 50m 200Mi 300m 400Mi

mf-operator 300m 500Mi 450m 600Mi

Security Signing Manager 50m 100Mi 100m 300Mi

CouchDB (TA) 400m 1Gi 600m 4Gi

Ta 400m 1Gi 600m 2Gi

TA UI 400m 1Gi 600m 2Gi

TA Operator 300m 500Mi 450m 600Mi

Table 2: New resource quotas for test-case 2.

In table 2 it can be seen that the total CPU limit is 3320 millicores and the total memory limit is 10100 Mi. The total CPU limit is set to be lower than 4000 millicores since there already exists pods with limits as will be seen in the result for test-case 1.

3.5.5 Test-case 3 - Guaranteed QoS

For test-case 3 all limits and request will be the same. However, since the addition of all limits will exceed the nodes capacity, the limits and request will be as defined in test-case 2. By setting the limits and requests to equal values the pods should have a Guaranteed QoS and should therefore have a top priority.

3.5.6 Test-case 4 - Resource Quotas

As describe by 2.6.3 Kubernetes has an object called ResourceQuotas. ResourceQuotas are used to set constraints on projects/namespace level and will be the focus of this test case. If ResourceQuotas is beneficial for the installation time and CPU utilization it could enforce the administrator to use this option. The benefits with using ResourceQuotas instead of individual quotas is that it is more manageable (since it is less namespaces than pods/containers).

To implement this test the total limits from the original installer are added together and for the projects that has a total limit over the node’s capacity a ResourceQuota will be set equal to the cluster’s node capacity.

(29)

4 Evaluation

This section shows the analysis and evaluation of the implementation described by section 3.

4.1 Data Collection Tool

As described by section 3.4 a tool for collecting and plotting CPU and memory data was developed.

To illustrate the difference between the developed Python program and the Prometheus/Grafana combination the two following figures are given:

16 17:25 16 17:30 16 17:35 16 17:40 16 17:45

500 1000 1500 2000 2500 3000 3500

CPU usage / node 10.196.12.53

10.196.12.42 10.196.12.52

(a) CPU usage for a parallel installation produced by the Python script.

(b) CPU usage for a parallel installation produced by Prometheus and Grafana.

Figure 4: Comparision of tools

For both subfigures in figure 4 the three different lines corresponds to a separate worker node in the OpenShift cluster. The Y-axis represents the CPU usage on a node and the X-axis is the time axis. Both subfigures illustrates the CPU usage during the installation process of Cloud Pak for Applications and the red vertical line in figure 4a represent where the installation container outputs that the installation is done.

By looking at the two subfigures it is noticeable that the two metric tools capture the same process, however, it is also possible to notice some differences. The different nodes are not equally colorized in the two figures and that is mainly because the Grafana/Prometheus combination does not allow a simple configuration of colorization. Another observation is that the Grafana/Prometheus graph does not include a sign where the installation is done (as in the case with the Python tool). The most evident and most important difference between the two graphs is that the Python tool has an increased polling frequency which results in a more detailed graph.

It should be mentioned that throughout this report it will mainly be graphs produced from the Grafana/Prometheus combination. This is mainly because this combination can produce graphs for CPU saturation.

4.2 Variance Across Zones

It will be obvious later in the report that the installation times will vary a lot even when the exact same cluster configuration and Cloud Pak installation is used. One factor could be that the OpenShift cluster has its nodes located in different (physical) zones. To further see the effect on

(30)

how the installation times behaves when the Cloud Pak installation is done on a cluster that has the same setup and that is located on the same zone the following times were recorded:

# real user sys

1 16m29.627s 4m7.169s 0m35.885s 2 15m24.166s 4m20.529s 0m35.878s 3 15m6.628s 3m28.726s 0m30.359s 4 15m51.106s 4m39.490s 0m40.180s 5 16m11.002s 4m42.535s 0m40.885s Table 3: Recorded times from the zone test.

The times from table 3 are from the original Cloud Pak for Applications installation. The mean installation time is 15m and 48.51s and the standard deviation is 0m and 30.17s. All the nodes for the clusters used in this test were located in Amsterdam (ams03).

4.3 Original Installation Times

To provide a baseline for the rest of this report, tests where done on the original installation of Cloud Pak for Applications, section 2.10. With cluster setups as described in Appendix B table 20, the following times were recorded:

# real user sys

1 15m46.830s 3m55.691s 0m34.182s 2 13m50.349s 3m48.059s 0m34.788s 3 16m20.800s 4m4.167s 0m35.588s 4 15m24.210s 3m35.594s 0m33.032s 5 14m5.768s 3m10.951s 0m28.978s Table 4: Installation times from the original installation.

As previously mentioned, the installation times differs a lot. For the original installation, the mean installation time is 15m and 5.59s with a standard deviation of 0m and 58.21s. As the clusters in this test were located in different zones it is interesting to compare this standard deviation with the standard deviation from the test where all installations were made in the same zone 4.2. The mean time is a bit shorter for this test, but the standard deviation is a bit higher, thus, by only looking at the values from these tests it cannot be guaranteed that the installation time will be consistent even if the cluster is located in the same zone.

4.4 Parallel Installation Times

This section analyses and reflects on the parallel implementation as described by section 3.3. With the cluster setups as described in Appendix B table 21 the following times were recorded for the parallel installation:

(31)

# real user sys 1 10m47.596s 2m36.140s 0m23.520s 2 10m53.424s 2m5.980s 0m19.1987s 3 10m15.861s 2m19.533s 0m18.380s 4 11m29.865s 2m43.524s 0m24.151s 5 9m40.562s 2m22.740s 0m22.318s Table 5: Installation times from the parallel installation.

As seen by the values from table 5 all the installation times from the parallel installation is shorter than any of the original installation times. This indicates that the parallel installation is faster, which can be further strengthened by the mean time which is 10m and 37.46s. A comparison by the mean times tells that the parallel installation takes around 70% of the original installation time. An interesting aspect is also that the standard deviation for the parallel installation is 0m and 36.90s which indicates that the installation time is more consistent than the original installation.

By analyzing the logs from the containers where the Ansible Playbook is running it is possible to retrieve timestamps for when the components are deployed to the cluster. This process is illustrated in the following figure:

Login Catalog servicemesh signing-

manager appnav Post-

Install

Login Kabanero tekton landing Kabanero - Create Instance Post-

Install

Login TA-O

00:00:06 00:00:50

TA create instance Post-

Install

00:05:35

00:04:40 00:05:15 00:07:40

00:07:25 00:00:06

00:00:06 00:09:45 00:11:29

Figure 5: Time-series representation of the containers installing Cloud Pak for Applications.

In figure 5 each color represent a separate container. As seen from the figure, all containers will complete their installation in different times. It is also possible to identify that the container that is containing the Kabanero setup is taking the longest time to complete. It should also be observed that since the green container is taking the longest time, the blue and orange containers have no need to be further improved (divided into more sub-containers) since they do not affect the overall time it takes to install Cloud Pak for Applications. However, it should be noticed that if the green containers installation time were reduced to be lower than either the orange or blue it could be beneficial to split up the blue container since the corresponding components does not have any dependencies among them.

To further investigate how the CPU is behaving during the parallel installation the following two graphs are given:

(32)

(a) CPU usage for a parallel installation. (b) CPU utilization for a parallel installation.

Figure 6: CPU behaviour during parallel installation.

There are several interesting aspects of these graphs. The first is that the CPU utilization as seen by figure 6a is relative low and never exceeds 32%. All the three nodes seem to suffer from high CPU saturation, but it differs a lot between them, the yellow node has a maximum value of around 90%, the blue has around 155% and the green has a maximal CPU saturation of over 200%. Another observation is that after the CPU saturation has climbed it also decreases, this indicates that there is a task or several task that are on heavy CPU load under a certain time. In perfect conditions the CPU saturation would consistently be as low as possible. Thus, a further investigation on CPU constraints was made.

4.5 CPU Constraints

Due to the low CPU utilization and the high CPU saturation discovered in section 4.4 this section covers attempts to speed up the installation by using different CPU constrains techniques available in Kubernetes. The installations throughout this section were executed with the parallel installer.

4.5.1 Test-case 1 - No Constraints

As described in section 3.5.3 this test is about removing the pre-defined limits in the installation container. After the test, the following times were recorded:

# real user sys

1 11m22.182s 2m32.276s 0m23.258s 2 8m25.422s 1m52.628s 0m17.778s 3 8m58.580s 2m1.294s 0m18.125s 4 10m11.732s 2m29.425s 0m22.661s 5 9m59.775s 2m27.526s 0m22.421s Table 6: Installation times for test-case 1.

The cluster setup for this test can be found in Appendix B table 22. The mean installation time is 9m and 47.54s and the standard deviation is 1m and 1.37s. The mean time is a bit lower than the installation time for the parallel installation with pre-defined limits. However, the standard deviation is also higher which implicates that no conclusion can be drawn that this test produces

Efficient parallel installation of software collections on a PaaS