Comparison between OpenStack virtual machines and Docker containers in regards to performance

(1)

Comparison between OpenStack virtual machines and Docker

containers in regards to performance

Victor Bonnier

18 June 2020

Faculty of Computing

Blekinge Institute of Technology SE-371 79 Karlskrona Sweden

(2)

This thesis is submitted to the Faculty of Computing at Blekinge Institute of Technology in partial fulﬁllment of the requirements for the bachelor’s degree in software engineering. The thesis is equivalent to 10 weeks of full-time studies.

Contact Information:

Author:

Victor Bonnier Vikkear@hotmail.com University advisor:

Nauman Ghazi

Department of Software Engineering (DIPT)

Faculty of Computing Internet :www.bth.se

Blekinge Institute of Technology Phone :+46 455 38 50 00 SE-371 79 Karlskrona, Sweden Fax :+46 455 38 50 57

(3)

Abstract

Cloud computing is a fast growing technology which more and more companies are starting to use throughout the years. When deploying a cloud computing application it is important to know what kind of technology that you should use. Two popular technologies are containers and virtual machines.

The objective with this study was to ﬁnd out how the performance diﬀers between Docker containers and OpenStack virtual machines in regards to memory usage, CPU utilization, time to boot up and throughput from a scalability perspective when scaling between two and four instances of containers and virtual machines.

The comparison was done by having two diﬀerent virtual machines running, one with Docker that ran the containers and another machine with OpenStack that was running a stack of my virtual machines. To gather the data from the virtual machines I used the command ”htop” and to get the data from the containers, I used the command ”Docker stats”.

The results from the experiment showed a favor towards the Docker containers where the boot time on the virtual machines were between 280-320 seconds and the containers had between 5-8 seconds bootup time. The memory usage was more than doubled on the virtual machines than the containers. The CPU utilization and throughput favored the containers and the gap in performance increased when scaling the application outwards to four instances in all cases except for the throughput when adding information to a database.

The conclusion that can be drawn from this is that Docker containers are favored over the OpenStack virtual machines from a performance perspective. There are still other aspects to think about regarding when choosing which technology to use when deploying a cloud application, such as security for example.

Keywords: OpenStack, Docker, CPU, Memory

(4)

Abbreviations and Deﬁnitions

CPU(Central Processing Unit) is the primary component of a computer that processes instructions.

Environmentis the state of the computer, usually determined by the current programs that are running and the hardware and software characteristics [1].

Hypervisoris the software that creates and runs virtual machines. There are two diﬀerent types of hypervisors, the ﬁrst type runs directly on top of the computers hardware while the other type runs on top of an operating system.

Load balanceris the software that redistributes traﬃc to the application and make sure that no server gets overworked.

Neutron is the component in OpenStack that provides ”networking as a service” between the diﬀerent interface devices [2].

Nginx[3] is an open source software that have been used in this study for load balancing between the containers.

Operating System (OS) is the system software that manages the computers hardware and software resources. It also provides services for the computer programs.

Throughputis the amount of requests that gets through within the given time frame in this study.

Virtual Machine (VM) is a software that behaves as a separate computer system and is able to perform most of the functions of a physical computer [4].

(5)

1 Introduction

Cloud computing is a fast growing technology which makes makes it possible for users to access an application on demand through the internet. Cloud computing have many diﬀerent characteristics but the ﬁve main characteristics are that the application should be available through the internet at any given time.

The application should be self serving which means that no human interaction should be necessary, the application should be scalable whenever needed. The application should be available to multiple users at the same time and the users should only pay for the amount that they use [5].

An example of a widely used virtual machine service is Amazons EC2 instances [6], which are available through Amazon Web Services while an example of container technology is Docker which is a commonly used technology over the world.

The growth of application deployed on the public cloud can not be denied and is considered a top priority investment by organizations since they want to be able to stay on top of the modern technology. Cloud computing also have a very steady growth each year [7].

The problem that occurs with this technology is that it is not really clear what available option that you want to use when deploying an application on the cloud and since cloud computing can be very expensive [8], it is very important that you choose the alternative that suits your application the most so that your company will not waste money because of lack of knowledge.

Scalability and elasticity are two important aspects of a cloud service which is about the services ability to increase the capacity of the service when it is needed [9, p.10-11]. Examples of when scaling is needed is when a service suddenly re- ceives a high amount of traﬃc to the point where the amount of resources is not enough which will lead to the service not being available. When a service has good elasticity, the application is scaled ahead in time in a way so that there is no downtime on the service.

Another aspect that companies have to take into consideration when deploying an application on the public cloud is the security aspect. Generally, an application that is deployed through virtualization is more secure than an application that is containerized because of how containers runs directly on top of the host operating system which gives an attacker a bigger surface to attack [10].

Two common options when developing cloud solution or cloud services are virtual machines and containers. The choice between these two technologies can be very important to the cost and efficiency of the service and can be very difficult to choose in certain scenarios. Because of this, it is important to further study the performance difference between these two technologies.

(8)

The goal of this study was to compare the performance diﬀerence between containers deployed through Docker, and virtual machines deployed through Open- Stack from a scalability perspective when running an application that can fetch data from a database, write to it and also can handle ﬁle uploads. The reasoning was that these are very common use cases on an application and it helps getting the study closer to a realistic application deployment.

In the scope of this study I chose to focus on the performance diﬀerence between Docker containers and OpenStack virtual machines. Security was not included in the scope since it would make the scope too big for this kind of study.

The reasoning behind choosing Docker for hosting the containers is because it is a very popular ”Container as a service” and is used all over the world. The reasoning behind choosing OpenStack for hosting my Virtual machines is because of a combination between its popularity and also because they oﬀer a free de- velopment environment which made it easier to set up the experiment and run it.

The rest of this study will be existing of the following. First, a more in depth view of how OpenStack and Docker is built up from an architectural aspect. One will then see works related to this study and what kinds of results that they got.

Then the research questions will be presented along with the reasoning behind the questions chosen. Then my research method of getting the results will be shown, along with how the data was gathered. Then the results gathered from my study method will be presented together with graphs that are produced by the data. One will then look at my analysis of the results along with the assumptions that can be made based by the results and also relevant validity threats. Lastly, one will see the conclusions made along with points that can be researched in future work.

1.1 Introduction to Docker and OpenStack

1.1.1 Docker

Docker made their ﬁrst public release in 2013 which made it possible for devel- opers to use Docker containers for the ﬁrst time [11]. Docker containers runs directly on the host operating system and packages code and dependencies together at the application layer. This makes it possible for multiple containers to run on the same machine and even share the same operating system kernel [12]. The overall docker architecture consists out of three main components.

The ﬁrst one is the client which is the command line interface used by the users to interact with Docker. The second component is the daemon which acts as a listener for API requests and manages all of the images, containers, networks and volumes. The last component is the registry which is used to download images from either public or private repositories [10].

(9)

Figure 1: The Docker architecture [10]

1.1.2 OpenStack

Openstack originates from the ﬁrst months of 2010 where Rackspace wanted to refactor the code that was running their Cloud servers. They decided to use the newly released ”Nova” to form what later became Openstack with the mission to create a Open Source Cloud Computing platform that is very user friendly and performs well in regards of scaling in both private and public clouds [13].

OpenStack consists of a dashboard on the web that can be used to control all of the parts of the architecture such as the images, computations, network settings and volumes. The overall architecture is very complex and is shown in ﬁgure 2.

(10)

Figure 2: The OpenStack architecture [14].

The main components in ﬁgure 2 that have been utilized in this study and their purpose are the following:

• Glance, this component contains the images that the user has uploaded to it. This component also includes metadata deﬁnitions [15].

• Nova, this is the component that creates and runs the virtual machines.

It runs as a set of daemons on top of the existing Linux servers [16].

• Neutron, this is the component in OpenStack that provides ”networking as a service” between the diﬀerent interface devices [2]. In other terms, this component creates the physical or virtual network that the virtual machines connects to on boot.

• Cinder, this component is mainly responsible for creating and providing volumes that the virtual machines should use [17].

(11)

• Keystone, this component is responsible for the authentication when using either the web interface or the command line interface [18].

1.1.3 Diﬀerences in architecture

The main difference between how containers and virtual machines works is that the virtual machines runs on top of a hypervisor and have to simulate an entire operating system which takes up a lot of system resources in comparison to the containers that runs directly on top of the host operating system [19]. This architectural difference is illustrated in figure 3.

Figure 3: Containers vs VMs illustration.

(12)

2 Related work

There have been a lot of diﬀerent comparisons done between containers and virtual machines throughout the recent years and the methods for testing out the performance diﬀerence varies.

Qi et al. [20] found that containers provides an overall better performance than the virtual machines in regards to CPU and memory in a big data environment by using Spark jobs. They also found that containers performs better from a scalable perspective where they compared the performance of between 2 and up to 512 diﬀerent containers and virtual machines. Their experiment result showed that the bootup diﬀerence between containers and VMs increased rapidly after 16 instances and upwards.

In another experiment, Kyoung-Tack et al. [21] build their own clouds and performs a comparison experiment between their container cloud and their virtual machines cloud. They set up their diﬀerent clouds using OpenStack for the virtual machines with a KVM hypervisor and Docker for their containers. The results from the experiment showed the average bootup time for the containers were around 1.53 seconds and the average time for the VMs was 11.48 seconds. They also recorded the diﬀerence in calculation speed by using a python program that calculated the factorial of a number and found that the virtual machines have a slightly faster calculation time when calculating the factorial of 100000 and an even better performance in comparison to the containers when calculating the factorial of 200000.

Herrera-Izquierdo and Grob [22] performed a performance comparison experiment with focus on CPU, RAM, network and disk performance using Docker for the containers and VirtualBox as the VM engine. The results from the experiment was that the containers were more eﬃcient on all of the ﬁelds except for the disk performance where the virtual machines outperformed the containers vastly.

In a study that is very similar to this thesis when it comes to how the virtualization environment is set up, Vestman [23] runs an experiment and compares the performance of virtual machines deployed through OpenStack and containers deployed through Docker in regards to CPU operations, disk read/write, main memory usage and ﬁle transfer speed between host and application. By the results, Vestman came to the conclusion that the virtual machine performed better in handling requests faster while the containers performed better in ﬁle transfer and resource usage. He also came to the conclusion that virtual machines should be used when privacy is higher priority than performance.

Chae et al. [24] made a performance comparison between linux containers and KVM virtual machines by running between one and four diﬀerent instances with focus on CPU, memory and disk I/O. They both had about the same CPU us-

(13)

age in idle mode with the VMs having slighly less average CPU idle but with more deviation. The result from the memory comparison showed that the virtual machines used up between 3.6 and 4.6 times more memory than the linux containers.

Judging by all of the related works, containers seems to be the better alternative when only looking at it from a performance perspective. Of course, there are other aspects that should come into mind when choosing which technology to use such as the security aspect for example. In most of the reviewed work, containers are in lead in almost all of the performance categories, speciﬁcally CPU, memory, disk and network usage. Although the results vary when it comes to the disk performance as the virtual machine was better than the containers in some related works and vice versa.

(14)

3 Research method

3.1 Research questions

The research questions that I chose to study further were the two following:

Research question 1: How does Docker containers and OpenStack virtual machines perform in comparison when running an application that can handle database requests along with ﬁle uploads in regards of Memory usage, CPU utilization, throughput and time to boot up the applications?

Research question 2: How does the performance change between virtual machines and containers when scaling the application out to three and four diﬀerent instances, in regards to memory usage, CPU utilization, throughput and time to boot up?

The first research question gives insight in how the performance differs in these specific cases since getting data from a database, inserting data into a database and uploading a file that is stored on the file system are all very common cases in an application which makes this study more relevant to an actual product.

The reasoning behind choosing memory usage, CPU utilization, throughput and boot up time was because they are four crucial statistics when looking at the performance of a service.

The second research question shows us in what way the performance gap changes between containers and virtual machines. For instance, if the performance gap increases in the favor of the virtual machines or vice versa.

The prediction that I made on these research questions was that the Docker containers were going to outperform the OpenStack virtual machines in all of the categories and that the gap between the containers and the VMs were going to increase with every instance that scales out. The reasoning behind this prediction is based in how the architecture behind how containers and virtual machines works.

3.2 Theoretical study

For the theoretical part of the study, I have looked up studies that checks the performance on containers or virtual machines that is included into my ”Related Work” section. My criteria for the related work was that the study should be about either container or virtual machine performance or preferably a comparison study between the two. The database that I used mainly for ﬁnding relevant work is the IEEE database because it provides a good amount of research that can be chosen from.

(15)

The keywords that I used for ﬁnding related works were the fol- lowing: ”Container vs VM performance”, ”container performance”, ”Virtual machine performance”, ”scalability”, ”performance”, ”CPU”, ”Memory”.

The related works that I chose consisted of studies focused on comparing the diﬀerence in performance between containers and virtual machines.

3.3 Empirical study

For the empirical part of the study, I set up a controlled experiment where the dependant variables were the memory usage, CPU utilization, throughput and bootup time and the independent variables were the virtualization method, which in this study was Docker containers and OpenStack virtual machines. The reasoning behind performing a controlled experiment was that the results from the experiment would be more reliable and reproducible. In the controlled experiment, I set up a containerized environment through docker version 19.03.11 build dd360c7 and virtualization environment deployed through OpenStack to be able to answer my two research questions. To be able to do this, I set up two diﬀerent Ubuntu Server 18.04 virtual machines using VirtualBox [25] with the same limitation to make this study fair. The host machine had the following relevant speciﬁcations:

• Motherboard: MSI Z170-A PRO

• Processor: Intel Core i7-6700K CPU 4.00GHz

• Physical Memory (RAM): 16 GB

• Graphics Card: MSI GeForce GTX 970 4GB

• Operating System: Microsoft Windows 10 Home 10.0.18362 Build 18362 Both of the virtual machines had their RAM limited at 8 GB for fairness and had their network set up the same way by using a host only adapter.

3.3.1 The application

The application that have been used for the study consists out of two diﬀer- ent components. The ﬁrst component is the database which is a MYSQL [26]

database in this study. The database consisted of one table which our application fetches all rows from and also ﬁlls depending on the call made to the backend. The code to set up the database can be found in Appendix A.1.

The second component of the application is the backend that is written in Flask which is a micro web framework that is written in Python [27]. The application consists out of three different requests. The first request gets all of the users from the database table, the second request inserts a row on the table with the body sent and the last request uploads a file and stores it on the applications file

(16)

system. The premise behind these three types of calls is that they explore three different areas. The first one checks the application performance of getting information from a database that while the second call checks how fast the application can insert information into a table while the last call checks how the performance is when transferring files and storing them in the file system.

The application in itself is not too realistic since it would need a proper front end. However, the purpose of the application was to measure the performance in the three diﬀerent scenarios. The code used for the application can be found in Appendix B.1.

3.3.2 OpenStack setup

OpenStack was setup using Devstack [28] which is a series of scripts that creates an OpenStack environment on the virtual machine so that it can be used locally through either the horizon web interface or the command line interface depending on preference. The network on OpenStack was set up with the public interface connected to the host only adapter to make it available to my host operating system so that I can connect to the virtual machines properly. This needs to be set up in the ”local.conf” -ﬁle along with all of the passwords for the OpenStack services and the plugins that is needed when running the Devstack install script.

When installing Devstack I also made sure to include the Heat [29] plugin in the ”local.conf” -ﬁle which was later used when deploying a stack of virtual machines. After the installation was completed, I had to conﬁgure the DNS settings so that the virtual machines could properly connect to the internet. I also downloaded an Ubuntu Bionic Beaver 18.04 image that I deployed using the web interface on OpenStack.

After the initial setup, I set up the virtual machine with the application and the dependencies so that it was ready for usage, then I created a image using the snapshot feature on OpenStack. The application was then deployed using Heat together with my application image to create a stack of virtual machines that already have my application ready to go whenever booted up. Lastly, to make the virtual machines available through the internet I had to set up a ﬂoating IP address for each instance which easily could be done using the horizon web interface. There was also a load balancer that is set up using OpenStacks own load balancer through ”Neutron”. The MYSQL database was deployed on its own virtual machine image since it would not be fair for the experiment if I hosted it on my host operating system.

3.3.3 Docker setup

Docker have been set up on the Ubuntu Server by using the command ”snap install docker” which sets up the Docker environment for you. After the installation, I have my Dockerﬁle (Appendix C.1) that sets up the Flask environment

(17)

with all of the dependencies that are needed for the application to work on the container. The Dockerﬁle is shown in ﬁgure 4.

Figure 4: Dockerﬁle for my Flask application.

The application was then deployed through a ”docker stack” which initially launched two instances of my application along with the MYSQL database and my load balancer which is Nginx [3] in this study. When this was setup, the application could be further scaled by using the Docker command line interface and inputting how many instances of the application that I wanted to run.

Nginx is then used to direct all of the incoming traffic in a way so that all of the instances gets an equal amount of load in regards to the CPU usage in this case. The configuration file for Nginx is shown in Appendix C.2.

3.3.4 Gathering data

To increase the load of the applications and actually get some useful data, I had setup a couple of scripts that tests each of the three available requests in a loop, creating a lot of traﬃc. My test scripts tested each of the requests individually because I wanted to focus on the performance on the three diﬀerent parts individually rather than together. The scripts also shows at the end of the run how many requests actually got through with a correct response. The code for the test scripts can be found in Appendix D.

Gathering the data from my Docker containers was done by using the ”Docker stats” command when my applications are running and the scripts that increases the load. Gathering the data from the virtual machines was done by using the

”htop” command that displays the CPU and Memory usage individually as shown in ﬁgure 5.

(18)

Figure 5: Example look of htop command.

The tests were running for exactly 45 seconds and the CPU utilization and memory usage was recorded every second so that they could be used in a graph displaying how the performance looks over time. The time to boot up the containers and the virtual machines was manually timed because there are no commands for showing the bootup time in OpenStack that I could ﬁnd. I started the timer when I started the container or virtual machine and stopped the timer when they were ready for traﬃc.

Combining all of the data mentioned above gave results that answered both of the research questions mentioned earlier.

(19)

4 Results

After running the entire experiment. I got a lot of results that will be detailed below in the sub sections for each of the request type. Overall, the containerized application have a lead on the performance on CPU utilization, memory usage, overall throughput and also on the boot up time.

The results from the boot up time test is that it took between 5-8 seconds for my containers to start up and be ready for usage while it took the virtual machines between 280-320 seconds for them to boot up and be ready to handle requests.

When it comes to the memory usage, the Docker containers used up between 35-36 MB per instance throughout the whole of the tests without much variation, while the OpenStack virtual machines used up between 80-86 MB per instance throughout the testing. This variation was consistent between all of my three diﬀerent test scripts and did not change much when scaling up the application up or down.

The following subsections will show graphs containing information about how much of the CPU over all of the instances running that the containers and virtual machines actually uses over a total of 45 seconds. The following subsections will also reveal how many requests that got handled by each of the requests for both the containers and the virtual machines in each of the types of requests.

4.1 Fetching from database

Running my ﬁrst test script which tests the applications ability to fetch information from the database and send it back to the user gave the following result when using two to four diﬀerent instances:

(20)

Figure 6: Get from database with 2 instances

This graph shows us that the Docker containers were outperforming the virtual machines when fetching information from a database table. The average CPU utilization by the containers throughout the time came up to an average of 18.5 percentage while the virtual machines used up 23.3 percentage. There was a diﬀerence in the overall throughput as well as the containers managed to handle 1071 diﬀerent requests while the virtual machine managed to handle 982 requests.

(21)

This graph displays a similar result where the containers again outperforms the virtual machines when it comes to the CPU utilization total over the three diﬀer- ent instances. The average CPU usage from the containers was 11.2 percentage while the average CPU usage from the virtual machines was 15.6 percentage.

The relative diﬀerence in CPU usage between the virtual machines and the containers is increased when scaling out from two to three instances. The container application managed to deal with 1140 requests while the virtual machines handled 1063 requests.

(22)

The last graph in this section still shows us that the total CPU usage over the four instances was better than the virtual machines CPU usage. The average CPU usage for the containers was 6.4 percent while the virtual machines was 10.5 percentage. Again, the relative diﬀerence between three and two instances is increased when scaling out to four diﬀerent instances, favoring the containers.

The containers handled 1293 requests while the virtual machines handled 1134 requests.

The results from these graphs shows us that containers are in the overall lead in all of the areas in this type of request. The thesis will go more in depth of these results in the ”Analysis” section of the report.

4.2 Adding to database

This section shows the results from my second test script which tests the applications ability to insert a row into the table in the database.

(23)

Figure 9: Add to database with 2 instances

Just like the previous graphs, the containers were performing better than the virtual machines when it comes to the total CPU usage over the diﬀerent instances. The average CPU usage of the containers was 9.8 percentage while the virtual machines used 16 percentage. In this user case, the diﬀerence in the throughput was slightly increased as the containers managed to handle 2411 requests while the virtual machines only managed to handle 2141 requests.

(24)

When scaling the application out to three diﬀerent instances I got the results shown in the graph above. The average container CPU usage landed on 5.5 percentage while the average CPU usage on the virtual machines landed on 11 percentage. The overall throughput on the containerized application was 2485 requests while the throughput on the virtual machines was 2274 requests.

(25)

Lastly, when scaling the application out to 4 diﬀerent instances, the average total CPU usage on the containers went to 4.7 percentage while the virtual machine average total CPU usage went to 7.9 percentage. The overall throughput on the containers were 2542 requests while the throughput on the virtual machines was 2357 requests which is a slight improvement in the big relatively on the throughput with two and three instances.

The overall impressions from the above graphs is that there is a lot to look into considering that the performance gap decreased when scaling up to three instances from two but increased when scaling out further to four instances.

The overall throughput gap was also decreased when the application was scaled out. More about this will be described in the ”Analysis” section of the report.

4.3 File upload

Lastly, the last test script was testing the applications performance when uploading ﬁles that the application received and stored in the ﬁle system.

(26)

Figure 12: Uploading a ﬁle with 2 instances

This first graph shows up the total CPU usage over two instances that received the file that the test scripts uploaded to it. The containerized application used up an average of 13.5 percentage of the total CPU while the virtual machines used up an average of 21.4 percentage. This difference in performance favors the containers. The containers managed to receive and store 1687 files while the virtual machines managed to receive and store 1542 files.

(27)

Scaling the application up to three different instances decreased the gap in CPU usage as the overall usage from the containers went to 9.5 percentage while the virtual machines used an average of 14.9 percentage. The overall throughput of the containers was 2286 files while the virtual machines had a throughput of 2169 files.

(28)

On the last graph, the containers outperformed the virtual machines even more as the average CPU usage of the containers was 7.2 percentage while the virtual machines used up 12.4 percentage of the total CPU available. The overall throughput of the containers went to 2417 ﬁles and the throughput of the virtual machines went up to 2245 ﬁles.

Just like the other two tests, the containers were ahead in performance in CPU usage, memory usage and overall throughput. A deeper look into the results shown above is presented in the ”Analysis” section of the report.

(29)

5 Analysis

Below, I have analyzed the results from my scripts in regard to requests handled, memory usage, bootup time and CPU utilization and lastly I have answered the research questions and mentioned some validity threats to this experiment.

5.1 Amount of requests handled

In all of the comparisons done in this experiment, regardless of how many instances, containers were outperforming the virtual machines in how many requests that the application could handle in 45 seconds. The average diﬀerence in throughput on all of the tests landed on around 158 requests with the stan- dard deviation of 57.2 requests.

When scaling the application outwards to three different instances, the overall throughput gap shrunk in all three cases while scaling out to four instances lead to the gap increasing when fetching from the database and uploading files while the gap when inserting into database kept on shrinking. This overall data indicates that the difference in the amount of requests handled will keep on increasing as the application keeps scaling out when it comes to fetching data from a database and uploading files. The data also indicates that the gap will further shrink or even go to the virtual machines favor when inserting information into a database.

5.2 Memory usage

In regards to the memory usage, the virtual machines used up more than double the amount of memory that the containers used up. This is not surprising considering that the virtual machines have to emulate an entire OS on its own while the containers does not [19]. In this specific case, this difference is huge considering that you can have two containers running on the same amount of memory as one of the virtual machines given that the CPU can handle it. In other cases, where the application could be more heavyweight and take more memory overall, the difference would not be that severe.

When scaling the application outwards, the memory usage kept on increasing linearly for each instance. In this case that when I scaled out the application to four diﬀerent instances, the virtual machines more than double the amount of memory than what my containers did, meaning that I could instead be running eight diﬀerent containers on the same amount of memory given that my CPU is powerful enough.

5.3 Bootup time

From the results section I got that it took between 5-8 seconds to start up a container and get it ready for traﬃc and it took between 280-320 seconds from

(30)

the moment you started up a virtual machine until the moment that it was ready for traﬃc. This is a huge diﬀerence where it can take a virtual machine 64 times more time to boot up in a best case scenario for the containers and a worst case scenario for the virtual machines.

This result is important because when a service gets exposed to a lot of traﬃc at the same time until the point where the system needs to scale out in order to not get overloaded, having a slow bootup time can cause a slow or non existing response for the customers which in turn could lead to them leaving before the virtual machine is fully ready.

5.4 CPU utilization

As shown in the graphs in the results section, the CPU usage of the containers are overall much better than the CPU usage of the virtual machines. The overall CPU utilization becomes even better when scaling outwards as one can see when the application is getting users in two instances and when the application is getting users in four instances. When two instances were running the test, the containers used up about 79 percentage of what the virtual machines used up while when it was scaled up to four diﬀerent instances, the containers used up around 61 percentage of what the virtual machines did. This also applies for when the application scaled up to three diﬀerent instances where the containers used up around 72 percentage of what the VMs did. The gap increases by 7 percentage when scaling up to three instances and then the gap increases by another 11 percentage, indicating that the gap could further be increased favoring the containers by adding further instances.

When uploading ﬁles, the CPU utilization diﬀerence stayed about the same when scaling up to 3 instances but the gap is increased, favoring the container when scaling up to four instances, going from around 63.5 percentage of the virtual machines CPU usage to around 58 percentage. This change also implies that the gap will further increase when scaling out even more.

When inserting users into the database, I got a slightly diﬀerent result than with the other two routes. When scaling out from two to three instances, the gap goes from 61.3 percentage to 50 percentage while scaling out to four instances, the gap goes up to 59.5 percentage. This still implies that the gap between containers and virtual machines will further increased with more scaling since the gap in the end goes from 61.3 percentage at two instances to 59.5 percentage at four instances.

5.5 Answering the ﬁrst research question

The ﬁrst research question was the following: ”How does Docker containers and OpenStack virtual machines perform in comparison when running two applications that can handle database requests along with ﬁle uploads in regards of

(31)

Memory usage, CPU utilization, throughput and time to boot up the applications?”. The full answer to this question have already been shown earlier in this section and in the result section but to summarize it into a few sentences:

• The Docker containers have the overall advantage over the OpenStack virtual machines in regards to memory usage, CPU utilization, bootup time and the throughput.

• The diﬀerence in bootup time favors the containers and could cause the application not being reachable for a while if the application suddenly gets a lot of traﬃc and the virtual machine does not start quickly enough to balance the load out.

• The difference in how many requests the instances could handle did not differ too much as the difference never was around 10 percentage and never went over 12.5 percentage.

• In this study, the virtual machines uses up more than double the amount of memory that a container uses up which is a diﬀerence which favors the containers.

• When running two diﬀerent instances, the containers used up between 61.3 percentage and 79 percentage of the CPU that the virtual machines used up, depending on which test script was running.

5.6 Answering the second research question

The second research question was the following: ”How does the performance change between virtual machines and containers when scaling the application out to three and four diﬀerent instances, in regards to memory usage, CPU utilization, throughput and time to boot up?”. Scaling the application changes the performance diﬀerence even more, favoring the containers over the virtual machines:

• The bootup times for both the containers and the virtual machines remains the same as in the answer to the ﬁrst question.

• The difference in throughput increased, favoring the containers in two out of the three cases. The difference favored the containers when fetching information from a database and also when uploading files and the difference favored the virtual machines when the application was inserting new information into the database.

• As the application scales outwards, the diﬀerence in memory usage at least doubles for every new instance created.

• The CPU utilization diﬀerence gets a change when scaling outwards, fa- voring the containers.

(32)

5.7 Comparing the results to the related work

The results from my controlled experiment was that the containers are overall better in a performance perspective over the virtual machines, which all of the related works could agree on. The differences between my results and the related works is that the amount of requests handled in Vestmans [23] study favored the virtual machines while this thesis favored the containers. The reasons for this is most likely because we are using two entirely different applications and it could also be because of different network configurations. Another difference is that the my virtual machines only used up double the amount of memory as the containers while in Chae et al. [24] experiment, the virtual machines used up between 3.6 and 4.6 more memory. The reasons for this could be many but is again depending on the application used and also in what image that the virtual machines were using.

5.8 Validity threats

The results from this study should be replicable as long as you are using the same setup as I did, especially in regards to the limitations on the virtual machines and the networking conﬁguration in OpenStack with the same virtual machine image (Bionic Beaver 18.04) that I use to create my VM inside Open- Stack.

When looking at the validity threats in this experiment, I have looked at the four diﬀerent types described in ”Experimentation in software engineering” [30, p.102-104], conclusion, internal, construct and external validity. When it comes to conclusion validity, the conclusions drawn from the experiment is valid because there is a direct relationship in how the load on the application increases along with the memory, CPU and throughput. This experiment is valid in internal validity since the relationship between the treatment and the outcome is a causal relationship where my test scripts are the cause of the increased load.

When it comes to construct validity, this experiment is valid in this sense since the results corresponds well to the related works.

When looking at the external validity, the application used in the experiment becomes a concern considering that it is very lightweight which has an eﬀect on the result, especially in regards to the memory usage. A more heavyweight alternative would take up more memory which would make the diﬀerence in memory usage relatively smaller than in this study where the virtual machines took up around double the memory. More about this is covered in the ”Future Work” section.

(33)

6 Conclusion

This study was focused on looking at the performance diﬀerence between containers deployed through Docker and virtual machines deployed through Open- Stack from a scalability perspective. The reasoning behind the research was to ﬁnd out if the performance gap between the containers and virtual machines increases or decreases when scaling a lightweight application outwards in regards to Memory usage, CPU utilization, throughput and time to boot up the applications..

Research question 1: How does Docker containers and OpenStack virtual machines perform in comparison when running two applications that can handle database requests along with ﬁle uploads in regards of Memory usage, CPU utilization, throughput and time to boot up the applications?

Research question 2: How does the performance change between virtual machines and containers when scaling the application out to three and four diﬀerent instances, in regards to memory usage, CPU utilization, throughput and time to boot up?

The experiment was performed on two different virtual machines on VirtualBox that were running the operating system ”Ubuntu Server 18.04” with the Docker containers on one machine and the OpenStack environment with the virtual machines on the other machine. The application that the containers and virtual machines were running is a basic Flask application that can add information to a database, get information from it and also allows you to upload files to it that it stored on the applications file system.

When the containers and virtual machines were ready to go, test scripts were ran that each checks one diﬀerent part of the application at a time. The information needed to answer the research questions were gathered using ”htop” on the virtual machines and ”Docker stats” on the containers.

The results from the experiment shows that the containers overall performs better on all of the focused areas. With the virtual machines having much higher time to boot up than the containers and the memory usage being more than double the usage on the virtual machines than the containers. The difference in throughput is not huge when having between two different instances running but becomes a larger difference when scaling the application outwards. The difference in CPU utilization favored the containers heavily depending on which test script that was running and the performance gap increased when scaling the application outwards.

(34)

7 Future work

There are a lot of further studies that could be done based on the results from this study. Below are two examples of what could be done in a further study.

To further expand upon the experiment, it would give more accurate results if instead of scaling out to three or four diﬀerent instances, you scale out to 10 and 100 instances instead to give a more accurate result of what happens when scaling outwards.

Trying out the same experiment but instead of a lightweight framework like Flask, you could use a more heavyweight application that uses up more CPU and memory overall too see how the performance diﬀers between the containers and the virtual machines when using a more heavyweight alternative.

(35)

References

[1] environment. https://www.webopedia.com/TERM/E/environment.html, 2020. Accessed 2020-06-15.

[2] Welcome to neutron’s documentation! https://docs.openstack.org/

neutron/latest/, 2020. Accessed 2020-06-15.

[3] What is nginx? https://www.nginx.com/resources/glossary/nginx/, 2020. Accessed 2020-06-15.

[4] Virtual machine (vm). https://www.techopedia.com/definition/4805/

virtual-machine-vm, 2020. Accessed 2020-06-15.

[5] Peter Mell, Tim Grance, et al. The nist deﬁnition of cloud computing.

2011.

[6] Amazon ec2. https://aws.amazon.com/ec2/, 2020. Accessed 2020-05-15.

[7] Katie Costello. Gartner forecasts worldwide public cloud revenue to grow 17.5 percent in 2019, Apr 2019. Accessed 2020-02-24, [Online] Avail- able: https://www.gartner.com/en/newsroom/press-releases/2019-04-02- gartner-forecasts-worldwide-public-cloud-revenue-to-g.

[8] NCC Group. What is the cost of cloud computing?, Aug 2019. Accessed 2020-02-24, [Online] Available: https://www.nccgroup.trust/uk/about- us/newsroom-and-events/blogs/2019/august/what-is-the-cost-of-cloud- computing.

[9] Steﬀen Becker, Gunnar Brataas, and Sebastian Lehrig. Engineering Scalable, Elastic, and Cost-Eﬃcient Cloud Computing Applications: The CloudScale Method. Springer, 2017.

[10] Docker overview. https://www.kissclipart.com/

docker-architecture-clipart-docker-architecture-mi-iltxq8/, 2018. Accessed 2020-05-02.

[11] The future of linux containers. https://www.youtube.com/watch?v=

wW9CAH9nSLs, 2013. Accessed 2020-05-02.

[12] What is a container? https://docker.com/resources/what-container, 2020. Accessed 2020-05-02.

[13] Introduction: A bit of openstack history. https://docs.openstack.org/

project-team-guide/introduction.html. Accessed 2020-05-02.

[14] Openstack docs: Design. https://docs.openstack.org/arch-design/

design.html#logical-architecture. Accessed 2020-05-02. Licensed un- der CC BY 3.0.

(36)

[15] Welcome to glance’s documentation! https://docs.openstack.org/

glance/latest/, 2019. Accessed 2020-06-15.

[16] Openstack compute (nova). https://docs.openstack.org/nova/

latest/, 2020. Accessed 2020-06-15.

[17] Openstack block storage (cinder) documentation. https://docs.

openstack.org/cinder/latest/, 2020. Accessed 2020-06-15.

[18] Keystone, the openstack identity service. https://docs.openstack.org/

keystone/latest/, 2019. Accessed 2020-06-15.

[19] Doug Chamberlain. Containers vs. virtual machines (vms): What’s the difference? https://blog.netapp.com/blogs/containers-vs-vms/, 2018.

Accessed 2020-06-15.

[20] S. Shirinbab, L. Lundberg, and E. Casalicchio. Performance evaluation of container and virtual machine running cassandra workload. In 2017 3rd In- ternational Conference of Cloud Computing Technologies and Applications (CloudTech), pages 1–8, Oct 2017.

[21] Kyoung-Taek Seo, Hyun-Seo Hwang, Il-Young Moon, Oh-Young Kwon, and Byeong-Jun Kim. Performance comparison analysis of linux container and virtual machine for building cloud. Advanced Science and Technology Letters, 66(105-111):2, 2014.

[22] Luis Herrera-Izquierdo and Marc Grob. A performance evaluation between docker container and virtual machines in cloud computing architectures.

MASKANA, CEDIA, 8(1):127–133, 2017.

[23] Simon Vestman. Cloud application platform-virtualization vs containeriza- tion: A comparison between application containers and virtual machines, 2017.

[24] MinSu Chae, HwaMin Lee, and Kiyeol Lee. A performance comparison of linux containers and virtual machines using docker and kvm. Cluster Computing, 22(1):1765–1775, 2019.

[25] Oracle vm virtualbox. https://www.virtualbox.org/, 2020. Accessed 2020-05-15.

[26] Mysql. https://www.mysql.com/, 2020. Accessed 2020-05-15.

[27] Welcome to ﬂask – ﬂask documentation (1.1.x). https://flask.

palletsprojects.com/en/1.1.x/. Accessed 2020-06-15.

[28] Devstack. https://docs.openstack.org/devstack/latest/, 2020. Ac- cessed 2020-05-02.

[29] Heat and devstack. https://docs.openstack.org/heat/latest/

getting_started/on_devstack.html, 2020. Accessed 2020-05-02.

(37)

[30] Claes Wohlin, Per Runeson, Martin H¨ost, Magnus C Ohlsson, Bj¨orn Regnell, and Anders Wessl´en. Experimentation in software engineering.

Springer Science & Business Media, 2012.

(38)

Appendix

A Database Setup

A.1 completeSetup.sql

1 -- C r e a t e s the d a t a b a s e

2 C R E A T E D A T A B A S E

3 IF NOT E X I S T S c l o u d c o m p u t i n g ;

4

5 USE c l o u d c o m p u t i n g ;

6

7 DROP TABLE IF E X I S T S u s e r s ;

8

9 C R E A T E T A B L E u s e r s

10 (

11 u s e r n a m e V A R C H A R(30) ,

12 p a s s w o r d V A R C H A R(30)

13 ) ;

14 15

16 I N S E R T I N T O u s e r s

17 ( username , p a s s w o r d )

18 V A L U E S

19 (’ Hej ’, ’ 123 ’) ;

(39)

B PyFlask application

B.1 Index ﬁle

1 from f l a s k i m p o r t Flask , request , j s o n i f y

2 from f l a s k _ m y s q l d b i m p o r t M y S Q L

3 from w e r k z e u g . ut il s i m p o r t s e c u r e _ f i l e n a m e

4 app = Fl as k ( _ _ n a m e _ _ )

5

6 # N e e d s to be c h a n g e d on the v i r t u a l m a c h i n e s to the p r o p e r IP a d r e s s

7 app . c o n f i g [’ M Y S Q L _ H O S T ’] = ’ l o c a l h o s t ’

8 app . c o n f i g [’ M Y S Q L _ U S E R ’] = ’ root ’

9 app . c o n f i g [’ M Y S Q L _ P A S S W O R D ’] = ’ 1234 ’

10 app . c o n f i g [’ M Y S Q L _ D B ’] = ’ c l o u d c o m p u t i n g ’

11

12 app . c o n f i g [’ U P L O A D _ F O L D E R ’] = ’ t e s t F i l e s ’

13 app . c o n f i g [’ A L L O W E D _ E X T E N S I O N S ’] = set(

14 [’ txt ’, ’ pdf ’, ’ png ’, ’ jpg ’, ’ jpeg ’, ’ gif ’])

15

16 m y s q l = M y S Q L ( app )

17

18 host = " 1 2 7 . 0 . 0 . 1 "

19 port = 5000

20 21

22 def main () :

23 app . run ( host = host , port = port , debug = True )

24 25

26 @app . route (’ / ’)

27 def h e l l o _ w o r l d () :

28 r e t u r n ’ he l l o ’

29 30

31 @app . route (’ / g e t u s e r s ’)

32 def g e t _ u s e r s () :

33 cur = m y s q l . c o n n e c t i o n . c u r s o r ()

34 cur . e x e c u t e (" S E L E C T * FROM us e r s ")

35

36 data = cur . f e t c h a l l ()

37 cur . c l o s e ()

38

39 r e t u r n j s o n i f y ( data )

40 41

42 @app . route (’ / a d d u s e r ’, m e t h o d s =[’ POST ’])

43 def a d d _ u s e r () :

44 d e t a i l s = r e q u e s t . f o r m

45 u s e r n a m e = d e t a i l s [’ u s e r n a m e ’]

46 p a s s w o r d = d e t a i l s [’ p a s s w o r d ’]

47

48 cur = m y s q l . c o n n e c t i o n . c u r s o r ()

49 cur . e x e c u t e (" I N S E R T I N T O u s e r s ( username , p a s s w o r d ) V A L U E S (% s , % s ) ",

50 ( username , p a s s w o r d ) )

51

(40)

52 res = cur . f e t c h a l l ()

53 m y s q l . c o n n e c t i o n . c o m m i t ()

54

55 r e t u r n j s o n i f y ( res )

56 57

58 @app . route (’ / u p l o a d f i l e ’, m e t h o d s =[’ POST ’, ’ GET ’])

59 def u p l o a d _ f i l e () :

60 f = r e q u e s t . fi l e s [’ file ’]

61 f . save ( s e c u r e _ f i l e n a m e ( f . f i l e n a m e ) )

62 r e t u r n ’ file u p l o a d e d s u c c e s s f u l l y ’

63 64

65 if _ _ n a m e _ _ == " _ _ m a i n _ _ ":

66 main ()

(41)

C Docker setup

C.1 Dockerﬁle

1 FROM p y t h o n :3

2 W O R K D I R / usr / src / app

3

4 COPY r e q u i r e m e n t s . txt ./

5 RUN pip i n s t a l l -- no - cache - dir -r r e q u i r e m e n t s . txt

6

7 COPY . .

8

9 CMD [ " p y t h o n " , "./ api . py " ]

C.2 nginx conﬁguration

1 s e r v e r {

2 l i s t e n 80 d e f a u l t _ s e r v e r ;

3 l o c a t i o n / {

4 p r o x y _ p a s s h t t p :// api : 5 0 0 0 ;

5 }

6 }

(42)

D Test scripts

D.1 testget.py

1 i m p o r t r e q u e s t s as r

2 i m p o r t sys

3 i m p o r t time

4

5 ip = sys . argv [1]

6 c o u n t e r = 0

7 t_end = time . time () + 45

8

9 w h i l e time . time () < t _end :

10 r . get (" http :// "+ ip +" / g e t u s e r s ")

11 c o u n t e r += 1

12

13 p r i n t(" A m o u n t of r e q u e s t s : " + str( c o u n t e r ) )

D.2 testinsert.py

2 i m p o r t sys

3 i m p o r t time

4

6 c o u n t e r = 0

8

10 r . post (" http :// "+ ip +" / a d d u s e r ", {’ u s e r n a m e ’: ’ V i c t o r ’, ’ p a s s w o r d ’: ’ 123 ’})

11 c o u n t e r += 1

12

D.3 testﬁle.py

2 i m p o r t sys

3 i m p o r t time

4

6 c o u n t e r = 0

8 f i l e n a m e = sys . argv [2]

9

10 f i l e s = {’ file ’: ( filename , open( filename , ’ rb ’) ) }

11

13 r . post (" http :// "+ ip +" / u p l o a d f i l e ", f i l e s = fi l e s )

14 c o u n t e r += 1

15

Comparison between OpenStack virtual machines and Docker containers in regards to performance