Linköpings universitet
Linköping University | Department of Computer and Information Science
Master’s thesis, 30 ECTS | Datateknik
2021 | LIU-IDA/LITH-EX-A--21/044--SE
Automating software installation
for cyber security research and
testing public exploits in CRATE
Att automatisera mjukvaruinstallationer för
cybersäkerhets-forskning och testandet av publika angreppskoder i CRATE
Johan Hedlin
Joakim Kahlström
Supervisor : Niklas Carlsson Examiner : Andrei Gurtov
Upphovsrätt
Detta dokument hålls tillgängligt på Internet - eller dess framtida ersättare - under 25 år från publicer-ingsdatum under förutsättning att inga extraordinära omständigheter uppstår.
Tillgång till dokumentet innebär tillstånd för var och en att läsa, ladda ner, skriva ut enstaka ko-pior för enskilt bruk och att använda det oförändrat för ickekommersiell forskning och för undervis-ning. Överföring av upphovsrätten vid en senare tidpunkt kan inte upphäva detta tillstånd. All annan användning av dokumentet kräver upphovsmannens medgivande. För att garantera äktheten, säker-heten och tillgängligsäker-heten finns lösningar av teknisk och administrativ art.
Upphovsmannens ideella rätt innefattar rätt att bli nämnd som upphovsman i den omfattning som god sed kräver vid användning av dokumentet på ovan beskrivna sätt samt skydd mot att dokumentet ändras eller presenteras i sådan form eller i sådant sammanhang som är kränkande för upphovsman-nens litterära eller konstnärliga anseende eller egenart.
För ytterligare information om Linköping University Electronic Press se förlagets hemsida https://www.ep.liu.se/.
Copyright
The publishers will keep this document online on the Internet - or its possible replacement - for a period of 25 years starting from the date of publication barring exceptional circumstances.
The online availability of the document implies permanent permission for anyone to read, to down-load, or to print out single copies for his/hers own use and to use it unchanged for non-commercial research and educational purpose. Subsequent transfers of copyright cannot revoke this permission. All other uses of the document are conditional upon the consent of the copyright owner. The publisher has taken technical and administrative measures to assure authenticity, security and accessibility.
According to intellectual property law the author has the right to be mentioned when his/her work is accessed as described above and to be protected against infringement.
For additional information about the Linköping University Electronic Press and its procedures for publication and for assurance of document integrity, please refer to its www home page: https://www.ep.liu.se/.
© Johan Hedlin Joakim Kahlström
Abstract
As cyber attacks are an ever-increasing threat to many organizations, the need for con-trolled environments where cyber security defenses can be tested against real-world at-tacks is increasing. These environments, called cyber ranges, exist across the world for both military and academic purposes of various scales. As the function of a cyber range involves having a set of computers, virtual or physical, that can be configured to replicate a corporate network or an industrial control system, having an automated method of config-uring these can streamline the process of performing different exercises. This thesis aims to provide a proof of concept of how the installation of software with known vulnerabili-ties can be performed and examines if the software is vulnerable directly after installation. The Cyber Range And Training Environment (CRATE) developed by the Swedish Defence Research Agency (FOI) is used as a testbed for the installations and FOI-provided tools are used for launching automated attacks against the installed software. The results show that installations can be performed without Internet access and with minimal network traf-fic being generated and that our solution can rewrite existing software packages from the package manager Chocolatey to work with an on-premises repository with an 85% success rate. It is also shown that very few publicly available exploits succeed without any man-ual configuration of either the exploit or the targeted software. Our work contributes to making it easier to set up environments where cyber security research and training can be conducted by simplifying the process of installing vulnerable applications.
Contents
Abstract iii
Contents iv
List of Figures vii
List of Tables viii
List of Listings ix Glossary x Acronyms xi 1 Introduction 1 1.1 Motivation . . . 1 1.2 Aim . . . 2 1.3 Research questions . . . 2 1.4 Contributions . . . 2 1.5 Delimitations . . . 3 1.6 Disclaimer . . . 3 1.7 Thesis outline . . . 4 2 Background 5 2.1 Cyber Range And Training Environment (CRATE) . . . 5
2.2 Package managers for Windows . . . 6
2.2.1 Chocolatey . . . 7
2.2.2 Windows Package Manager . . . 7
2.2.3 Scoop . . . 7 2.2.4 Others . . . 7 2.3 Automation tools . . . 8 2.3.1 Ansible . . . 8 2.3.2 Chef . . . 9 2.3.3 Puppet . . . 10 2.3.4 Salt . . . 12
2.4 Virtual machine setup tools . . . 13
2.4.1 Boxstarter . . . 13
2.4.2 Packer . . . 14
2.5 Vulnerability-related naming schemes . . . 14
2.5.1 Common Platform Enumeration (CPE) . . . 14
2.5.2 Common Vulnerabilities and Exposures (CVE) . . . 14
2.8 SVED Visualization Tool (SVIZ) . . . 16
2.9 Related work . . . 17
3 Method 20 3.1 Automating software installation . . . 20
3.1.1 Package managers selection . . . 21
3.1.2 Automation tool selection . . . 22
3.1.3 Summary of the tool selection . . . 23
3.1.4 Chocolatey list of packages . . . 23
3.1.5 Feedback from installation . . . 24
3.1.6 Database . . . 24
3.1.7 Mapping to vulnerabilities . . . 24
3.1.8 Online installation tests . . . 27
3.1.9 Rate limiting and excessive use . . . 27
3.1.10 Internal repository . . . 28
3.1.11 Offline installation tests . . . 30
3.2 Selecting exploits for evaluation . . . 31
3.2.1 Version difference . . . 31
3.2.2 Name and vendor difference . . . 32
3.2.3 Exploit selection criteria . . . 33
3.3 Automatic testing of exploits . . . 33
3.3.1 Preparing VMs . . . 34
3.3.2 Creating an attack sequence . . . 34
3.4 Manual testing of exploits . . . 35
3.5 Vulnerable state . . . 36
4 Results 37 4.1 Automating software installation . . . 37
4.1.1 With online access . . . 37
4.1.2 Evaluating the reliability of Chocolatey’s online repository . . . 38
4.1.3 Internal repository . . . 39
4.2 Selecting exploits for evaluation . . . 40
4.3 Automatic testing of exploits . . . 40
4.4 Manual testing of exploits . . . 41
4.5 Vulnerable state . . . 42 5 Discussion 44 5.1 Results . . . 44 5.1.1 Internal repository . . . 44 5.1.2 First usage . . . 45 5.1.3 Exploit testing . . . 46 5.2 Method . . . 47
5.2.1 Alternative methods of performing automated software installations . . 47
5.2.2 Less focus on automated testing . . . 47
5.2.3 Source criticism . . . 47
5.3 Challenges . . . 47
5.3.1 Hitting the rate limit during downloading and testing packages . . . 47
5.3.2 Database . . . 48
5.3.3 Corrupt output from Ansible . . . 48
5.3.4 Strange URLs . . . 48
6.1 Research questions . . . 51
6.2 Future work . . . 52
6.2.1 Improving the mapping from program and version to CPE . . . 52
6.2.2 Improving the exploit suggestion process . . . 52
6.2.3 Further automating the internalization process . . . 53
Bibliography 54
List of Figures
2.1 Simplified illustration of CRATE . . . 6
2.2 Typical attack sequence in SVED . . . 16
2.3 Screenshot of the attack graph creator in SVED . . . 16
2.4 Screenshot of the attack graph section of SVIZ . . . 17
3.1 Information flow for automated software installation . . . 21
3.2 Internal repository using Sonatype Nexus 3 . . . 30
4.1 Output from the software installation process . . . 37
4.2 Installation result statistics . . . 38
4.3 Installation results over time . . . 39
List of Tables
2.1 Example of CPE entries . . . 14
3.1 Latest versions and their age (days old as of 2021-02-23) available from different package managers . . . 22
3.2 Number of versions available from different package managers . . . 22
3.3 Chocolatey’s comparison of automation tool integration . . . 23
3.4 Example of database structure . . . 24
3.5 Possible installation result codes . . . 26
3.6 Example of Chocolatey packages with CPEs and actions . . . 27
3.7 Example of version differences for pairs of program versions and CPE versions . . 31
3.8 Example of name and vendor differences . . . 33
3.9 Example of name and vendor differences with partial matches allowed . . . 33
4.1 Installation result statistics . . . 38
4.2 Summary of exploit results . . . 41
4.3 Manually tested exploits . . . 42
4.4 Manually tested exploits, reason for failure and required action . . . 42
A.1 Exploit results, file format . . . 59
A.2 Exploit results, server . . . 60
List of Listings
2.1 Ansible YAML example . . . 9
2.2 Chef Ruby example . . . 10
2.3 Puppet example based on the official documentation . . . 11
2.4 Puppet Bolt example . . . 12
2.5 Salt example . . . 13
3.1 Package list in JSON format . . . 21
3.2 JSON example from Ansible . . . 25
3.3 List of file extensions that are downloaded by the internalizer . . . 29
3.4 Pseudocode for the version difference metric . . . 31
3.5 Pseudocode for the name and vendor difference calculation . . . 32
4.1 Playbook to put Elasticsearch into a vulnerable state . . . 42
4.2 Playbook to put FreeSWITCH into a vulnerable state . . . 43
5.1 URL with embedded variable . . . 45
5.2 Part of the JSON file that got corrupted . . . 49
Glossary
Administration network (CRATE) . . . see control plane (CRATE), 5 Capture The Flag (CTF) Cyber security competition where teams race to acquire text snip-pets (flags) through hacking exercises. . . 17 Control plane (CRATE) The part of CRATE where supporting systems such as exercise con-trol services and virtual machine management are located. . . see also CRATE, 5 Cyber range An isolated environment where cyber attacks and other security research can be performed in a controlled environment. . . 1, 2, 3, 5, 17, 18, 44, 49 Event plane (CRATE) The part of CRATE where vulnerable machines are located and where environments are created for various exercises. Isolated from all non-event plane net-works and the Internet. . . see also CRATE, 5, 15, 30 Game network (CRATE) . . . see event plane (CRATE), 5 Penetration testing A process where a group of people attempt to hack into a company’s systems (with their permission) for the purpose of finding and reporting issues that an attacker could take advantage of. . . 15 Regular expression (regex) A language used to specify patterns that can be matched against a given text input. . . 29 VirtualBox A program for managing and running virtual machines. . . 20, 23, 27, 30, 40, 41
Acronyms
API Application Programming Interface . . . 5, 15, 20, 23, 27, 28, 30, 40, 48 CLI Command Line Interface . . . 23 CPE Common Platform Enumeration . . . 14, 18, 24, 26, 29, 31, 32, 33, 34, 40, 52 CRATE Cyber Range And Training Environment . . . 1, 2, 5, 6, 12, 13, 14, 15, 17, 18, 20, 27,
28, 30, 34, 35, 40, 41, 47, 49
CSV Comma-Separated Values . . . 26 CTF Capture The Flag . . . see Capture The Flag (CTF), 17 CVE Common Vulnerabilities and Exposures . . . 14, 17, 18, 52, 53 CVSS Common Vulnerability Scoring System . . . 14 CWE Common Weakness Enumeration . . . 14 CyRIS Cyber Range Instantiation System . . . 18 DSL Domain-Specific Language . . . 9, 10 EULA End User License Agreement . . . 35, 45 FOI Swedish Defence Research Agency . . . 1, 2, 3, 5, 6, 14, 15, 16, 17, 23, 26, 28, 30, 32, 40,
47, 51, 52
GUI Graphical User Interface . . . 15, 19 HTTP Hyper Text Transfer Protocol . . . 5, 15, 38, 40, 48 ICMP Internet Control Message Protocol . . . 34 IP Internet Protocol . . . 27, 35 JSON JavaScript Object Notation . . . 20, 24, 48, 51 LAN Local Area Network . . . 42 MSI Windows Installer . . . 6 NAT Network Address Translation . . . 27 NVD U.S. National Vulnerability Database . . . 14, 52 OS Operating System . . . 1, 18, 34 PDF Portable Document Format . . . 45, 46
RDP Remote Desktop Protocol . . . 5 SCADA Supervisory Control And Data Acquisition . . . 1 SMB Server Message Block . . . 46 SSH Secure SHell . . . 5, 11, 12, 20 SVED Scanning, Vulnerabilities, Exploits and Detection . . . 15, 16, 33, 34, 35, 40, 41, 47 SVIZ SVED Visualization Tool . . . see also SVED, 16, 35, 40 URL Uniform Resource Locator . . . 21, 28, 29, 40, 44, 48, 53 VM Virtual Machine. . . 13, 14, 16, 28, 30, 34, 40 VPN Virtual Private Network . . . 5 VXLAN Virtual eXtensible LAN . . . see also LAN, 34 WinRM Windows Remote Management . . . 11, 12, 20, 30 XML eXtensible Markup Language . . . 28 YAML YAML Ain’t Markup Language (recursive acronym) . . . 8, 9, 11, 12
1
Introduction
This chapter presents an introduction to what the project is about and the reason behind it.
1.1
Motivation
As modern society is connected to the Internet in a wider range than ever before, there is also an increasing threat of cyber attacks [1, 2]. Systems that previously have not been connected to the Internet before such as Supervisory Control And Data Acquisition (SCADA) units are increasingly connected to meet a demand for remote control [3] and concepts like smart cities [4] introduce a wider attack surface [5]. During the last year, the demand to be able to work from home has increased due to the COVID-19 pandemic [6], which introduces yet another attack surface [7]. To counter this threat, multiple projects are in progress to increase both education and research within the cyber security area around the world [8, 9, 10, 11]. There are also efforts to improve the resilience against attacks on the underlying protocols that connect industrial devices [12]. Cyber ranges are great for doing controlled research and practice as they are isolated from the Internet and have the potential to provide the necessary resources that are needed as well as flexibility for different scenarios [13].
The Swedish Defence Research Agency (FOI) has an objective to increase the cyber defense capabilities of Sweden, where education and research are included. To reach this objective, FOI maintains the Cyber Range And Training Environment (CRATE) which is used to provide an environment where cyber attacks and exploits can be tested and demonstrated in a safe environment [14]. The cyber range can be set up with a large number of virtual machines that can, for example, simulate different components in a business network. When testing different exploits it is common for these to only work on a few versions of a vulnerable piece of software, thus necessitating the ability to install specific versions of many different applications in an efficient way. Today, FOI usually installs applications either by including them in the Operating System (OS) images that are deployed to the virtual machines, or through post-deployment scripts that have to be custom-made for each set of applications. Both of these solutions are rather inflexible, as preparing one OS image for every set of applications would lead to inefficient usage of disk space and take a lot of time to create. Creating installation
1.2. Aim
that the virtual machine contains. As Windows lacks a built-in package manager (although this might be about to change [15]), this would require a number of additional steps to either perform the installation through scripting or by first installing a third-party package manager. By using a bare-bones base image that only contains the operating system and necessary remote management tools, applications could be installed automatically without having to create multiple images or custom scripts for each combination of software. FOI has created tools and databases that can query which exploits a given application and version is vulnera-ble to. These can likely also be used for mapping published software vulnerabilities to their corresponding applications and exploits. Having the ability to automatically install the correct software and version to test a given vulnerability would make it faster to demonstrate it for a client or use it as part of an exercise, and it would make it easier to test a large number of exploits without having to manually install each piece of software.
1.2
Aim
The purpose of this thesis is firstly to create a program that can take in a list of applications that should be installed, and then perform the installation of these. By having this program also include information about the relevant vulnerabilities and exploits for each installable application, it should then be possible to install known-vulnerable versions as well. As the targeted machines will be located inside an isolated cyber range with restricted Internet access, the created program should also be able to function without Internet access. It should also minimize the amount of network traffic to these machines to avoid leaving traces that can interfere with exercises inside CRATE. Since publicly available attacks usually target older versions of applications, it is also important that the applications that can be installed today are also available several years in the future when new attacks against them might have been published. Directly relying on installer packages from the Internet might thus create problems if binaries and other installation dependencies are later removed by a software vendor.
A secondary aim of the thesis is to utilize this program’s ability to create a vulnerable system, taking advantage of the cyber range available at FOI to test a large number of exploits against a set of machines that have been configured with a large set of vulnerable programs. This serves to validate that the program can be used on a larger scale, as well as provide some data about the accuracy of publicly available exploits and if applications are vulnerable right after installation or if they usually require some configuration first.
1.3
Research questions
RQ1: How can older, potentially vulnerable software be stored reliably to support replicable security research?
RQ2: How can the installation process of such software be automated for scalable tests? RQ3: Is the software in a vulnerable state from the point of installation and if not, how can it
be placed into such a state?
1.4
Contributions
1.5. Delimitations
• Developing a tool that automatically installs a given set of software using the automation tool Ansible and the Chocolatey package manager. We show that software can be stored in an internal repository for stability and installed automatically into virtual machines in a cyber range without the need for Internet access.
• Providing an insight into how reliable community-made Chocolatey packages are over a period of six years, and developing a tool that can semi-automatically internalize existing packages such that they can be stored reliably for later use and be used without Internet access.
• Using automated attacks from the Metasploit framework to show that a set of auto-matically selected applications with known vulnerabilities are mainly not vulnerable to attacks right after they have been installed. A method of placing these into a vulnerable state as part of the installation process is also demonstrated for a small set of packages.
1.5
Delimitations
To keep the scope of the thesis focused on creating a process and prototype for automated software installation, the work will only focus on installing software on Microsoft Windows 10. Other versions of Windows such as Windows 8 or Windows 7 are also of interest and should work with small or no modifications to the prototype, but at least initially only one operating system will be targeted. Linux-based machines are also in use at FOI, and they would like to see support for Linux as well in the future, but for now, it would require a large amount of effort to maintain support for both platforms.
Regarding the exploit testing process, it will be divided into two methods. Automated testing will be limited to the types of attacks that can be automated efficiently. Reading through exploit scripts written by many different authors and trying to install dependencies, modifying the scripts to suit each target, and finding the correct parameters would take too much time to allow for large-scale testing of exploits. However, this will be done for a few selected programs as part of manual testing. This thesis will focus on using exploitation frameworks that include many scripts that conform to a given standard and can be configured through parameters that share the same name across all exploits. Exploits that require manual interaction on the targeted system will only be considered in manual testing unless these actions are of a kind that can be automated. Copying a file to the target system can be automated quite easily, but sending a sequence of keyboard inputs or mouse clicks will be more difficult since each program would require a different set of inputs. FOI has tools that can automate many common tasks and sequences in an exploitation attempt, including copying files to the target and executing them, and these will be utilized to test as many exploits as possible. Developing exploits from scratch or using standalone exploit modules which do not follow a common structure will be considered out of scope for this thesis.
1.6
Disclaimer
The authors of this thesis do not take a stand on whether certain package managers contain more vulnerable software than others. Any existing program could potentially be vulnerable, and a larger selection of software will also lead to a larger amount of possible vulnerabilities in this software. Since a vulnerability is usually patched as soon as possible after it has been discovered, all package managers which provide up-to-date versions of their programs are equal in terms of risk to the end-user. The availability of older, vulnerable versions does not
1.7. Thesis outline
1.7
Thesis outline
Chapter 2 presents the thesis background, the project’s environment, and the tools that are used. Tools related to security testing are then presented followed by related work. Chap-ter 3 contains a comparison between different tools for automation and program installation, presents the process of developing the automated software installer, how exploits are auto-matically chosen against a set of applications, the method of converting packages for offline use, and how exploits were tested against the applications. Chapter 4 contains the result of attempting to install and internalize a large set of applications, and the outcome of launching hundreds of attacks against a subset of these. Chapter 5 discusses the result and method along with some interesting observations and challenges. In Chapter 6 the conclusions are discussed and the research questions are answered. Lastly, thoughts about future work are presented.
2
Background
In this chapter, the project’s environment together with the tools used is presented. First, the environment and tools regarding the installation process are presented. Second, tools regard-ing the security tests used are presented. Third, related work within the area is presented.
2.1
Cyber Range And Training Environment (CRATE)
The Cyber Range And Training Environment (CRATE) is a cyber range that allows for a large number of virtual machines to be set up to simulate a large-scale network with multiple virtual organizations and industrial systems. Consisting of approximately 800 physical servers, it is developed and used by FOI for use in demonstrations, training, and other exercises [14].
The virtual machines in CRATE are split into an isolated game network, where all the vulnerable hosts and attacker-controlled machines are located, and an administration net-work, which interfaces the game network to the outside world and manages the state and configuration of all virtual machines located inside the game network. The administration network can be interfaced with using a set of Hyper Text Transfer Protocol (HTTP) Applica-tion Programming Interfaces (APIs), which can help automate the setup of large networks. A simplified illustration with the relevant information for this project can be seen in Figure 2.1. Each relevant part shown in the figure is discussed in the following sections and chapters.
In the figure, the administration network is called the control plane and the game network is known as the event plane. The easiest way to access the control plane is by using Secure SHell (SSH) through a Virtual Private Network (VPN), and the event plane by using either SSH or the Remote Desktop Protocol (RDP) tunneled through the physical nodes (which are themselves accessed through the control plane). The workstations that are used for develop-ment are located outside of this network, with access to both the Internet and CRATE. They are shown in the top-left of Figure 2.1.
2.2. Package managers for Windows
Database server
Internet
Virtualization servers
SVED SVED injector
Pegasus Chocolatey repo Our workstations
VPN tunnels
SRVproxy
Control plane Event plane
Virtual machines
Figure 2.1: Simplified illustration of CRATE
2.2
Package managers for Windows
While package managers that can install software without interaction from the user are the rec-ommended way of acquiring software in Linux distributions such as Debian [16], on Windows systems it is either downloaded directly from the vendor or through the Microsoft store [17]. While the Microsoft store provides a unified method of installing software from multiple ven-dors, automating the installation of packages from it does not seem straightforward1. Some software can be distributed in a format that allows for unattended or silent installations, such as Microsoft’s Windows Installer (MSI) format which has a /quiet option for performing the installation without displaying a user interface [18]. Other applications might be distributed in formats that use a non-standardized way of performing a silent install or might lack the option of a silent install altogether [19].
Since installing applications automatically is desirable for many system administrators [20], solutions have appeared that aim to overcome these issues by using custom-made installation procedures and scripts tailored for each application’s specific way of performing a silent install. Some of these solutions are discussed in the following sections. Only free-to-use solutions are considered since many paid editions of these programs are licensed either user or per-machine that the software is deployed to, of which the latter would be prohibitively expensive for FOI due to CRATE containing up to thousands of virtual machines. A user-based licensing model might be more feasible, but could also be very expensive and difficult to manage if every user that wants to create virtual machines in CRATE with a custom set of software each require their own license.
1 https://serverfault.com/questions/1018220/how-do-i-install-an-app-from-windows-store-using-powershell
2.2. Package managers for Windows
2.2.1
Chocolatey
Chocolatey2provides a set of tools that can be used to package standalone binaries, scripts, or installers into a unified format that can then be installed, uninstalled, or upgraded through their command-line utility [21]. There is also an officially supported repository3of community-made packages available for many common desktop applications and services such as Adobe Acrobat Reader, Google Chrome, VLC media player, Python 3, MySQL Server, and others. As of 2021-02-08, the repository contains 8 173 packages [22]. For many of these, there are also older versions available, which is of interest since many exploits are only applicable to certain versions before the vulnerabilities they exploit are patched. For example, the oldest version of Mozilla Firefox available in the repository is version 15.0 from 2012. However, due to issues with distribution rights the Chocolatey repository often cannot contain the installer binaries themselves, instead fetching them directly from the application vendor at install time [23]. Thus, it is not guaranteed that these old versions can still be installed. If manual packaging is an option, packages that include these installers can be created and hosted on a local repository for internal use.
2.2.2
Windows Package Manager
The Windows Package Manager4 (also known as WinGet) is a tool from Microsoft that is currently available as a beta version. Like the other package managers, it is capable of in-stalling packages through a command-line interface and retrieves the programs to be installed directly from the vendor [24]. Several common desktop applications such as Adobe Acrobat Reader, Google Chrome, VLC media player, and Python 3 are available from the community repository5, which as of 2021-02-08 contains 1 249 packages [25].
2.2.3
Scoop
Scoop6has around 2 013 packages available in its repository as of 2021-02-08 [26, 27], with a mix of developer-oriented command-line tools and applications with a graphical interface. Many common programs such as web browsers and media players are available. It lacks good support for older versions since all applications have one metadata file each, preventing two versions with the same program name from existing. While a repository of different versions is available7with 129 packages, it mainly contains beta versions and major releases of interpreters like PHP: Hypertext Preprocessor (PHP), Python, and Node.js where a new version might be incompatible with older code or make other large changes. The versions here have also had their names truncated a bit, so a package named ”python39” might install Python 3.9.2. Scoop also installs packages for the local user only by default, which could create problems for exploits that assume that programs are installed in a specified location, but packages seem to be able to be installed globally as well by supplying a command-line parameter.
2.2.4
Others
Several other package managers are also available, but many of these either do not target Windows software or have too limited of a selection of packages available.
2https://chocolatey.org/ 3https://chocolatey.org/packages 4https://docs.microsoft.com/en-us/windows/package-manager/ 5https://github.com/microsoft/winget-pkgs 6https://scoop.sh/ 7https://github.com/ScoopInstaller/Versions/tree/master/bucket
2.3. Automation tools
Ninite
Ninite8offers a selection of about 90 applications (as of 2021-02-08) [28] that can be installed by selecting which applications should be installed and then downloading a single program that installs them with minimal user interaction. However, this selection needs to be made on Ninite’s website and the resulting installer cannot be run silently without purchasing a per-machine license. The installer always installs the latest version of each selected application, even if the installer itself was downloaded a long time ago, making this solution poorly suited for replicability purposes based on the motivations and aims discussed in Sections 1.1 and 1.2. AppGet
AppGet9 contains about 1400 packages as of 2021-02-08 [29], but like Scoop it also lacks support for older versions due to the same architectural choice of only having one metadata file per application. On 2020-08-01 the service was shut down permanently due to Microsoft releasing WinGet (see Section 2.2.2) [29]. Since it is open-source and the package repository is still available for anyone to reuse for other projects, it would be possible (with some work) to install packages from AppGet, but compared to the alternatives that are actively maintained and offer older versions it might not be a good candidate for this thesis.
NuGet
NuGet10is another package manager that allows for automated installation of packages and is also what Chocolatey is built on. In contrast to Chocolatey, its repository11mainly deals with libraries for .NET development [30], and is thus of limited use for regular users. A version of Firefox seems to be available, but only a selected handful of versions exist, the newest of which being two years old. As of 2021-02-08, the repository contains 227 263 packages [31].
2.3
Automation tools
Several tools exist for installing software automatically and managing a large number of machines. Some of the most common ones are detailed in this section together with some examples of the configuration syntax for each of them. While most of these examples are tailored for Linux-based systems, they can be adapted to work on Windows as well.
2.3.1
Ansible
Ansible12is an automation tool currently maintained by Red Hat. It aims to automate system deployment, software installation, and other configuration in a simple-to-use way without requiring specialized agent software to be installed on the managed machines [32]. Custom actions can be implemented in Python if the included selection of 3365 modules13 proves insufficient. The YAML Ain’t Markup Language (YAML) format is used to define which actions should be performed on a set of machines. A file containing a set of such actions, structured into a list of tasks, is called a playbook. An example of a short playbook containing two tasks can be seen in Listing 2.1.
8https://ninite.com/ 9https://appget.net/ 10https://www.nuget.org/ 11https://www.nuget.org/packages 12https://www.ansible.com/ 13https://docs.ansible.com/ansible/2.9/modules/list_of_all_modules.html
2.3. Automation tools
- name: Configure web servers hosts: webservers
tasks:
- name: Install latest version of Apache apt:
name: apache2 state: latest
- name: Update Apache config file template:
src: files/apache_web.conf dest: /etc/apache2/apache2.conf
Listing 2.1: Ansible YAML example
2.3.2
Chef
Chef14, or Chef Infra, is another automation tool that is developed by Progress. It focuses on solving the problem of managing multiple machines in an organization. It is divided into three parts: a workstation from where the administrator can initiate changes and administer the network, a Chef server that collects the cookbooks and translates the instructions from the workstation into code and policies and distribute them towards the clients. The Chef clients are the nodes, computers, servers, or virtual machines that should be controlled [33].
This means that Chef uses a master-agent model where the administrator must upload the instructions, called recipes (or a cookbook if there are multiple recipes), to a server which in turn can convert the instructions into code that is sent to each machine. Each machine has an agent that receives the instructions and executes them. The instructions that are used to construct the recipes are written in Ruby Domain-Specific Language (DSL), which could allow for a more complex set of instructions than YAML but are also more complex to set up and grasp for users not familiar with Ruby. In its foundation it is open-source, but the company also offers an enterprise version where server hosting is provided and with additional support for a cost [34]. Listing 2.2 shows an example of a recipe for setting up an Apache server using Ruby.
2.3. Automation tools
node.default['main']['doc_root'] = "/vagrant/web"
execute "apt-get update" do command "apt-get update"
end
apt_package "apache2" do action :install
end
service "apache2" do
action [ :enable, :start ] end
directory node['main']['doc_root'] do owner 'www-data'
group 'www-data'
mode '0644'
action :create
end
cookbook_file "#{node['main']['doc_root']}/index.html" do source 'index.html' owner 'www-data' group 'www-data' action :create end template "/etc/apache2/sites-available/000-default.conf" do source "vhost.erb"
variables({ :doc_root => node['main']['doc_root'] }) action :create
notifies :restart, resources(:service => "apache2") end
Listing 2.2: Chef Ruby example [35]
2.3.3
Puppet
Puppet15is also a good candidate for automation. The main objective of puppet is to solve the problem of managing multiple servers and computers, much like Ansible and Chef. It is also open-source and written in Ruby. Similar to Chef, it uses an agent for each client that receives and executes the instructions. The instructions that are executed are written in Puppet’s DSL, also called Puppet Code. See Listing 2.3 for a sample of the syntax. This code is used to describe the desired state of the system, not how to achieve that state [36]. Puppet then applies the necessary steps to achieve this state using the agent. However, as the objective of this project is to only set up a certain environment and not maintaining its state, some of Puppet’s features might be irrelevant for this thesis.
2.3. Automation tools
case $operatingsystem {
centos, redhat: { $service_name = 'httpd' } debian, ubuntu: { $service_name = 'apache2' } } package { 'apache2': ensure => installed, } service { 'apache2': name => $service_name, ensure => running, enable => true,
subscribe => File['apache2.conf'], }
file { 'apache2.conf':
path => '/etc/apache2/apache2.conf', ensure => file,
require => Package['apache2'],
source => "puppet:///modules/apache2/apache2.conf",
# This source file would be located on the primary Puppet server at # /etc/puppetlabs/code/modules/apache2/files/apache2.conf
}
Listing 2.3: Puppet example based on the official documentation [37]
Puppet Bolt
Puppet Bolt16 is a newer solution from the Puppet developers which is agentless and can use SSH or the Windows Remote Management (WinRM) protocol to connect to machines in a network [38]. It is an open-source project which aims to perform as-needed orchestration directly from a local workstation. Scripts could be executed directly from the terminal or structured into YAML files much like how Ansible does it. See Listing 2.4 for an example of such a file. This could be a potential candidate as it is agentless and seems easy to integrate into an infrastructure.
2.3. Automation tools parameters: targets: type: TargetSpec steps: - name: install_apache task: package targets: $targets parameters: action: install name: apache2
description: "Install Apache using the packages task"
Listing 2.4: Puppet Bolt example [38]
2.3.4
Salt
Salt17or Saltstack, is an open-source automation tool now owned by VMware and is based on Python. It is a powerful and scalable automation tool that uses ZeroMQ for fast and flexible communication with the managed nodes [39]. There is also the possibility for a node to act as a proxy for another node so that proprietary systems or resource-limited machines can be managed as well [40]. Salt has a structure with a Salt master and several Salt minions, where the Salt master controls the Salt minions that are located on each machine as an agent. As it uses YAML code to structure its instructions, it is very similar to the other alternatives. An example can be seen in Listing 2.5.
VMware also offers Salt SSH, which is an agentless version of Salt that executes the com-mands using the SSH protocol. This however negatively affects the speed of Salt, as it loses the benefits of ZeroMQ. This would be an acceptable loss, but unfortunately Salt SSH does not seem to be supported on Windows unless you buy the enterprise version, which also uses WinRM instead of SSH [41]. This enterprise version seems to be part of VMware’s vRealize Automation offering, which costs around 1 400 SEK each year per managed node [42]. This would be very costly with the hundreds of machines in CRATE and is thus not an option.
2.4. Virtual machine setup tools apache2: pkg.installed apache2 Service: service.running: - name: apache2 - enable: True - require: - pkg: apache2 Turn Off KeepAlive:
file.replace:
- name: /etc/apache2/apache2.conf - pattern: 'KeepAlive On'
- repl: 'KeepAlive Off'
- show_changes: True - require: - pkg: apache2 /etc/apache2/conf-available/tune_apache.conf: file.managed: - source: salt://files/tune_apache.conf - require: - pkg: apache2
Listing 2.5: Salt example [43]
2.4
Virtual machine setup tools
In addition to the more general automation tools presented so far, there are also solutions that aim to provide a method of setting up a virtual machine with a certain configuration or set of applications. While these sound like an excellent fit for the use case described in 1.1, they come with some limitations that might pose a problem for future expansion or architectural changes.
2.4.1
Boxstarter
Boxstarter18, being created by the company Chocolatey Software, aims to simplify the deploy-ment of Windows machines by automatically installing a given set of Chocolatey packages in these. It can install packages on a remote system and handle a couple of common issues that might prevent packages from installing, such as pending reboots, other concurrent installation processes, Windows updates being installed simultaneously, or disk encryption being enabled [44]. One difficulty however is that it needs to run on a Windows system and can only manage other Windows systems. Since the physical nodes in CRATE are Linux-based, this means that Boxstarter would have to be run locally on each Virtual Machine (VM) and be controlled via Ansible. It would also not be possible to utilize it to control Linux/Unix-based systems in the future.
2.5. Vulnerability-related naming schemes
2.4.2
Packer
Packer19 aims to automate the creation of virtual machine images for different platforms and can as part of this process also run other automation tools such as those mentioned in Section 2.3 to install software or make other configuration changes [45]. While this would perhaps be a good solution to automate the deployment of virtual machines in CRATE, it would also require that large parts of the existing infrastructure around preparing these are replaced. For this reason, it might be a good option to consider if the deployment process is to be redesigned in the future, but for this thesis it would likely lead to incompatibilities and issues if the created installation tool should be capable of installing packages on VMs that have already been set up by FOI’s existing tools.
2.5
Vulnerability-related naming schemes
2.5.1
Common Platform Enumeration (CPE)
Common Platform Enumerations (CPEs) are a method of representing software products and packages in a structured way [46]. Each CPE entry contains the type of product (e.g. operating system or application), the vendor, product name, and version. The version field can be omitted or replaced with an asterisk if no specific version of the product is referenced. There can also be information about a specific hardware platform, edition, language, etc. included in the CPE, all of which can also be omitted if desired. The current revision of the CPE standard (as of the beginning of 2021) is v2.3 from 2011 [47], but older v2.2 IDs are still common. An example of CPEs for a few applications can be seen in Table 2.1.
Software CPE v2.3
Mozilla Firefox 60 cpe:2.3:a:mozilla:firefox:60.0:*:*:*:*:*:*:* VideoLAN VLC 3.0.12 cpe:2.3:a:videolan:vlc_media_player:3.0.12:*:*:*:*:*:*:* Windows 10 2004 64-bit cpe:2.3:o:microsoft:windows_10:2004:*:*:*:*:*:x64:*
Table 2.1: Example of CPE entries
2.5.2
Common Vulnerabilities and Exposures (CVE)
Common Vulnerabilities and Exposures (CVEs) are a way of assigning a unique ID to a pub-lished vulnerability in software. This ID usually takes the form of CVE-YYYY-NNNNNN, where YYYY is the year the vulnerability became publicly known (or when the CVE was requested, if earlier) and a unique number NNNNNN of 4 digits or more. Each CVE entry contains a description of the vulnerability, and references to relevant information such as security advisories or vendor websites [48]. The U.S. National Vulnerability Database (NVD) also maintains a database with additional information for every CVE, namely a Common Vulnerability Scoring System (CVSS) score indicating the severity of the vulnerability, a Com-mon Weakness Enumeration-ID (CWE) that indicates what category of flaw in the source code that gave rise to the vulnerability (such as out-of-bounds write, use-after-free, improper authentication. etc.), and affected CPEs [49].
2.6. Metasploit
2.6
Metasploit
Metasploit20 is a penetration testing framework developed by Rapid7 that contains many modules for scanning networks, exploitation, obfuscation, and more [50]. These modules are written in Ruby and conform to a standardized specification that makes it easy to run modules as part of an automated process. As of 2021-02-26, there are 2102 modules available in the ”exploits” category, which all aim to exploit either network-based or local vulnerabilities in applications or embedded systems. The exploits are further divided into groups depending on their method of delivery, or what type of service they target. Remote exploits are launched against a network service without user interaction, while browser exploits attempt to exploit a user’s web browser. File format exploits are delivered as a file that the user needs to open with the vulnerable program.
There are also 592 payloads that can be delivered via an exploit to perform some operation. There are payloads for many operating systems and programming languages that can be chosen depending on the exploit target, e.g. an operating system service or a web app. Many of these platforms also have multiple payloads for different actions such as creating a shell that connects back to the attacker, creating a remote desktop session, adding a user account, rebooting the machine, formatting the hard drive, or playing a message over the speakers using text-to-speech.
2.7
Scanning, Vulnerabilities, Exploits and Detection
(SVED)
Scanning, Vulnerabilities, Exploits and Detection (SVED) is a framework for designing and executing chains of attacks against any number of hosts in an automated way [51]. It is developed by FOI and used in CRATE for automating attacks using the Metasploit framework (see Section 2.6). An attack in SVED can consist of several actions and flow control that determines which action should be taken depending on the result of a previous one. For example, an attack against a network can start with a port scan of a certain machine. If port 80 (used for HTTP traffic) is open and the returned version matches an expected value, an exploit can be executed against that machine. If successful, the next action could be set up to gather passwords from the compromised machine which are then used to attack a database server, and so on. These attack chains could also include redundancy for failed attacks, such that another service is targeted instead of HTTP if no web server is running, or another machine might be selected as the target. These chains of events are set up manually, but FOI has also created a prototype of another framework that uses artificial intelligence to automatically select the best course of action for every given situation. This should allow for more automated exploitation of networks in CRATE, but so far it is not viable to use in practice.
For the purpose of this thesis, the main parts of SVED that are relevant are the administra-tive interface and the injectors. The administraadministra-tive interface contains an API and a Graphical User Interface (GUI) for creating sequences of actions, and the injectors are virtual machines that can automatically be placed into the correct network segment inside CRATE’s event plane for the purpose of actually launching the attacks that are directed by the administrative part of SVED. Both the SVED manager service and an injector are shown in the center of Figure 2.1. Also see Figure 2.3 for an example of this interface. An overview of the targeted network is displayed in the left-hand pane, and attacks and actions can be added and manipulated in the
2.8. SVED Visualization Tool (SVIZ)
right-hand pane. The puzzle-piece icon marked with a red background is the starting point of the attack graph. Depending on the result of the action, execution can continue along either the green arrow for a successful action or the red for a failed one. The displayed example shows a small segment of a larger attack test, where a typical sequence of events is followed as described in Figure 2.2. The VM is first restored from a previously taken snapshot, then the injector is placed into the correct network segment, after which it checks if the target is responding, and finally it launches the exploit. If it succeeds, the established session is logged as a success and then destroyed, and then (or if any step fails) the VM is again restored from a snapshot in preparation for the next attack.
Restore VM Prepare injector Check target VM Launch exploit Collect logs
Restore a snapshot of the VM's state taken before any attacks were launched
Place the injector into the same network as the target so that the injector can communicate with it
Attempt to ping the target VM to see if it is functioning correctly
The injector launches the exploit against the target machine
Logs are collected from the injector and stored for later analysis
Figure 2.2: Typical attack sequence in SVED
Figure 2.3: Screenshot of the attack graph creator in SVED
2.8
SVED Visualization Tool (SVIZ)
To analyze the logs generated from SVED, the SVED Visualization Tool (SVIZ) is used to gain an understanding of which actions and attacks succeeded and which failed. SVIZ was developed by Bedhammar and Johansson as part of their thesis project at FOI and can process the log files emitted by SVED to display the results in relation to their corresponding action in SVED [52]. See Figure 2.4 for an example of the interface. Exploits are represented with triangles and supporting actions such as resetting VMs are represented by cogwheels. Suc-cessfully completed actions are displayed in green, pending ones in yellow, and failed actions are marked with red. By using this tool, locating interesting events in a larger set of attacks is significantly easier, especially when having multiple machines running in parallel.
2.9. Related work
Figure 2.4: Screenshot of the attack graph section of SVIZ
2.9
Related work
Cyber range testing is an area that is currently evolving around the world [10, 13]. It’s not a new idea, but it is highly relevant now that we are connected online more than ever. The need for an isolated space where cyber security and cyber defense could safely be tested and practiced is increasing. A cyber range is also a suitable place to arrange Capture The Flag (CTF) competitions which are useful to gain practical experience within different areas of cyber security. There is development in progress within academia, the industry, and the military, some with more open information available than others [11]. This section will process some of the related work that is openly available.
At FOI there has been a lot of research already using CRATE. Holm and Sommestad [53] used CRATE to do an empirical study of how reliable automated attacks are with as little modification as possible to show if the need for advanced knowledge of systems and computers are necessary to perform a hack, or if a person without specialized knowledge can succeed thanks to the advancement of ready-made offensive cyber tools. The study performed 1223 exploitation attempts using 45 unique exploits on a total of 204 virtual machines, where the exploits were chosen based on the result of automatic vulnerability scanning. Only eight of the launched exploit attempts succeeded, all of which were from a single unique exploit module.
Gustafsson and Almroth studied in their article [54] the tools used in the automation of cyber ranges. First, they look at the current status and research within the field of automation for cyber ranges. Then they describe how CRATE is built, what automation tools are used within this system, and what role the tools have in research and training. Later the authors compare which automated tools are currently used at different cyber ranges and state which parts of each cyber range are automated. This article concludes that automation has been utilized within cyber ranges for many years, but there is still a need for further improvements to be able to meet the demand for research and training capabilities.
The task of finding relevant published vulnerabilities given a software’s name and version has been examined in earlier studies. Since CVEs are of interest to systems administrators that wish to keep the machines under their control safe from vulnerabilities, Sanguino and
2.9. Related work
Uetz [55] have examined the possibility of using an inventory of installed software products to automatically pick the best matching CPEs for each installed software. This would then be combined with automated monitoring of newly published CVEs to alert an administrator if a new vulnerability is published for a CPE that matches an installed piece of software. The authors use the edit distance between two strings to compare installed programs against CPEs, but also take advantage of the fact that their input data has separate fields for vendor and software name. This allows the matching algorithm to only match name Ø name and vendor Ø vendor instead of accidentally allowing matches between vendor and name from different software products. The authors conclude that using a fully automated assignment of CPEs in this fashion leads to many instances where the matching algorithm assigns erroneous identifiers because of differences in naming conventions between the application vendor and the CPE directory. Thus, a manual review of the suggested candidates from the automated system was necessary to maintain an inventory of CPEs that could be used for matching against the feed of new vulnerabilities.
Pham et al. propose the Cyber Range Instantiation System (CyRIS) [56] as a tool to create and set up a cyber range automatically. In many ways, it is similar to how CRATE works, with the use of virtual environment and setup automation. However, it is limited to one OS as of publication, CentOS 7, and is mainly focused on setting up specific training scenarios. CRATE already has many parts implemented and is more versatile in that it is used for both research and training in different scales. Parts of the work could be of interest but would be of more use in the setup of Linux machines, which are not in the scope of this thesis.
There are several methods for automating software installation as described in Section 2.2 which all rely on a preexisting repository of scripts for passing the correct flags or options to each piece of software. If the desired software is not available in such a repository, it would have to be packaged in some way to allow for unattended installation. A study of different methods of achieving this [57] examines four ways that applications could be automatically installed. A script that sends keyboard and mouse input to interact with the software installer is the basis for three of these, where the method of generating the script differs. It can either be created by hand, by automatically finding text and buttons that according to a ruleset indicate which stage the installation process is in and which action should be taken, or by monitoring a user installing the software and recording the actions taken by that user. The fourth method presented is to again let a user install the program, but record all files and registry entries that are created and packaging these into a bundle that can later be recreated on a target machine. The author concludes that the methods that rely on interacting with the installer might be too time-consuming for the process to be worth the effort, or too unreliable in different environments where parameters like screen resolution might change the position of a button from what the script is expecting. They instead suggest that the method involving recording changes to the filesystem and the registry is the most reliable one due to it not needing to interact with the software installer at all.
While a software repository with already written scripts for installing packages would likely be the most convenient solution, these alternative methods of installing applications can be more relevant when it comes to automatically preparing the software for exploitation attempts or general use. As stated in the research questions (Section 1.3), the software that is to be installed in CRATE should ideally be configured such that they are vulnerable to the attacks that are to be tested against them. This would in most cases include emulating a normal work-ing environment where the programs have been started at least once, as opposed to a fresh installation where a license agreement might be displayed before the program is launched. This could interfere with the attack if, for example, the program does not expose certain func-tionality (or even launch) until the agreement has been accepted. From an overview of the installation scripts supplied in the repositories of existing package managers for Windows (see Section 2.2) it seems like these are only concerned with installing the application and not with the initial launch or configuration of it. Using a script or a registry + filesystem snapshot
2.9. Related work
like the author examines [57] could be a solution for configuring applications and software packages to be in the proper state for them to be vulnerable to attacks.
A study by Dashevskyi et al. [58] aims to provide a solution for creating environments for security testing in a repeatable way. The authors mainly focus on web applications and related exploits, with Docker21containers being used instead of virtual machines to host the vulnerable services. Configuration of these can be done through the use of Dockerfiles, which can contain instructions on how to build the containers via shell commands and the copying of files from the host system. This approach could likely be used for other applications as well, but might not be a good fit for client-side applications since Docker containers are supposed to run isolated from both each other and the rest of the system [59]. Combined with the limited support that Docker has for Windows, this could make it difficult to run regular desktop applications that use GUIs or use other types of functionality that might not be accessible from inside a container.
3
Method
This chapter presents the method that was used, the process of building the tool and testing the result.
3.1
Automating software installation
The contents of this section primarily serve as a method of answering RQ2, but they also contain information that is relevant for the other research questions.
The information flow that this work aims towards can be seen in Figure 3.1, where the administrator specifies in JavaScript Object Notation (JSON) format which programs each machine should have installed. An example is shown in Listing 3.1. This should then be processed to be compatible with an automation tool that orchestrates the installation process. As CRATE utilizes VirtualBox to host the virtual machines, it would be preferable if it would be possible to let the communication go through VirtualBox’s API, as this would reduce the amount of network traffic generated that could disturb exercises. The use of SSH or WinRM could be used if the VirtualBox API does not work. The automation tool then instructs the package manager on the virtual machine what programs to install. Logs from the installation should then be transferred back to the administrator in JSON format.
To decide which package manager and automation tool would be suitable for this project, those presented in Sections 2.2 and 2.3 will be compared in the following section.
3.1. Automating software installation { "vm_name": "win10-ent2004-x64-off2013.src", "username": "Administrator", "password": "admin_password", "packages": [ { "name": "vlc", "version": "3.0.14", "platform": "win_chocolatey" }, { "name": "firefox", "version": "89.0", "platform": "win_chocolatey" }, { "name": "git", "version": "2.31.1", "platform": "win_chocolatey" } ] }
Listing 3.1: Package list in JSON format
Physical Administrator
Control Plane Event Plane
Node
Virtual machines
Virtual machines
Figure 3.1: Information flow for automated software installation
3.1.1
Package managers selection
A comparison of the latest available versions and their age for a few applications is shown in Table 3.1. Some package managers have packages that refer to installers or Uniform Resource Locators (URLs) which point to the latest version available from the software vendor, these are marked with ”latest” in the table. These packages do not need to be updated by their author whenever a new version of the packaged application is released, but it becomes difficult to determine which version will be installed since the one specified in the package’s description or metadata will not be accurate.
While having recent versions of software available is important for many users, the avail-ability of older versions is of interest for the questions examined in this thesis. Since as many exploits as possible should be able to be tested, there is a need for a large set of older versions to increase the chance that a vulnerable version that matches the exploit is found. A compari-son of the availability of older versions of a few common applications is shown in Table 3.2. Note that Google Chrome is only available as an auto-updating package from Chocolatey and WinGet, meaning that it is not possible to install older versions since all older versions
3.1. Automating software installation
Package manager
Google Chrome Mozilla Firefox LibreOffice VLC
Version Age Version Age Version Age Version Age
Chocolatey latest 1 85.0.2 14 7.1.0 20 3.0.12 37
WinGet latest 1 84.0 70 7.0.1 179 3.0.11 253
Scoop 87.0.4280.88 83 85.0.2 14 7.1.0.3 17 3.0.12 37
AppGet latest 1 latest 0 6.4.2.2 334 3.0.11 253
Ninite 88.0.4324.190 1 85.0.2 14 7.1.0 20 3.0.12 37
Table 3.1: Latest versions and their age (days old as of 2021-02-23) available from different package managers
in the repositories still download the latest version of Chrome during the installation process. From Table 3.2 it is clear that Chocolatey offers the largest selection of versions, and thus it is the most relevant for use in this project. While the presence of older versions in the package repository is not a guarantee that they all work, as some applications may have been removed by the vendor or otherwise broken over time, having the ability to attempt an installation of an older version is better than only being able to install the latest version.
As mentioned in Section 1.6, the presence of older versions, which might contain vulner-abilities, does not at all indicate that a certain package manager is more vulnerable or worse than another. A regular user should typically always install the latest available stable version of each package, which is the default for all the examined package managers. The availability of older versions does not constitute a security risk to regular users of these managers.
Scoop, AppGet, and Ninite all lack the ability to specify a version when installing a pack-age, which limits the number of available versions to one. Installing older versions with Scoop or AppGet could be done by extracting older versions of a package’s metadata from the ver-sion control systems used by these tools to manage their software repositories. However, this requires additional work if two applications from different points in time are to be installed at the same time, and also complicates the process of locating the correct package given a desired software name and version.
Package manager
Google Chrome Mozilla Firefox LibreOffice VLC
Versions Versions Versions Versions
Chocolatey 1 192 42 47
WinGet 1 18 9 2
Scoop 1 1 1 1
AppGet 1 1 1 1
Ninite 1 1 1 1
Table 3.2: Number of versions available from different package managers
3.1.2
Automation tool selection
The automation tools listed in Section 2.3 all present similar solutions to the same problem: performing a sequence of actions automatically on a set of remote machines. Thus, the choice of which one to use is mostly dependent on the need for special features, such as the
prefer-3.1. Automating software installation
to define actions, or which have been used earlier for other projects. In this case, existing work has already been done at FOI using Ansible in combination with VirtualBox, and we have some previous experience with Ansible and Python. The Chocolatey developers have also compared different automation tools that Chocolatey works well with [60]. As seen in Table 3.3, Ansible and Puppet both support all examined features from Chocolatey. As Ansible has support for all the examined features and is agentless, these facts make it a fitting solution to use for the rest of the thesis.
Configuration managers Ansible Chef Puppet Salt
Manage packages 3 3 3 3
Install Chocolatey 3 3 3 3
Install Chocolatey from internal source 3 3 3 7
Manage sources 3 3 3 3
Manage source type 3 7 3 7
Manage features 3 7 3 7
Manage config settings 3 3 3 7
Table 3.3: Chocolatey’s comparison of automation tool integration [60]
3.1.3
Summary of the tool selection
From Sections 3.1.1 and 3.1.2 it can be seen that Ansible and Chocolatey seem fitting for use in this thesis, and they will therefore be the basis for the rest of the work described in this chapter. While tools like these can be used in combination to install programs automatically, FOI would like a tool that can produce a report on what software and versions are available to be installed, how the installation went, what errors if any were encountered, etc. in a format that can be integrated into another interface. To avoid making large changes to this future integration, it would also be beneficial to have this tool act as a unified interface to several package managers if one lacks some desired applications or other operating systems are to be supported. This means that Ansible and Chocolatey can be used to do the heavy lifting, with our developed tool serving as a wrapper to these that exposes the needed functionality in a machine-readable format. If needed, this construction also allows for the replacement of Ansible/Chocolatey if another tool is later determined to be a better option.
3.1.4
Chocolatey list of packages
While FOI had an existing list of 1 615 Chocolatey packages with different versions for 149 unique packages, this was a small subset of the thousands of packages available in Choco-latey’s community repository. There was also more information available about each package, such as the name of the vendor, a description, publication date, and more, which could be use-ful to have. Unfortunately, this information is not easily gathered without making thousands of requests to Chocolatey’s web API.
Through monitoring the traffic generated when using Chocolatey’s Command Line In-terface (CLI) tools to view available packages, it was discovered that it sends around 700 requests totaling 120 MB of data to display the names and latest version of around 5000 pack-ages. Based on a few package samples, it was determined that each package has around 10 versions on average, which would require 2 000 requests to download all packages and ver-sions. Since this would be equivalent to making three "list packages" calls with the CLI tools, it was deemed non-invasive enough that it would be feasible. A one-second delay was intro-duced between each request to decrease the impact on Chocolatey’s servers, and the entire
