Linköpings universitet SE–581 83 Linköping
Linköping University | Department of Computer and Information Science
Master thesis, 30 ECTS | Datateknik
2018 | LIU-IDA/LITH-EX-A--18/012--SE
Penetration testing for the
in-experienced ethical hacker
–
A baseline methodology for detecting and mitigating
web application vulnerabilities
Penetrationstestning för den oerfarne etiska hackaren
–
En gedigen grundmetodologi för detektering och mitigering av
sårbarheter i webbapplikationer
Per Lindquist
Henrik Ottosson
Supervisor : Jan Karlsson, Ulf Kargén Examiner : Nahid Shahmehri
Detta dokument hålls tillgängligt på Internet – eller dess framtida ersättare – under 25 år från publiceringsdatum under förutsättning att inga extraordinära omständigheter uppstår. Tillgång till dokumentet innebär tillstånd för var och en att läsa, ladda ner, skriva ut enstaka kopior för enskilt bruk och att använda det oförändrat för ickekommersiell forskning och för undervisning. Överföring av upphovsrätten vid en senare tidpunkt kan inte upphäva detta tillstånd. All annan användning av dokumentet kräver upphovsmannens medgivande. För att garantera äktheten, säkerheten och tillgängligheten finns lösningar av teknisk och admin-istrativ art. Upphovsmannens ideella rätt innefattar rätt att bli nämnd som upphovsman i den omfattning som god sed kräver vid användning av dokumentet på ovan beskrivna sätt samt skydd mot att dokumentet ändras eller presenteras i sådan form eller i sådant sam-manhang som är kränkande för upphovsmannens litterära eller konstnärliga anseende eller egenart. För ytterligare information om Linköping University Electronic Press se förlagets hemsida http://www.ep.liu.se/.
Copyright
The publishers will keep this document online on the Internet – or its possible replacement – for a period of 25 years starting from the date of publication barring exceptional circum-stances. The online availability of the document implies permanent permission for anyone to read, to download, or to print out single copies for his/hers own use and to use it unchanged for non-commercial research and educational purpose. Subsequent transfers of copyright cannot revoke this permission. All other uses of the document are conditional upon the con-sent of the copyright owner. The publisher has taken technical and administrative measures to assure authenticity, security and accessibility. According to intellectual property law the author has the right to be mentioned when his/her work is accessed as described above and to be protected against infringement. For additional information about the Linköping Uni-versity Electronic Press and its procedures for publication and for assurance of document integrity, please refer to its www home page: http://www.ep.liu.se/.
c
Per Lindquist Henrik Ottosson
Abstract
Having a proper method of defense against attacks is crucial for web applications to ensure the safety of both the application itself and its users. Penetration testing (or ethical hacking) has long been one of the primary methods to detect vulnerabilities against such attacks, but is costly and requires considerable ability and knowledge. As this expertise remains largely individual and undocumented, the industry remains based on expertise. A lack of comprehensive methodologies at levels that are accessible to inexperienced ethical hackers is clearly observable.
While attempts at automating the process have yielded some results, automated tools are often specific to certain types of flaws, and lack contextual flexibility. A clear, simple and comprehensive methodology using automatic vulnerability scanners complemented by manual methods is therefore necessary to get a basic level of security across the entirety of a web application. This master’s thesis describes the construction of such a methodology. In order to define the requirements of the methodology, a literature study was per-formed to identify the types of vulnerabilities most critical to web applications, and the applicability of automated tools for each of them. These tools were tested against various existing applications, both intentionally vulnerable ones, and ones that were intended to be secure.
The methodology was constructed as a four-step process: Manual Review, Testing, Risk Analysis, and Reporting. Further, the testing step was defined as an iterative process in three parts: Tool/Method Selection, Vulnerability Testing, and Verification. In order to ver-ify the sufficiency of the methodology, it was subject to Peer-review and Field experiments.
thålla säkerheten i webbapplikationer, både vad gäller applikationen själv och dess använ-dare. Penetrationstestning (eller etisk hacking) har länge varit en av de främsta metoderna för att upptäcka sårbarheter mot sådana attacker, men det är kostsamt och kräver stor per-sonlig förmåga och kunskap. Eftersom denna expertis förblir i stor utsträckning individuell och odokumenterad, fortsätter industrin vara baserad på expertis. En brist på omfattande metodiker på nivåer som är tillgängliga för oerfarna etiska hackare är tydligt observerbar.
Även om försök att automatisera processen har givit visst resultat är automatiserade verktyg ofta specifika för vissa typer av sårbarheter och lider av bristande flexibilitet. En tydlig, enkel och övergripande metodik som använder sig av automatiska sårbarhetsverk-tyg och kompletterande manuella metoder är därför nödvändig för att få till en grundläg-gande och heltäckande säkerhetsnivå. Denna masteruppsats beskriver konstruktionen av en sådan metodik.
För att definiera metodologin genomfördes en litteraturstudie för att identifiera de typer av sårbarheter som är mest kritiska för webbapplikationer, samt tillämpligheten av automatiserade verktyg för var och en av dessa sårbarhetstyper. Verktygen i fråga tes-tades mot olika befintliga applikationer, både mot avsiktligt sårbara, och sådana som var utvecklade med syfte att vara säkra.
Metodiken konstruerades som en fyrstegsprocess: manuell granskning, sårbarhetstest-ning, riskanalys och rapportering. Vidare definierades sårbarhetstestningen som en iterativ process i tre delar: val av verkyg och metoder, sårbarhetsprovning och sårbarhetsverifier-ing. För att verifiera metodens tillräcklighet användes metoder såsom peer-review och fältexperiment.
Acknowledgments
The authors of this thesis would like to do the following acknowledgements:
Ulf Kargén, thank you for great academic supervision and excellence in answering our ques-tions during our thesis work.
Jan Karlsson and Secure State Cyber, thank you for great support and the possibility to be able to complete this work.
Peter Österberg, Staffan Huslid and Jonas Lejon, thank you for your intelligent answers during our industry and expert interviews. Your insight in this field helped us tremendously. Edward Nsolo, thank you for bringing constructive criticism and knowledge by proof reading and questioning.
Lastly, the authors would like to thank each other for great support, hard work and excel-lent company.
Abstract iii Acknowledgments v Contents vi List of Figures ix List of Tables x 1 Introduction 1 1.1 Motivation . . . 2 1.2 Aim . . . 2 1.3 Method Overview . . . 2 1.4 Research questions . . . 2 1.5 Delimitations . . . 3
1.6 Secure State Cyber . . . 3
2 Notable Vulnerabilities 4 2.1 National Vulnerability Database (NVD) . . . 4
2.2 Open Web Application Security Project (OWASP) Top 10 . . . 5
1 - Injection (I) . . . 5
2 - Broken Authentication (BA) . . . 5
3 - Sensitive Data Exposure (SDE) . . . 6
4 - XML External Entities (XXE) . . . 6
5 - Broken Access Control (BAC) . . . 7
6 - Security Misconfiguration (SM) . . . 7
7 - Cross-Site Scripting (XSS) . . . 8
8 - Insecure Deserialization (ID) . . . 8
9 - Using Components with Known Vulnerabilities (UCwKV) . . . 9
10 - Insufficient Logging and Monitoring (IL&M) . . . 9
3 Penetration Testing 11 3.1 Information Gathering . . . 11
3.2 Vulnerability Testing . . . 12
3.3 Risk Assessment . . . 13
3.4 Reporting . . . 13
4 Existing Tools, Methods and Environments 14 4.1 Methods of Analysis . . . 14
4.2 Vulnerability scanning requirements . . . 15
FedRAMP . . . 15
4.3 Existing Tools . . . 16
Zed Attack Proxy . . . 16
Burp Suite Community Edition . . . 16
Sqlmap . . . 16
IronWASP . . . 17
Vega . . . 17
Arachni . . . 17
Wapiti . . . 17
4.4 Related Works: Tools being tested against special environments . . . 17
4.5 Existing Methods . . . 18 Manual Crawling . . . 18 Logical Flaws . . . 19 Vulnerability Verification . . . 19 4.6 Existing Environments . . . 19 DVWA . . . 19 OWASP Bricks . . . 19 WACKOPICKO . . . 19 Peruggia . . . 19 OrangeHRM . . . 20
Bila Nu & Presentshoppen . . . 20
5 Method 21 5.1 Literature study . . . 21
Interviews . . . 22
5.2 Requirements . . . 22
5.3 Selection, Testing and Evaluation of Tools and Features . . . 23
5.4 Comprehensive Methodology . . . 24
5.5 Design of Interactive Algorithm . . . 24
5.6 Verification of Comprehensive Methodology . . . 24
Empirical Validation . . . 24 Peer-Review . . . 24 Field Experiment . . . 25 6 Results 26 6.1 Literature Study . . . 26 6.2 Requirements . . . 26
6.3 Selection, Testing and Evaluation of Tools and Features . . . 27
VEGA . . . 27
Zed Attack Proxy . . . 27
Arachni . . . 27
Sqlmap . . . 27
IronWASP . . . 28
Wapiti . . . 28
BURPsuite Community Edition (CE) . . . 28
Non-viable Tools . . . 28
6.4 Comprehensive Methodology . . . 28
Manual Review . . . 29
Tools and Method Selection . . . 30
Vulnerability Testing and Verification . . . 30
Risk Analysis . . . 31
Reporting . . . 32
6.5 Design of Interactive Algorithm . . . 32
Field Experiment . . . 33
7 Discussion 34 7.1 Results . . . 34
Literature Study and Sources . . . 34
Interviews . . . 34
Requirements . . . 34
Selection, Testing and Evaluation of Tools and Features . . . 35
Comprehensive Methodology . . . 36
Design of Interactive Algorithm . . . 36
Verification of Comprehensive Methodology . . . 36
7.2 Method . . . 37
Literature Study and Sources . . . 37
Requirements . . . 37
Selection, Testing and Evaluation of Tools and Features . . . 37
Comprehensive Methodology . . . 38
Design of Interactive Algorithm . . . 38
Verification of Comprehensive Methodology . . . 38
7.3 The work in a wider context . . . 38
Existing Methodology . . . 38
Ethical Aspect: The potential for malicious use . . . 39
Societal Aspect: Increasing ease of penetration testing . . . 39
Societal Aspect: Increasing Cybersecurity Threats . . . 39
8 Conclusion 40 8.1 Main Contributions . . . 40
8.2 Research Question Summary . . . 40
8.3 Future Work . . . 41
A Appendix 1 - Vulnerability Scanner evaluation 42 B Appendix 2 - Interviews with industry experts 45 Interview with Peter Österberg, 2018-02-19 . . . 45
Interview with Staffan Huslid, 2018-02-26 . . . 47
Interview with Jonas Lejon 2018-03-21 . . . 48
C Appendix 3 - OrangeHRM Field Experiment 50 Manual Review . . . 50
Tools and Method Selection . . . 51
Vulnerability testing and Verification . . . 51
Risk Analysis . . . 51
Reporting . . . 51
List of Figures
3.1 Risk Severity . . . 13 6.1 Comprehensive Methodology . . . 28
2.1 Log States . . . 10
4.1 Scanners and OWASP top 10 vulnerability mapping as claimed by each scanner’s documentation . . . 17
4.2 Environment-Vulnerability mapping . . . 20
6.1 Tools and their respective fulfillment of requirements . . . 28
A.1 Initial testing . . . 42
A.2 Wackopicko . . . 43
A.3 DVWA . . . 43
1
Introduction
The digital age has resulted in a considerable number of practical web applications, giving users access to large quantities of information and functionality. With this increase of avail-ability, the concerns about application security increase accordingly, as concluded by Post et al. [1] in 1991, before many of the concepts behind the modern web application had been introduced. Ease of access to an application could be considered inversely correlated to ap-plication security, in the sense that an increase in the exposure of an apap-plication is likely to create new attack vectors [2].
Still, organizations tend to mainly design their applications to be as user-friendly, acces-sible and visually interesting as posacces-sible, which may well result in the introduction of new security flaws, as pointed out by professional penetration tester Peter Österberg [2] in an in-terview conducted as part of this thesis. Accordingly, Jang et al. [3] reported findings on 12 practical attacks that they claim were made possible specifically by the introduction of cer-tain accessibility features in modern operating systems. As attackers work quickly to discover and exploit such vulnerabilities, owners of web applications need to act even faster towards detecting and resolving those same vulnerabilities. With the speed at which the situation evolves, David Talbot [4] from the MIT Technology Review notes that organizations wishing to protect their sensitive data can find it increasingly hard to even determine the level of security they have achieved at any given time.
Of course, there are professionals available to assist with such matters. There are two distinct titles within the industry that describe the concept of non-malicious hacking, the title of ethical hacker1, and the title of penetration tester. Universities2,3 and other organizations4 have created educations to be able to meet the increasing market demands for this type of professional [2].
Moreover, there are organizations dedicated to the structural analysis of common vul-nerabilities and attack vectors, such as the non-profit Open Web Application Security Project (OWASP). OWASP maintains a Top 10 list of the security issues currently considered the most notable, such as Injections, Cross-Site Scripting and Sensitive Data Exposure [5].
1https://en.oxforddictionaries.com/definition/ethical_hacker 2EP3370: https://www.kth.se/student/kurser/kurs/EP3370?l=en 3EN2720: https://www.kth.se/student/kurser/kurs/EN2720?l=en
Due to the considerable variation of potential attacks, purely manual testing tends to be-come unwieldy, especially in larger applications, as noted by Kupsch et al. [6]. Purely auto-mated testing, on the other hand, has its own set of issues, regardless of the specific field in which it is applied. While effective as a time saving effort in largely unchanging and repet-itive situations, automated systems tend to lack sufficient flexibility, which creates the need for skilled operators to manually verify the results [6].
Unfortunately, skilled operators may soon become short supply, given the increased mar-ket demands [7], [8]. As the penetration testing profession remains largely individualistic and experience-driven, testers generally choose their own tools and work according to their own methods [9], which causes an almost complete lack of documented general standards, both regarding methodology and results. Due to this, an upcoming generational shift may lead to knowledge loss among penetration testers, as senior experience is not sufficiently transferred to juniors [2].
1.1
Motivation
While penetration testing is an established field, the expertise remains largely undocumented. Studies have been performed regarding specific (or closely coupled) vulnerabilities and how to neutralize them, but no comprehensive methodology appears to have been published re-garding the combination of the existing automated tools with manual penetration testing. In order to allow inexperienced testers to perform at acceptable levels, comprehensive standard methodologies, upon which expertise and specialization may be built, must be introduced.
1.2
Aim
This master’s thesis is expected to result in a methodology for web application security anal-ysis, including automated and manual tools and methods, covering common critical flaws, as per industry definitions. In order to enhance the credibility of the methodology, it will be verified by security experts.
1.3
Method Overview
The general approach will be as follows:
1. Perform a literature study of common attack vectors and their impacts, based on estab-lished industry data.
2. Create a set of requirements for complete vulnerability detection and mitigation ser-vices, including both automated and manual tools and methods.
3. Research and investigate currently existing automated services and manual methods for performing vulnerability analysis of web applications.
4. Based on the identified automated services and manual methods, develop a compre-hensive methodology that could be used to get a full overview of critical vulnerabilities, including recommended mitigating measures.
5. Verify the validity of the solution.
1.4
Research questions
This master’s thesis is intended to answer the following questions: • What notable vulnerabilities are most critical for web applications?
1.5. Delimitations
• For what types of vulnerabilities are existing automated scanners suitable to use? • What manual methods are needed to complement the automated services to develop a
comprehensive methodology?
• How can the effectiveness of the methodology be verified?
1.5
Delimitations
This master’s thesis will not involve the construction of any new vulnerability scanners or specific methods, nor concern any matters of design and development of new web applica-tions, nor will it concern matters of information security outside of web application bound-aries, such as social engineering or physical security.
The methodology presented in the Master’s thesis is intended for use by knowledgeable but inexperienced penetration testers, and will therefore require basic information security and penetration testing knowledge. It will not go into technical detail regarding the use of any tools or methods included within it.
1.6
Secure State Cyber
Secure State Cyber5, for which this thesis work was performed, specializes in cyber secu-rity and information secusecu-rity. They offer certified cybersecusecu-rity services based on the cus-tomer’s needs and conditions. They help customers solve complex issues in a structured way throughout all stages of the cybersecurity process, to identify, protect, detect, respond and recover.
Secure State Cyber was founded in 2005 and operates from their headquarters in Sweden. At the beginning of 2018, they had roughly 20 employees.
There are many potential threats against web applications and many ways in which they can be important to consider, be it due to high frequency, high potential harm, or the ease of acting them out. In order to provide a comprehensive methodology for detection and mitigation, the most critical and common threats must be defined. In this chapter, organizations devoted to the documentation and ranking of vulnerability types are introduced. Descriptions of these vulnerabilities, and ways they can be specifically counteracted, are also detailed.
2.1
National Vulnerability Database (NVD)
The Department of Homeland Security’s National Cyber Security Division1has sponsored an open-source database for common vulnerabilities. It is based on predefined standards for identification (Common Vulnerabilities and Exposures (CVE)) and rating (Common Vulner-ability Scoring System (CVSS)).
These two standards have been used to pioneer other vulnerability databases and CVE is maintained by the non-profit organization Mitre Corporation2, while CVSS is an open indus-try standard3.
The CVE4is a list of known vulnerabilities. Each entry details the specific technical vul-nerability, a practical description of how the vulnerability can be exploited, and references to relevant documentation. Each such vulnerability is given an CVE number and added to the list when reported. The primary purpose of this is to allow for independent security or-ganizations and tools to have a common reference to specific issues, removing the need for extensive internal databases, and easing communication between entities which may each have given the same issue a unique designation. While extensive and detailed, the CVE li-brary is generally of limited use for web application security assessment, as such systems are rarely prominent or large enough to warrant inclusion [2].
1NIST, National Vulnerability Database:
https://www.nist.gov/programs-projects/national-vulnerability-database-nvd
2Mitre Corporation: https://mitre.org/ 3CVSS: https://nvd.nist.gov/vuln-metrics/cvss 4CVE: http://cve.mitre.org/
2.2. Open Web Application Security Project (OWASP) Top 10
In turn, CVSS5provides a way to calculate a vulnerabilities severity numerically, based on a number of factors.
2.2
Open Web Application Security Project (OWASP) Top 10
The OWASP Top 10 list of critical risks, which according to penetration tester Peter Österberg [2] covers 95-99% of actually occurring vulnerabilities, was updated in November of 20176. Notable changes from the 2013 version7include the introduction of three new issues, XML External Entities, Insecure Deserialization and Insufficient Logging Monitoring [5]. The cur-rent list is as follows:
1 - Injection (I)
Injection flaws generally allow attackers to submit data which is interpreted as commands to be executed by the system but may also be used in attempts to overflow buffers or cause other harm to a system [5]. Depending on the design of the system, injection flaws include various types, such as SQL, Xpath and LDAP injection [10]. Such attacks are suitable to launch across networks and are often used to gain access or create insertion vectors for various types of malware or further attacks [11].
Countermeasures and detection
There are established methods of protection against injection type attacks, including param-eterizing statements and validating input [10], [12]. By constructing limitations of what is accepted as input, the potential for harmful content being processed is limited. Such valida-tion can also be achieved with the use of whitelisting, accepting only input which follows an established set of rules with regards to traits such as content and size.
Other methods focus less on the rejection of suspicious input, opting instead for prevent-ing the system from executprevent-ing the received instructions, such as the one proposed by Gaurav et al. [11], which employs the use of randomized instruction sets. This method is proclaimed to protect against any type of injection, by making the executable code unique to the partic-ular environment, preventing attackers from providing accepted instructions to the system, without first having access to the randomization key.
While manual penetration testing against injection flaws is entirely possible, attacks of this kind are easiest to perform using automated tools, as confirmed by the proprietary web vulnerability scanner vendor Netsparker [13].
2 - Broken Authentication (BA)
Broken Authentication refers to an event where an attacker successfully deduces the creden-tials of other users through any of a multitude of methods, such as by stealing keys, session to-kens or passwords [5]. Often, simple password deduction is achieved by brute-forcing log-in credentials using external tools in combination with known vulnerabilities and/or dictionar-ies of common passwords [14], bypassing the need to actually intercept any communication. Countermeasures and detection
Huluka et al. [15] has researched the root causes of broken authentication. They have iden-tified some causes, including lack of security knowledge, mixing of confidential data with non-confidential data, and the use of in-house developed functions instead of well tested
5CVSS: https://www.first.org/cvss/
6OWASP Top 10 2017: https://www.owasp.org/images/7/72/OWASP_Top_10-2017_(en).pdf.pdf 7OWASP Top 10 2013: https://www.owasp.org/images/f/f8/OWASP_Top_10_-_2013.pdf
external alternatives. Strong and secure defence mechanisms can be very costly and cause inconveniences for both owners and users, but the consequences can be severe if they are not implemented. Suggestions on what a proper defence mechanism should include are: vali-dation to ensure that users use strong passwords, avoiding the use of session ids in URLs, using image recognition tasks to detect non-human behavior, and not giving attackers useful information in error messages [15], [16]. In addition to this, broken authentication also con-cerns how passwords are stored, whether a good hashing algorithm is being used, and if the passwords are salted [13].
Automated scanners are able to find some divergence from safe patterns for some of these mechanisms, such as session IDs or passwords being sent in clear text, but many of the vulnerabilities require logical analysis to discover, based on behavior which cannot be detected/verified by an automated scanner [13]. As such, while automated tools are able to discover some blatantly insecure circumstances, they cannot be relied upon to provide suffi-cient security on their own, requiring additional manual analysis.
3 - Sensitive Data Exposure (SDE)
Without specific protection, sensitive data is at risk of interception. Such data may include financial details and health records, possibly allowing for a variety of further crimes if stolen [5]. According to Risk Based Security [17], the yearly number of exposed records saw a marked increase in the early 2010s, most notably between 2012 and 2013 (an increase by 212%). Countermeasures and detection
The potential exposure of sensitive data is an important aspect of any data handling system. Poorly encrypted or even unencrypted data is at great risk from a number of types of attack. In order to prevent such issues, some basic methods exist [5]: Firstly, sensitive data should be stored to as small an extent as possible, depending on the purpose of the application in question. If the data is not stored, it cannot be stolen from the repository. Secondly, any data that is stored must be sufficiently encrypted. Thirdly, to prevent interception attacks, any sensitive data must also be sufficiently encrypted during transit.
Apart from those types of data which is required by applicable law to be especially pro-tected, such as social security numbers, credit card details and user credentials, exactly what constitutes sensitive data is highly subjective, and depends on each individual stakeholder. While automated scanners are able to scan for patterns matching legally protected structures, the subjectivity of the matter makes automation a very limited solution [13]. In cases where known sensitive data is stored, transmitted, or displayed, policies are likely already in place. It is therefor much simpler to review documentation and practices manually, rather than at-tempting to automate the process [10].
4 - XML External Entities (XXE)
Incorrectly configured XML processors may allow for external actors to access internal files and functionality, resulting in the potential of backdoors being placed to enable further at-tacks [5]. Additionally, sensitive data may be exposed, and there is potential for DOS atat-tacks, a risk made more prevalent due to this vulnerability remaining present in the default config-urations of many current XML parsers [18].
Countermeasures and detection
The vulnerability of XXE consists of the possibility for a malicious user to cause a variety of issues by supplying the parser with a reference to an external entity. These vulnerabilities may exist in any parser, including those sold by high-profile vendors [19]. Any customer of parsers should therefore pay attention to known exploits and weaknesses continuously, as
2.2. Open Web Application Security Project (OWASP) Top 10
the situation may well change over time. As this type of vulnerability is highly technical, automated scanners can detect these types of vulnerabilities with high accuracy [13].
Notably, removing the threat of external entities is often as simple as configuring the parser to not accept them [18], which will often have the additional benefit of preventing many other types of malicious XML injection attacks (such as the Billion Laughs8 DOS
at-tack). During a manual assessment, it is often worth questioning whether the XML parsing is actually providing any key business value, and removing the threat entirely if possible.
5 - Broken Access Control (BAC)
Broken Access Control, where non-authorized users/attackers are able to gain access to data by tampering with access rights or compromising user accounts that have access to confiden-tial files [5]. Notably, static analysis is very difficult to perform, as most attacks are highly dynamic, and the vulnerabilities may leave very limited traces in the actual source code [20]. Countermeasures and detection
If the validation of user access rights is manipulated or the access control system is miscon-figured, confidential information could end up in the wrong hands. Mitigation for Broken Access Control has considerable inherent overlap with mitigation of Broken Authentication, as well as various types of injection flaws, as such vulnerabilities may allow for access to re-stricted data, meaning that mitigation against injection flaws and broken authentication may also mitigate risks relating to access control, as shown by Zhu et al. [20].
Sandhu et al. [21] state that access control should be accompanied by log auditing, where logs track what users do in the system to confirm that the access control system works in the correct manner. They further point out that the use of log auditing is also helpful when verifying that administrators do not misuse their privileges. Such auditing can generally be handled by automatic tools, to detect situations like a non-admin user being able to access restricted pages like /admin/.
However, access control is primarily an organizational field, as poor access control is of-ten possible to detect (and exploit) even without intrusive software manipulation, making manual analysis preferable, possibly aided by the use of a proxy [10]. In addition to this, automated systems are difficult to design to recognize expected levels of access, further com-plicating the use of automated tools [13].
6 - Security Misconfiguration (SM)
Security Misconfiguration is often caused by the use of default configurations, deploying applications with debugging messages and detailed error reports still in place, or making poorly planned changes, resulting in a variety of issues [5], [10]. It can also easily be caused by connected systems, frameworks, and libraries not being maintained and updated. Default installations are often publicly known, and their attack vectors are easily made visible by using an automated scan [13], [22].
This issue is further propagated, in part, due to many web applications being dependent on the same frameworks [22]. If such frameworks allow for easy mistakes when configuring security, the same specific vulnerabilities are likely to be represented in many other-wise independent applications.
Countermeasures and detection
OWASP [5] suggests that prevention to security misconfiguration should be configured to a company standard, so that new development environments will be deployed with ease, making security configuration very efficient.
As correctly configuring security is often not as technically complex as many other aspects of security, many mitigation techniques, such as the one proposed by Eshete et al. [22], are primarily focused on simply identifying and reporting such issues.
7 - Cross-Site Scripting (XSS)
Input fields that do not have decent validation in place are vulnerable to a variety of at-tacks, which can allow the attacker to execute scripts or insert elements into the browsers of other users without their knowledge or consent. The users can then face identity theft or be redirected to malicious sites [5]. XSS attacks are commonly grouped into DOM/Type-0, stored/persistent/Type I, or reflected/non-persistent/Type II.
Stored XSS refers to malicious code being stored on a server, often hidden within user supplied material such as images or comments, which will then be executed by any browser which attempts to display the content [23].
Reflected XSS describes attacks in which the malicious code is provided by a client and then reflected from the server back to the client’s browser, which executes it. The XSS re-quires the input to be part of the response, which often means that error messages or search results are used as attack vectors. Attacks generally involve tricking the victim to do this to themselves through links or similar means [10], [23], [24].
DOM Based XSS vulnerabilities exist within the DOM (Document Object Model), rather than the actual HTML. The primary difference from the other types of XSS is that the HTML source code and responses are completely unchanged, making detection and prevention more difficult. As the malicious parts of the code is never sent to the server, server-side filtering also has no effect [25].
Many automatic tools are equipped with capabilities of detecting the possibility for per-forming XSS attacks, making such detection quite suitable for automation [13], [26]. Manual analysis, while applicable, will generally be considerably less effective.
Countermeasures and detection
Detecting XSS vulnerabilities is fairly straight-forward through penetration testing. Once po-tential insertion points are identified, vulnerabilities can often be confirmed non-intrusively by attempting to insert basic special characters such as < or > [10].
To prevent XSS there are, according to Jeremiah Grossman, several methods [24]. Of these, the most common is to filter or block user input [24]. It should be noted that error messages resulting from blocking invalid user input could itself become an attack vector for XSS, for example if the input is incorporated into the response. Filtering input is often preferred for its relative simplicity, but has its own set of issues, as it can often be circumvented. As with many other types of input handling, whitelisting is preferable to blacklisting.
On an architectural level, OWASP [5] recommends a basic separation of verified and un-verified application content, through frameworks, escaping or context-sensitive encoding.
8 - Insecure Deserialization (ID)
If applications deserialize objects supplied by users, they may have been written/modified by an attacker, potentially altering user rights or tampering with other aspects of the application [5].
For example, as shown by Charlie Lai [27], Java classes inheriting from the standard Java interface Serializable, generally allows objects to be created from serialized streams, avoiding any checks that are part of the class’s constructor, potentially allowing malicious objects to be instantiated within an application.
2.2. Open Web Application Security Project (OWASP) Top 10
Countermeasures and detection
According to OWASP themselves, the only way to be truly safe from insecure deserialization is to simply not deserialize any user-submitted objects, except from trusted sources (white-listing)[5]. Secondary solutions include requiring digital signatures, which would be applied as part of legitimate and approved serialization methods, or isolating the deserialization pro-cess from sensitive or important systems.
Automation, while somewhat applicable for this type of vulnerability, has limited uses. Tools may search for patterns implying serialized objects, or probe the application with its own serialized objects, but generally require some degree of specific source-code knowledge [13].
9 - Using Components with Known Vulnerabilities (UCwKV)
The use of components with known vulnerabilities in applications implies that the system as a whole has the same vulnerabilities as the component, as they are often running with the same levels of privilege as the application itself [5]. For the same reasons, attacks using such vulnerabilities may be vulnerable in spite of existing security measures, which might be bypassed by the privileged components [10].
A study by Inoue et al. [28], which included a total of 123 open-source projects using one or more of three chosen libraries, showed that 84 (68.3%) used outdated versions of these libraries, potentially exposing them to various vulnerabilities. This would imply that, at least for small-scale projects, the issue of vulnerable components is fairly common.
Countermeasures and detection
Preventing exposure to known vulnerabilities of used components is largely centered on choosing components, and handling them properly. Avoiding the use of third-party com-ponents to the extent possible, only using comcom-ponents that are truly needed, and making sure that they are kept up-to-date at all time are some basic mitigation methods [5]. Auto-mated scanners can be used to map specific components to certain known vulnerabilities, but access to a database with all vulnerabilities cannot be expected, limiting the usefulness of automated mapping [13]. Presuming the existence of documentation of used components and their version numbers, manually checking them against vulnerability databases or their own respective documentation regularly is necessary to ensure continued security. If possi-ble, use of third-party components should be avoided, or limited to cases where they provide considerable business value.
10 - Insufficient Logging and Monitoring (IL&M)
Lack of sufficient logging and monitoring will likely increase the time to detect when some-thing has gone wrong, thereby delaying reactive countermeasures and damage control. Events to log and monitor can be either intended or unintended behaviors that need reac-tive controls. Lack of a correctly configured Intrusion Detection System (IDS) to monitor the logs will make this problem increasingly severe [5].
Countermeasures and detection
Insufficient logging and monitoring is stated as a vulnerability that is impossible to detect automatically [13], since a detection system could monitor a web application from a remote server. Therefore, when attempting to verify the quality of logging and monitoring processes, one first needs to ask the system administrator whether a detection system exists at all.
Logs can reduce detection time of an attack or an intrusion. Such intrusions may not be the actual attack itself, but general vulnerability probing in preparation for a more substantial
Table 2.1: Log States State Description
Normal Normal user behavior.
Alert Something has diverged from the normal site, such as a series of failed log in attempts.
Incident Something is not working as intended, potentially disrupting the service. Critical Something critical has changed, like root privileges or root commands, either
with malicious intent or accidentally.
breach [5]. If the detection system is properly configured, the time to detect a breach will almost be infinitesimal.
However, logs on their own are not enough to reduce detection time, as proper monitor-ing tools are also needed. A monitormonitor-ing tool will have the logs as an input and then categorize them into different states, notifying the operator when there is actually something that needs to be taken care of. Possible types of states, as defined by the European Commission Infor-mation System Security Policy [29] are shown in table 2.1.
Preferably, logging should be performed consistently across all aspect of an application, an organization, or even an entire industry, to ease analysis between instances. However, it is important to note that the quality of this system is not guaranteed just by its presence.
NIST, the National Institute of Standards and Technology, has presented a Guide to Com-puter Security Log Management[30] where they propose a four-step process, which admin-istrators should follow for the system which they are responsible for:
• Configure the log sources, including log generation, storage and security.
Depending on organizational policies, administrators must decide what compo-nents of their host systems should be part of the logging infrastructure. Log sources must be coupled to their respective logs, and the particular data to be logged for each type of event. It is prudent to limit logging to interesting events, given that the logs may otherwise grow excessively large. In addition to this, the storage of entries must be formalized, as different events may be suitable to store for longer periods of time, or in different systems altogether.
• Perform analysis of log data.
In order to make any use of the logs, the administrators need to obtain a good grasp of the logs. For this to be as simple as possible, the logs need to be constructed in an good way, providing enough context and clarity to be easily understood. Entries should be prioritized according to time of day of occurrence, frequency, log source, and other factors, and marked as interesting for different levels of the application when applicable.
• Initiate appropriate responses to identified events.
When analyzing logs, there must be procedures in place to identify the need for response. Administrators should not only make sure that such procedures are in place, but also be ready to assist in the efforts.
• Manage the long-term storage of log data.
Some data may be necessary to store for long periods of time, perhaps even for several years. In such cases, additional factors must be taken into consideration. The data logs must have a format decided for such storage, storage devices must be chosen, and the method of storage, regardless of technical details, must be phys-ically secure.
3
Penetration Testing
Penetration testing, the process of simulating attacks on a system in order to detect any po-tential vulnerabilities, and usually confirm their existence by exploiting them (penetrating the system), is a common technique throughout the industry. The InfoSec Institute [10] defines the process as having four phases, each of which is expanded upon below:
• Information Gathering • Vulnerability Testing • Risk Assessment • Reporting
Many other methodologies exist, such as those presented by OWASP in their testing guide [31], which also goes into considerably greater technical detail, but the general outline of the penetration testing methodology does not deviate much from the InfoSec Institute’s descrip-tion.
3.1
Information Gathering
The information gathering phase of penetration testing consists primarily of a manual review of the application and its environment. To be able to understand how a specific web appli-cation operates and what kind of vulnerabilities might exist, such reviews are a crucial part of the initiation phase of the penetration testing [2], [10]. A manual reviewer, as opposed to an automated tool, has the advantage of being able to have concepts and design decisions explained to them, to better identify and assess potential security concerns [31]. Solving such problems requires a great deal of flexibility and knowledge, something which automated systems would struggle to handle [32].
The review may generally investigate matters such as the verification of the established application requirements, the information classification, the security- and logging policy and the risk analysis. These activities make it possible to detect logical vulnerabilities and to iden-tify vulnerabilities such as insecure deserialization and insufficient logging and monitoring, before any resources have been spent on intrusive testing.
While the review should include documentation, policies, requirements and other non-code circumstances, one of the core activities is the actual source non-code review [31]. For this purpose, implementing some manner of automation often saves considerable time and effort. Specifically, Biffl et al. [33] point out the worth of automation in cases of large code bases, and looking for simple and recurring problems relating to code quality itself. Still, as automatic tools are unable to independently surmise the intent of any particular piece of code, they cannot reliably identify flaws within the design, and is likely to result in large amounts of false positives. Even though it may find security issues based on coding errors, manual review will always be necessary [31], [34].
The review also allows the reviewer to get a comprehensive idea of how the web appli-cation works and understand what parts may be of particular interest for further analysis. The primary areas of interest for the reviewer should be the same as for a potential attacker. Regarding many of the more organizational vulnerabilities, a basic rhetorical survey may be enough to detect them, including questions like: Is there a correctly configured Web Appli-cation Firewall in place? Are the activities within the web appliAppli-cation being logged? Does it seem like the web application has client-server requests being sent to a database? What type of database is being used?
Such surveys would generally be a good and inexpensive first step, in preparation for the planning of more complex analysis and mitigation.
3.2
Vulnerability Testing
During the vulnerability testing phase, the penetration tester investigates the system while it is running, attempting to identify vulnerabilities, generally by applying various exploits. While the manual review will have provided guidance towards some of the existing vul-nerabilities, there is a large degree of experimentation involved in vulnerability testing [2]. Initially, focus should be on mapping all available paths and points of interaction. Perform-ing such mappPerform-ing manually may provide great insight into potential flaws and exploits, but it can become unwieldy for larger applications [10].
To mitigate this issue, tools can be used to automate the process, either partially or com-pletely. Relevant features to support the dynamic analysis include intercepting proxies, man-ual request editors, fuzzers, crawlers and active/passive scanners.
An intercepting proxy is located between a user and the internet. During malicious attacks, the user/client would not normally be aware of the proxy, but the client/server communi-cation is being intercepted. The proxy can then perform actions, including redirection, input modification, or simply logging the communication [35].
Manual request editors can make requests to specific targets at any time with desired pa-rameters. They are also capable of modifying parameters of current requests, potentially bypassing front end validation.
A fuzzer is a black-box functionality commonly included in vulnerability tools, which al-low an attacker to submit illegitimate data to a target, in desire to find an exploitable vul-nerability. Irrelevant data could include all sorts of basic SQL-injection code, XSS-scripts and common passwords [36]. This malicious input is often supplied in a high-speed brute-force fashion, though application-specific knowledge may allow for the input types to be specifi-cally chosen, depending on the suspected vulnerability.
A crawler works by crawling web pages of a website to map resources, using existing hyperlinks. Benefits include a faster and more extensive mapping process than a regular manual mapping and the possibility to find hidden resources that are not meant to be public. This type of feature is a common type of automated scan, and is often a basic feature of web scanners.
An active scanner will scan a website and perform common attacks on the website trying to find potential vulnerabilities. Active scanners suffer when there is dynamic content in live
3.3. Risk Assessment
Figure 3.1: Risk Severity
applications, such as if user supplied content is added or modified during scanning. In such cases, the scanner will likely give a report of limited accuracy.
A passive scanner, on the other hand, is used to monitor requests and responses without modifying them. This causes considerably less strain on network capabilities, while still al-lowing for a lot of information to be gathered [37]. Drawbacks to such scanners concern the lack of thoroughness and accuracy, as it is inherently limited to observing existing traffic.
Unfortunately, black-box penetration testing cannot be guaranteed or even expected to find all security flaws, even those that may be obvious upon manual source code analysis. In addition to this, automated scanners can only detect the existence of known vulnerabilities, and has no capabilities to detect operational or procedural issues [34].
Moreover, verification of any identified potential vulnerabilities is an important part of vulnerability scanning and must be performed manually. There is considerable risk for any automated system to issue alerts and warnings due to issues that do not actually present any vulnerability or issue worth considering for mitigation. While relevant in many fields of study, the issue of false positives is of particular interest to this field and must be taken into consideration [38].
3.3
Risk Assessment
Not all findings from a vulnerability test are necessarily relevant to the specific evaluated web application, or cause enough damage to business values to motivate taking action against. A risk assessment must be done in order to decide which findings are relevant to the studied web application and to decide what risk level each finding has [10]. One method for deter-mining the severity of a specific risk is to estimate the likelihood of an attack and the expected impact, rating them as either low, medium or high, and rating the result in accordance with the severity levels shown in figure 3.1 [10].
3.4
Reporting
A well written report of the findings of each studied web application is desired. The report should contain the list of found vulnerabilities, the severity and the likelihood of each vul-nerability to occur and be sorted in a prioritizing order. Moreover, it needs to clearly give a list of relevant readings as well as a description on how to mitigate each vulnerability [10].
Environments
Different types of vulnerabilities require different approaches to detect and mitigate. To aid with this, a variety of tools and methods are available, as well as specifically constructed environments against which to test them.
4.1
Methods of Analysis
Industry experts [39] point out different issues as being applicable for different types of anal-ysis, such as by differentiating between logical and technical vulnerabilities. Technical vul-nerabilities refer to issues best found by automated systems, as they are rule-intensive and time-consuming. Examples include scanning for potential injection points via HTML-forms or other user input. Logical vulnerabilities, on the other hand, generally concern design choices, such as what data is passed via URLs, or what data is stored by the application. These logical vulnerabilities generally require human analysis to identify.
Within the industry, companies are generally promoting automated systems for vulnera-bility scanning as an alternative to or even replacement of manual penetration testing [40], [41].
Nevertheless, methods of analysis depend on the specific type of task being performed. Just as other areas of analysis, vulnerability scanning have mainly two types, static and dy-namic analysis [33], [38], [42].
Static analysisof software refers to the act of checking whether some piece of software matches a set of requirements, without executing the software in question.
Dynamic analysis, in turn, is performed during run-time, observing how the system be-haves when interacted with, often in explicitly malicious or otherwise non-standard ways, as described by Arkin et al. [42]. They also point out the efficient use of automated tools doing the grunt work, but that it cannot be fully reliable without a skilled security analyst. While dynamic analysis and penetration testing is widely used by many companies as the primary, if not the only, approach to test security, this does not provide complete safety from all vul-nerabilities [31]. Unless external circumstances prevent it, security analysis should always contain a mixture of many techniques and methods.
For these different types of analysis, different tool features are required. A feature could be capable of operating in both automatic and manual environments, or in only one of the two.
4.2. Vulnerability scanning requirements
Through the types of analysis, individual tool features can further be mapped to relevant vulnerabilities, to verify the extent of the coverage.
4.2
Vulnerability scanning requirements
Relevant requirements to choose vulnerability scanners for the comprehensive methodology are presented below.
FedRAMP
FedRAMP, the Federal Risk and Authorization Management Program, is an American gov-ernmental program which provides standardization of security measures for cloud-based products and services.
Their mission is to raise the quality of governmental IT infrastructure by moving from insecure legacy system to modern cloud-based ones. By providing an authorization program to private cloud service providers, other American agencies can trust those CSPs to provide high quality cloud-based IT solutions.
In order to maintain high quality, as a part of their business practices, they require that Cloud Service Providers (CSPs) identify and apply security evaluation tools for three distinct categories: Operating System and Networks, Databases, and Web Applications [43]. They require CSPs to perform these scans monthly and submit the reports to authorization officials at FedRAMP. These reports are critical, as they allow FedRAMP to continuously monitor the state of the CSP’s systems.
For the reports to fulfill their purpose, FedRAMP has set up a list of requirements for vulnerability scanners, as shown below:
• Authenticated/credential Scans
Scans must be performed with full access credentials, as a lack thereof is likely to produce limited results.
• Enable all Non-destructive Plug-ins
Scanners must scan all findings, including add-ons and plug-ins. • Full System Boundary Scanning
Scans must include any and all components that are part of the system. • Scanner Signatures Up to Date
Scanners must be continuously maintained with current versions of vulnerability data, as well as being updated themselves.
• Provide Summary of Scanning
Each scan report must be complete with various data, including what ob-jects/components have been scanned, what specific tools were used, and scanner settings.
• POA&M All Findings
All findings must be noted in a Plan of Action and Milestones, until the vulnera-bility has been dealt with. [43]
Several points in the above list are corroborated by other similar requirement guidelines. For example, A Hong Kong Government Vulnerability Scanners Overview [34] points out the importance of vulnerability scanners being frequently updated. It also makes note of the potential for overlapping instances of the same vulnerability, and the need for consistent and well-structured scan reports. Similarly, Netsparker [44] point out that if there are parts of the
web application that require authentication of any type to access, scanners should be able to not only investigate the authenticating mechanism, but also be possible to configure to log in automatically and test the application components it is protecting.
Experienced Penetration Testers
In addition, experienced penetration testers prioritize configurability of tools as an important factor [45]. While focused tools are often optimized for their particular purposes, being able to modify and add features creates flexibility.
Another requirement of note is the cost factor [9]. Smaller businesses often do not have the resources to buy licences for state-of-the-art penetration testing tools, which can range between a few hundred dollars per year to tens of thousands, depending on the amount of licences required1,2,3,4.
4.3
Existing Tools
The market for automated vulnerability scanning software is already well established, and white papers from companies selling such services, including Beyond Security [40] and the for-profit SANS Institute [41], present many benefits of using them.
Independent research, however, points out that while automated vulnerability services are indeed useful and being continuously improved, there is still a need for a skilled operator to sort out the false positives and to know how to prioritize the identified vulnerabilities [6].
Below, such services and tools are listed and described.
Zed Attack Proxy
Zed Attack Proxy (ZAP)5 was developed based on the OWASP guidelines. It is marketed as a beginner’s product, supporting both automated services and manual security testing. It includes a variety of features, including active and passive scanner, an intercepting proxy and manual request editor, a port scanner and a fuzzer [14].
Burp Suite Community Edition
Burp Suite Community Edition6 is a free edition of the commercial Burp Suite, including
essential manual tools, though not the advanced set nor the vulnerability scanner. In an evaluation concerning The Web Application Security Scanner Evaluation Criteria (WASSEC) the full version performed top three out of 13 web application scanners [46].
Sqlmap
The open source sqlmap7performs automatic detection scans for SQL injection flaws in spe-cific URLs and provides tools for exploiting these same flaws. It includes support for a num-ber of techniques, including stacked queries, time-based blinds and error-based attacks, as well as dictionary attacks on automatically identified hashed passwords, among other fea-tures. While lacking any automated crawling capabilities, the specific features allow sqlmap to be used as an effective verification and exploitation tool [47].
1Netsparker Pro: https://www.netsparker.com/pricing/ 2Acunetix: https://www.acunetix.com/ordering/ 3Detectify: https://detectify.com/pricing 4Nstalker: http://www.nstalker.com/buy/ 5ZAP: https://github.com/zaproxy/zaproxy 6Burp Suite: https://portswigger.net/burp 7Sqlmap: http://sqlmap.org/
4.4. Related Works: Tools being tested against special environments
Table 4.1: Scanners and OWASP top 10 vulnerability mapping as claimed by each scanner’s documentation
I BA SDE XXE BAC SM XSS ID UCwKV IL&M
Vega X X X X X X ZAP X X X X X X X Arachni X X X Sqlmap X X X IronWasp X X X X X X X Wapiti X X X X BurpSuite CE X X
IronWASP
The IronWASP8(Iron Web application Advanced Security testing Platform) framework can be used for vulnerability scanning of web applications for a variety of well known vulner-abilities. It gives fairly detailed reports of the found errors and is capable of detection of false positives/negatives. In a test [48] run, against the WackoPicko9web application, it was able to detect the most vulnerabilities of the subject scanners, which included OWASP ZAP, IronWASP and Vega.
Vega
Vega10is capable of assisting with the detection and validation of SQL Injection, XSS, among other vulnerabilities and threats. It is still in development and does not support post-authentication scanning in its current version (1.0) [48].
Arachni
The Arachni11web application security scanner framework is a combined crawler and pen-etration testing tools, checking for a variety of common vulnerabilities, including Injection, XXS and XXE, among many other issues. It is also capable of detecting and adapting to dynamic changes during path traversal, which is not a common feature in scanners [49]. Arachni has two versions, one with a graphical user interface (GUI) and one with execution from command line.
Wapiti
The web-application vulnerability scanner Wapiti12is designed for purely black-box scanning through crawling, attempting to discover a variety of types of injection flaws through fuzzing [50].
4.4
Related Works: Tools being tested against special environments
In general, research within the field of automatic vulnerability scanning is comprised of com-parisons between existing tools and frameworks. They are almost exclusively academic, and rarely discuss matters of practicality, instead focusing on structured testing in specifically de-signed environments. Given the potential security risks involved, real life industry method-ology and processes are rarely made public.8IronWASP: https://ironwasp.org/index.html
9WackoPicko: https://github.com/adamdoupe/WackoPicko 10Vega: https://subgraph.com/vega/
11Arachni: http://www.arachni-scanner.com/ 12Wapiti: http://wapiti.sourceforge.net/
The paper Why Johnny Can’t Pentest: An Analysis of Black-Box Web Vulnerability Scanners[26] from 2010 describes the testing of 11 scanners, three of which were open-source, against the vulnerable application WackoPicko, which was specifically constructed for this experiment. It was found that, at the time, none of the scanners were able to find even half of the vulnerabilities, while a group of students described as being of "average" skill were able manually to find all but one. The limited performance of the scanners was concluded to be due to a small set of critical limitations. Foremost of these limitations is the scanner’s general inability to detect logical vulnerabilities and great difficulties in performing multi-step attacks. In addition, they were unable to access parts of the application that required authorization.
In the years following the above study, the field was studied further, generally coming to similar conclusions, such as Antunes and Vieira [51], who performed a study on four scan-ners, three of which were commercial. That study reports that even when used together, these four scanners were only able to collectively find less than a third of the known vulner-abilities, compared to a group of experts who were able to manually identify 98%. Notably, they also show a considerable amount of false positives, with over half of the vulnerabilities reported by two of the scanners not actually being considered valid. They use this to point out that trusting automated scanners is likely to result in large amounts of work being put into solving problems that do not actually pose any threat, or possibly do not exist at all. As such, neither the coverage nor the precision of available tools were found to be satisfactory, nor able to compare to human expertise.
Alsaleh et al. [52] point out that, based on their testing of four scanners, there was no-ticeable discrepancy between the results that were reported, even when performance levels were similar, pointing out a lack of comprehensive studies on the subject. While generally satisfied with the quality of the tools’ respective user interfaces, they make note of question-able crawler coverage, as well as the accuracy, speed, and false positive rates being little more than "acceptable".
In the master’s thesis of David A. Shelly, Using a Web Server Test Bed to Analyze the Limitations of Web Application Vulnerability Scanners[53], a method for evaluating the effectiveness of automatic vulnerability scanners is presented. It used a specifically modi-fied environment in two versions (secure and insecure) to test six scanners, primarily focus-ing on the analysis of false negatives and false positives. Interestfocus-ingly, it is found that even vulnerabilities normally considered very suitable for automation, such as basic form-based SQL-injection and reflected XSS vulnerabilities, have quite low detection rates, averaging at 59.7% and 43.3% respectively. For less simple vulnerabilities, the scanners were found to be far from sufficient. Interestingly, regarding false positives, it was found that many such cases were caused by duplicate reports, rather than truly incorrect results. While false positives were generally not deemed very common, it was speculated that this was strongly correlated with a generally limited capability of the scanners to find vulnerabilities at all, whether they were actually present or not.
4.5
Existing Methods
Some vulnerabilities are difficult to detect with automated tools, instead requiring the use of various manual techniques. Some such methods are described below.
Manual Crawling
An automated spider crawl is likely to add irrelevant URLs to its findings, especially in the case of large scale web applications. Occasionally, some URLs are also considered irrelevant due to specific scope restrictions having been set for the penetration testing. Tools like Burp-Suite have the feature to detect URLs from a manual browse, helping the operator to optimize run-time and vulnerability findings of e.g. active scanners.
4.6. Existing Environments
Logical Flaws
Logical flaws are not generally something an automated tool is able to detect [13]. For exam-ple, if a discount code is either possible to enroll multiple times or modify, it is considered to be a logical flaw which a human penetration tester easily could detect.
Many of the logical vulnerabilities require creative thinking to detect and exploit. As two penetration testers have stated in separate interviews performed as part of this thesis, penetration testing can be considered to be as much of an artform as a technical skill [2], [9].
Vulnerability Verification
Whenever a vulnerability is suspected to exist, whether because of having been reported by scanners or through intuition, they must be thoroughly verified to exist before being eligible to be reported. Oftentimes, the most practical way to do this would be to perform the appro-priate attack. The specific methods are as varied as the vulnerabilities they exploit, and may be performed with or without tools, depending on the situation.
4.6
Existing Environments
The following applications were chosen for use in the testing of various tools and methods. With the exception of Bila Nu, Presentshoppen and OrangeHRM, they were deliberately de-signed for such testing.
DVWA
The Damn Vulnerable Web Application (DWVA)13 is a deliberately vulnerable web appli-cation intended primarily for eduappli-cational penetration testing, both manual and tool-based. The application contains ten sets of vulnerabilities (each with three difficulty levels), includ-ing Broken Authentication, SQL injections and XSS attacks. DVWA has seen some use in academic research on the subject of vulnerability testing, such as by Lebau et al. [54], who attempted to automate model-based vulnerability testing of web applications.
OWASP Bricks
OWASP Bricks14, a currently inactive project, is a small set of specific vulnerability exercises. Totalling 13, the "Bricks" are divided into "Login", "File Upload" and "Content", focusing on various forms of code injection.
WACKOPICKO
As part of a paper on automated penetration testing [26], the WACKOPICKO application was constructed for the purpose of creating a completely known list of vulnerabilities against which to measure scanner results. It attempts to mimic a real web application, with user cre-ation, user submitted content (including image files) and other similar features. As a result, it has a range of different vulnerabilities in naturally occurring environments.
Peruggia
Peruggia15is described as a web application environment intended for basic educational
pur-poses, allowing its users to apply basic attacks on a realistic application. The web application is small and one-man developed but has not been updated since 2015 at the time of writing.
13DVWA: http://www.dvwa.co.uk/
14Bricks: https://www.owasp.org/index.php/OWASP Bricks 15Peruggia: https://sourceforge.net/projects/peruggia/
Table 4.2: Environment-Vulnerability mapping
I BA SDE XXE BAC SM XSS ID UCwKV IL&M
DVWA X X X X X X OWASP Bricks X X X WACKOPICKO X X X X X X Peruggia X X X X X X OrangeHRM X X X X X X
OrangeHRM
OrangeHRM16 is a continuously updated open-source Human Resources management ap-plication, which is not in any way intended to be vulnerable. Old versions of the software is, however, still available, complete with vulnerabilities that have been patched out in the current version.
Bila Nu & Presentshoppen
For their respective Bachelor’s theses, the authors of this master’s thesis took part in the development of two separate single page web applications, Bila Nu and Presentshoppen. They were constructed with minimal focus on security, but using well established python and JavaScript libraries and frameworks such as Flask and SQLAlchemy, making them very interesting to test against.
5
Method
In this chapter, the methods used for creating the complete vulnerability scanning method-ology is described. This includes the process of the literature study, the creation of the re-quirements, and the testing and evaluation of automated and manual vulnerability scanning methods. Finally, the design, construction and verification of the comprehensive methodol-ogy, including the underlying logic, is presented.
5.1
Literature study
In order to form an understanding of the subject at large, find relevant tools and methods to use, as well as to be able to define suitable requirements, a literature study was conducted.
Primary subjects of interest included existing vulnerability scanning tools and methods, existing environments for evaluating these same tools and methods, a basis for effectiveness requirements, notable existing vulnerabilities (including their respective mitigation and de-tection techniques), as well as the differences between the respective applicability of manual and automated vulnerability analysis.
These subjects were split into two categories: Practical industry phenomena, and primar-ily academic subjects.
For the more academic subject, such as the general applicability of automation, peer-reviewed scientific sources were found by searching journal and conference repositories such as the IEEE Xplore Digital Library and publishing houses like Springer Link, mainly through aggregate search engines like Google Scholar and the LiU Digital Library.
To find examples of existing software and methodologies, as well as environments against which to test them, similar methods were used, though mainly to find white-papers, specific software documentation and industry blogs related to current industry phenomena. The ac-tual finding of the software examples was not strictly part of the literature study, but the literature study was used to provide a better understanding of how well any specific piece of software can be expected to perform when used to scan different web applications, as an initial reference point to decide whether they were worth considering for further experimen-tation.
Regarding the categorization and internal ranking of known vulnerabilities, there were already established industry standards in place. The OWASP top 10 list, which since its first version in 2007 has been updated approximately every third year, is often used for similar
