Control Gates for Assuring Information Security during System Development: A Case Study

(1)

MASTER'S THESIS

Control Gates for Assuring Information Security during System Development

A Case Study

Fredrik Emanuelsson

Master of Science (120 credits) Information Security

Luleå University of Technology

Department of Computer Science, Electrical and Space Engineering

(2)

Acknowledgement

Thanks to Professor Tero Päivärinta and Heidi Hartikainen for all help during this thesis. Thanks to all students at the program opposing and helping out with valuable comments.

Many valuable sources have been used during this work. Some of the important is Gurpret

Dhillon for his work on the informal aspects of information security and Åge Garnes for his work on project management and reducing risk in projects. Dan Harnesk classes in information

security discussing work from Siponen.

(3)

1 | P a g e

Abstract

Has information security its Achilles’ heel in the impact analysis? The hypothetical question appeared during the case study of the governmental IT department working with integration.

Complexity and lack of support from the business line was causing inadequate risk and impact assessments. This case study reveals that risk and impact assessment can fail due to complexity, lack of criteria for prioritization, unclear roles and responsibilities. Information security risk formulas requires input but in the real case it was impossible to assess the required input due to the complexity of the governmental IT projects with many vendors and complex portfolios, unclear roles and responsibilities, lack of support from the business department and complex organizational dynamics. The contradictory results indicate that calculating business impact was more difficult than assessing risk/cost or benefits. The difficulties of setting business

prioritizations became visible when the client hired consultants to conduct prioritizing

assessments. Later the consultants answered that they were unable to prioritize. This incident match what is known in information security research that even if the policies prescribes existence of information security standards that ensures quality it’s unsure whether or not they are successful in the real case. At last the IT department solved the problem by using experienced consultants who made their own priorities. Research shows that support from top-management is critical for success which is visible in the case by weak collaboration from the business

department in the cost/benefit/risk and impact assessments which affected the possibility to succeed with the important assessments in the early stage of the project. The case highlights some difficulties in getting the business department involved in the strategic IT-meetings during

development.

Keywords: control gates information security SDLC quality assurance risk management business impact cost benefit assessment

(4)

2 | P a g e

List of figures

Figure 1. Cooper (2001) Stage gate model………...10

Figure 2. Cooper (2001) Stage gate ……….11

Figure 3. Whitman, Mattord (2009) Concept of the system development life cycle……….11

Figure 4. Guttman, Roback (1995) Computer security plan………..……12

Figure 5. Company documentation screenshot intranet ………....34

Figure 6. Company documentation screenshot email conversation………...35

Figure 7. Company documentation planning board………...35

Figure 8. Company documentation screenshot from vendor……….36

Figure 9. Company documentation screenshot delivery planning……….37

List of tables

Table 1. Attributes of risk………..………..14

Table 2. Research methodology ……….………23

Table 3. Research tactics………..………24

Table 4. Empirical Data Questions about new elements………..70

Table 5. Empirical Data Questions about product queue meetings……….…70

Table 6. Empirical Data Questions about impact evaluations……….…71

Table 7. Empirical Data Questions about prioritizing……….….71

Table 8. Empirical Data Questions about Problem handling………...71

Table 9. Empirical Data Questions about Problem handling………...72

Table 10. Empirical Data Analysis Overview……….….72

(7)

5 | P a g e

Introduction

Today it’s common that organizations develop custom made software to fit their needs.

Developing the new software solution for the organization often means integration with previous already existing software including many interfaces to subsystem which can also span external systems outside the organization. This complex situation makes quality assurances of the system development process an important issue. Development of the software means adapting to existing programs and can also include external supplier’s software. The complex situation with many interfaces towards the subsystems that arises makes the quality assurance become a difficult task that must be addressed systematically and incrementally.

“In such an environment assuring software quality is a non-trivial task, and thus over the years numerous techniques have been introduced in an attempt to increase software quality”

(Ambartsoumian, Dhaliwal & Meservy 2011, p.29)

One best practice method for incremental quality assurance is called the stage gate method and was introduced by Cooper(2001) where the assurance of quality is done after every step in the process from idea to delivery. This case study describes experiences from implementing quality assurance steps for IT- projects in the governmental sector.

In the first two chapters the problem and research question is presented. Next in chapter three fundamental concepts is explained. Control gates, SDLC, Information security and

risk/cost/benefit/impact analysis are explained together with some important concepts from project management. In Chapter four the methodology of this research is described. Chapter five describes the empirical part with the results and finally findings are discussed.

(8)

6 | P a g e

1. Problem and objectives

Research findings highlights despite that we have information security standards implemented it does not mean we increase the quality or secure the core business. (Siponen 2006)

It is just so that an information security standard does not provide us with the information how we proceed with the process in real life. As Siponen(2006) argues in his article with very accurate title “Information Security Standards Focus on the Existence of Process, Not Its Content The existence of prescribed security processes in organizations does not mean the goals of the processes are achieved”. We find similar statement in NIST 800-12 standard. ”Simply issuing policy, with no follow-up to implement that policy, may not suffice” (Guttman, Roback 1995)

The writer means that even if we have security management standards in place in our working processes they do not answer how people should carry out their work, they just ensure the processes exist. “Clearly, it is not important that something is done, but what really matters here is how well it is done.” (Siponen 2006) He refers to four prestigious information security

management standards:

1. BS ISO/IEC17799: 2000 2. GAISP

3. SSE-CMM (2003)

4. The Standard of Good Practice for Information Security

So we do not know how to implement standards in a real case, this is up to organizations to find out.

Control gates is introduced as a mean to reduce risk and improve control of projects.(Cooper 2001) Control gates are elaborated in the NIST standard Security consideration 800-64 with clear milestones to be checked for every phase of the system development life cycle. In the first control gate the NIST standard includes for example financial reviews, budgetary constraints, business requirements, risk management and impact level. “Engineering security into a product’s initiation

(9)

7 | P a g e phase typically costs less than acquiring technologies later” (Richard et al. 2008) p.6 The idea of the control gate is well defined but the NIST standards does not provide with full understanding how it should be implemented in a real case. There is a need for example cases to give more understanding about control gate implementation.

(10)

8 | P a g e

2. Research question and delimitation

This study was delimited to quality assurance (from now on called QA) of cost/benefit/impact and risk assessments in the SecSDLC. This delimitation was interpreted to checkpoints (control gates) of quality assurance in other words checkpoints in the process. From the NIST800-64 standard the first and second phase control gates has been used because the study focused on the early phase in the system development life cycle. The term checkpoint is equal to QA in this study. (control gate is also used for the word checkpoint) The research question is ‘How can control gates be implemented and what are the experiences from when you try to use control gates for cost/benefit/risk and impact analysis?’ This case study aimed to give an example how the real situation looked like within an IT-department in the governmental sector. Consequences of this research question meant to understand the business perspective in the control gates to be able to fulfill the steps for the SecSDLC, because we cannot prioritize if we don’t assess the core business needs. We must investigate what is worth protecting in the organization. We must also understand how to measure and control assets like intellectual property. One part of the control gates is to understand cost/benefit of the gate to be able to make a go/no go decision. Therefore the identified risks must be are clearly understood before the system development advances to the next life cycle phase. Because if the risk isn’t clearly understood it not possible to calculate cost of countermeasures and benefit of not implementing them. This study will not describe technical measures like antivirus and firewalls but focus on the process and describe the content of the control gate assuring quality.

(11)

9 | P a g e

3. Theoretical frame

In relation to the research question the main concepts were the control gate and

risk/impact/cost/benefit assessments. The main concepts were defined and then elaborated towards NIST800-64, 800-12, 800-55, 800-30, 800-37 standard documents which were further elaborated in relation to theories based on concepts from Meier et al. (2011) , Dhillon(2006,2007) and Garnes(2004,2009).

Meier(2011) has a relation to de-facto software industry, and Garnes(2004,2009) is expert in project management with focus on risk. Dhillon(2006,2007) adds perspective to (Meier et al.

2011) by discussing the informal and social aspects of information security.

3.1 Control gate in the system development life cycle

In Coopers stage gate model the system development process is controlled at the gates.

Coopers model uses the gates to make go/no go decisions during the development process.

By using the concept it’s possible to assure quality in big projects and gives the management a tool for control of the entire process. In NIST800-64 information security standard the stage gate is named control gate. So stage gate is equal to control gate.

(12)

10 | P a g e Figure 1. Cooper (2001) Stage gate model. Quality is assured at the control gates.

Figure 2. Cooper(2001) Stage gate.

The gates reduce the risk for bad quality and increase chances for success. Problems in the

project can be corrected in an early phase of the project which saves cost. A gate can result in that a project is killed or continue by reasons presented in the control gate which is the main idea with the control gates. Not all projects should be implemented. Coopers concept is flexible and can be adapted to the real case situation. Whitman,Mattord(2009) highlights the difference between Security Systems Development Life Cycle (SecSDLC) and the Systems Development Life Cycle (SDLC). In this comparative matrix the author use six phases spanning from Investigation to Maintenance. Comparing the SDLC and the SecSDLC, (Whitman, Mattord 2009 ,p.25)

(13)

11 | P a g e Figure 3. Concept of the system development life cycle. (Whitman, Mattord 2009, p.25)

(14)

12 | P a g e Figure 4. In the picture Computer Security plan is to be integrated during the system life cycle (Guttman, Roback 1995, p.78)

To perform the practical implementation of the control gates in big organizations can fast become a difficult task due to the complexity. This means there is a need to have a simple tool to be able to produce efficient information that can be the base for control to the management. In other words the management needs information to control projects. Risk/cost/benefit and impact analysis are examples of control information which is necessary to make go/no go decisions for the project.

” Projects don’t arrive at their conclusion perfectly executed and delivering all the benefits promised in the Business Case, at the advertised cost. They must be measured along the way to ensure they are developing to plan” (Nielsen 2008) This is also a well-known fact in information security which can be related to NIST 800-55 standard “what gets measured gets done, when a Board of Directors requires the CEO to report regularly the values of specified metrics, it

communicates to the CEO what the directors consider important” (Legrant et al. 10 jan 2005, p.5) Information security NIST 800-64 standard defines a control gate. “Identifies general control gates, or established points in the life cycle, when the system will be evaluated and when

(15)

13 | P a g e management will determine whether the project should continue as is, change direction, or be discontinued. Control gates should be flexible and tailored to the specific organization. Control gates are valuable in that they provide the organization with the opportunity to verify that security considerations are being addressed, adequate security “ (Richard et al. 2008)

3.2 Risk/Impact/Cost/Benefit assessments

Risk management is according to Whitman,Mattord(2009) the process of identifying and controlling the risks facing an organization which includes assessments and calculations of risks/impacts/costs/benefits. The purpose of risk management and information security is to reduce the competitive disadvantages and optimize the competitive advantages over other

actors/companies on the market. “Risk management is the process of identifying vulnerabilities in an organization’s information systems and taking carefully reasoned steps to ensure the

confidentiality, integrity, and availability of all components in the organization’s information system.” (Whitman,Mattord 2009, p.112)

Gallagher(2011) NIST800-30 elaborate risk as a measure of what extent a resource is threatened by a potential event and the impact in case that event occurs. Risks involve loss of

confidentiality, integrity or availability of information or information systems and reflect the impact on the organization/company. “A risk assessment is the process of identifying, prioritizing, and estimating information security risks. Assessing information security risk requires the careful analysis of threat and vulnerability information to determine the extent to which circumstances or events could adversely impact an organization and the likelihood that such circumstances or events will occur….risk is the combination of the likelihood of a threat event’s occurrence and its potential adverse impact” (Gallagher 2011,NIST800-30 ,p.6,10)

Assessments methods can be qualitative or quantitative. Where quantitative refers to using actual number values on cost and losses, and qualitative refers to risk scales spanning from low to moderate to high risk values instead of numerical values. Third option is a semi-quantitative scale using 1-100 for assessing the qualitative risk. Gallagher(2011) NIST800-30 describes the process to start with Identification of threats, sources and events that can be harmful. Next step is to

(16)

14 | P a g e identify vulnerabilities and determine likelihood of occurrence. Finally the process is to

determine the impact and conclude the risk.

The risk assessment process can produce a sum in numerical value of the risk using the following formula:

Risk = (likelihood of occurrence of a vulnerability) * (value of the asset) - (current controls) + uncertainty

(Whitman,Mattord 2009, p.132)

Risk modeling using Bayesian index is discussed by Chan(2011) as a tool to deal with the complexity. Likelihood ratios go from low to high and the risk attribute is ranked in relation to other risk attributes.

Attributes of is risk Likelihood ratios Rank

Analysis ability of ‘impact risk management for continuing business’

(+) 2.419 to 1 24

Without analysis ability of

‘impact risk management for continuing business’

(−) 1 to 3.220 14

Support and coordination from top management

(+) 7.299 to 1 1

Without support and coordination from top management

(−) 1 to 11.959 1

Table 1 Attributes of risk (Chan 2011, p.633)

Similar tables are of help to categorize and determine proper ratios and rank when calculating risk. In the process of using the method the attributes can be selected and during assessments only the answer yes or no is needed to select if the attribute is presented or not. The ratios are

predefined by a board of experts.

As Dhillon(2007) discuss the following options can be considered after risk has been calculated.

 Do nothing. If the risk is considered acceptable.

 Risk avoidance. If the risk is recognized but ways to avoid it can be done.

(17)

15 | P a g e

 Risk prevention. If the risk can be prevented.

 Risk planning. If the risk can be handled by planning and prioritization.

 Risk recognition. If the risk is known and research is done to be able to handle it.

 Risk insurance. Transfer the risk to someone else by insurances.

Risk need to be addressed in an early stage of the project. “When security requirements are considered as an integral subset of other information system requirements, the resulting system has fewer weaknesses and deficiencies, and therefore, fewer vulnerabilities that can be exploited in the future.” NIST 800-37 (Gallagher2010, p.9)

3.3 NIST SecSDLC

According to Richard(2008) the System development life cycle in the standard has five phases.

1. Initiation

2. Development/Acquisition 3. Implementation/ Assessment 4. Operation/Maintenance 5. Disposal

During the first phase early integration, threats and requirements are considered. Security is considered from a business perspective. This means that the business requirements and

cost/benefit/impact analysis need to be ensured. In the second phase a detailed risk assessment is required and also functional, testing requirements. Risk, impact and cost/benefit analysis are part of the first and second phase to be able to make go/no go decisions. “Security discussions should be performed as part of (not separately from) the development project to ensure solid

understandings among project personnel of business decisions and their risk implications to the overall development project “ (Richard et al. 2008, p. 13)

The idea and purpose with the NIST 800-64 standard document discussed by Richard(2008) is to integrate security into the system development life cycle. This process starts with the business view and moves towards implementation of the application. This study will stay within the first

(18)

16 | P a g e two phases of the SDLC described in the NIST 800-64 and therefore include these concepts in the theoretical framework.

As Richard(2008) say the idea with the initial phases is to identify control gates to be able to determine if the system will change direction or be discontinued. “Control gates should be flexible and tailored to the specific organization. Control gates are valuable in that they provide the organization with the opportunity to verify that security considerations are being addressed, adequate security is being built in, and identified risks are clearly understood before the system development advances to the next life cycle phase. ” (Richard et al. 2008, p. 11)

This statement from the author is more or less identical which was displayed by Cooper(2001) which indicates that the control gates in NIST standard is adopted from Coopers stage gate concept. Richard(2008) explains that the NIST800-64 considers interdependencies between tasks so the information security activities does not impact negatively and affect the process contra productive.

NIST Interdependencies in the early phases

According to Richard(2008) the first step regarding mapping of interdependencies is to create a schedule of all coming information security activities to be able to plan and look at resources related to the activities. The next step is to approach a holistic perspective on all activities in relation to the business impact analysis. The enterprise architecture needs to be considered and evaluated together with the plans for how the system will share information with other systems.

The impact analysis and risk analysis is highlighted as key assessments in the early phases. “The BIA is a key step in the contingency planning process….The risk assessment is a primary tool to identify if the tailored security controls are effective to address an organization’s risk tolerance.

” (Richard et al. 2008, p. 17, 24)

NIST 800-64 Control gate Phase one

Richard(2008) discusses that in the first phase the business requirements needs to be assured to be able to determine the acquisition strategy. There need to be a rough concept to be able to verify that the system can be completed within budget and that the concept is in line with the organization goals. There is a need for an early Performance specification. The first phase

(19)

17 | P a g e requires an enterprise architecture which is harmonized with the organizations IT-architecture, and general business requirements. A financial review is required which includes cost, risk and impact assessments of the system.

NIST 800-64 Control gate Phase two

Richard(2008) says that in the second phase there need to be an design review that includes integration with other systems. In the second phase there is a need for a system functional review which includes test cases. Financial reviews need to be enough detailed so it’s possible to monitor cost/benefit. The second phase include follow up on the risk management. Phase two together with phase one covers the early stage of the project with cost/benefit, risk and impact analysis with focus on business requirements.

Roles and responsabilities

Information security roles and responsibilities needs to be clear in an early stage of the project. It is important that employees knows who is responsible for what. The standard suggests that key roles are identified in the first phase. ” With any development project, it is important to involve appropriate information security personnel as early as possible, preferably in the initiation phase.” (Richard et al. 2008, p.8)

In the NIST 800-55 standard we see a statement that relates information security measurement with the importance of roles and responsibilities as a critical success factors. “Clearly assign information security responsibilities, and lay the foundation needed to reliably measure progress and compliance” (Chew, Swanson & Stine 2008, p.3)

NIST800-12 definition of information security

In this study the concepts presented in the NIST 800-12 standard handbook by Guttman(1995) represents the main concepts together with NIST800-64 and defines information security with control gates. The NIST800-12 is composed in 1995 and uses the term ‘computer security’ which could be discussed as old fashion today when newer research uses information security as

concept. The motivation to use these ‘older’ concepts is that the basic concepts are well rooted in the NIST framework where the NIST800-12 present the core terms in a simple way and display a good overview of what is considered information security also today. Guttman(1995) defines

(20)

18 | P a g e computer security as the following:

NIST 800-12 discusses three perspectives involving management controls, operational and technical controls. Where the management controls address security topics which can be considered managerial as for example risk management. The operational control which are executed by people as opposed to system but can include technical and management activities.

The technical perspective has focus on the security controls in regard to the computer system.

Guttman(1995) includes software and hardware as assets but not people in the definition of computer security. ”The protection afforded to an automated information system in order to attain the applicable objectives of preserving the integrity, availability and confidentiality of information system resources (includes hardware, software, firmware, information/data, and telecommunications).” (Guttman 1995, p. 17)

In this sense information/data can be stored in many ways for example in people minds which therefore also includes people to preserve the security.

Guttman(1995) uses the concepts confidentiality, availability, integrity which means both data integrity and system integrity. Integrity required that the data is timely, accurate, complete, and consistent. Data integrity means that data is changed in a controlled manner. System integrity means that a function is processed and “free from deliberate or inadvertent unauthorized manipulation of the system”. (Guttman 1995, p.6)

Availability means that the service works fine and is available for the users. Confidentiality means that the unauthorized users cannot see the confidential data.

De facto definition of information security

In this study the concepts and theory discussed by Meier(2011) represents the de facto software industry definition of information security as the following:

 Authentication

 Authorization

(21)

19 | P a g e

 Auditing

 Confidentiality

 Integrity

 Availability

Authentication refers to the process when a user identifies himself during login. Authorization refers to the process when the user’s rights to use system resources are verified. Auditing refers to logging procedures where the users cannot deny for example an online buy. Confidentiality refers to keeping the information private so it cannot be seen by someone unauthorized. Integrity refers to keeping the information unchanged intentionally or unintentionally. Availability refers to the degree the system is available for example 99.99% of time during a year. Meier argues that information security issues can be explained by the relation and context the four concepts is related (’asset’,’threat’,’vulnerability’,’attack’.)

” To summarize, a threat is a potential event that can adversely affect an asset, whereas a Successful attack exploits vulnerabilities in your system.” (Meier et al. 2011)

A asset is something worth protecting, for example a credit card number or intellectual capital. A threat can potentially destroy, hurt or steal an asset. Vulnerability refers to a weakness that makes a threat possible, which can be a bad design or a mistake in the configuration. An attack means an action that uses the weakness. This can be sending malicious code in some field with the

intention to destroy or steal.

Dhillon (2006; 2007) uses the same definition as Meier(2011) as base but then extends the definition by introducing the concepts of responsibility, integrity, trust and ethicality. Where he discusses actors that are driven by responsibilities and with integrity to avoid unauthorized to get their hand on the company secrets. This can be done by the external actors using social

engineering where it’s about manipulating people to get the protected information. Big projects can suffer from problems with keeping the intellectual capital safe when having a high degree of external consultants moving in and out the company. It is not enough to keep control with a written contract, it requires on site active management. Without proper management and information security it’s possible to find ways to subvert controls that is only in the form of a

(22)

20 | P a g e policy. Dhillon(2006; 2007) describes trust as an important part where actors must trust on each other to make the collaboration to work smoothly. It is not possible to control everything and it would increase cost which is contra productive. The writer uses ethicality as a concept where the actor carries an inner compass which directs decisions and is a vital component in information security. These definitions are the core of the informal security which cannot be implemented by technical measures like antivirus, and firewall protection. “The major causes of loss due to an organization's own employees are: errors and omissions, fraud, and actions by disgruntled employees” (Guttman, Roback 1995)

Dhillon(2006; 2007) add the informal zone in information security where human relations and social aspects is an component. The relational component has an economic value because we choose to do business with people we know has a good history, or that we know somehow. It is for example more difficult to cut out a relative from a deal because it has consequences thereby the social relation has a value in business transactions. (Garnes 2009, Garnes 2004)

Dhillon(2006; 2007) discuss the informal aspects of information security and gives perspective on efficient strategies for management. The writer describes the security situation where the informal aspects are important because an informal aspect means company culture and collaboration among people and group of peoples.

“The major causes of loss due to an organization's own employees are: errors and omissions, fraud, and actions by disgruntled employees. One principal purpose of security awareness, training, and education is to reduce errors and omissions. However, it can also reduce fraud and unauthorized activity by disgruntled employees by increasing employees' knowledge of their accountability and the penalties associated with such actions. Management sets the example for behavior within an organization. If employees know that management does not care about security, no training class teaching the importance of security and imparting valuable skills can be truly effective. This "tone from the top" has myriad effects an organization's security

program.” (Guttman, Roback 1995)

If an employee leaves the company the intellectual property leaves with him. Surely we do not want to have disgruntled employees because it might lead to the asset in the form of intellectual capital leaves the company.

(23)

21 | P a g e

3.3 Meaning of social relations in project management

An important basis for business theory is the economic theories which Garnes(2004,2009) discuss and highlights that human relations and trust has economic value in economic theories.

Social relations are essential in economic theories. Dhillon(2006,2007) links business theories to information security and says that culture and communication is part of information security.

“Good business communication is essential to ensure integrity of operations, which in turn helps in ensuring security” Dhillon(2006,2007, p.195)

Garnes(2004,2009) discuss the meaning of human relations in projects and displays two statements”

1. Mennesker handlar med og mot sine omgivelser og samhandlar med og mot hverandre 2. Mennesker inngår i relasjoner med hverandre” Garnes(2004,2009, p.58)

These statements mean that humans interact with and against their environment and collaborate with each other. Humans make relations with each other. Human relations are an important parameter to be considered and therefore also affect aspects of information security. As Dhillon(2006,2007) argues that the social aspect might be the best way to enforce control.

Garnes(2004,2009) states that human relations is an important part of the social aspects and therefore we could learn from both writers how relations and trust can matter. This knowledge can be used as an alternative theoretical explanation to understand information security incidents.

In other words an incident can have started due to misused trust or other relational aspects which leads to security breaches.

“Familien og vennene våre inkluderas når vi vurderar nytteverdien” Garnes(2009, p.35) Which means that the family and friends is included when we value the benefits. This statement from Garnes(2009) highlights that the people value relations as one part of decisions they make.

Support from the upper management is crucial to succeed with projects

Garnes(2004,2009) discuss the importance of support from upper management in an early stage to be able to succeed with projects and reduce risk in an early stage. This statement can also be

(24)

22 | P a g e found in the NIST 800-55. “The foundation of strong upper-level management support is critical, not only for the success of the information security program, but also for the program’s

implementation.” (Chew, Swanson & Stine 2008, p.3)

Chan(2011) estimates lack of support from upper-management as a high risk factor regarding the success of the project.

(25)

23 | P a g e

4. Methodology

4.1 Research logic

During the study a deductive approach was chosen to construct and evaluate arguments.

Deductive means that the research constructed explanations and conclusions from a general principle. The principle derived from the theoretical framework and data that did not match the theoretical framework was identified.

4.1.1 Literature

Search engine tools were used for literature reviews: google scholar, scopus, proquest, web of knowledge, Luleå University Library search engines for books, magazines and libris search engine.

As one example a search with scholar.google.se on keywords: control,gates ,information,security,sdlc,case,study, governmental gave 168 results.

During investigation of the search results, it was found that control gates and

risk/impact/cost/benefit analysis were well explained in the literature but no case study discussing control gates in the governmental sector similar to my research question was found. This gap in knowledge was handled by studying control gates with context as well as possible at a real case site and using earlier known research from information security, project management and control gates ,risk/cost/benefit and impact analysis.

4.2 Research design

Yin(2009) suggest important components for covering the research design, which in this study was interpreted to being the following:

- A Selection of research methodology - B Define unit of analysis

- C Data collection

- D Interpretation of the data.

(26)

24 | P a g e - E Logical decisions models for linking data to the research question.

As a help to choose research methodology Yin (2009) enlist the following matrices.

Method Form of research

question

Requires control of behavioural events?

Focuses on

contemporary events

?

Experiment How,why ? Yes Yes

Survey Who, what, where,

how many, how much?

No Yes

Archival analysis Who what, where, how many, how much?

No Yes/no

History How,why? No No

Case study How,why? No yes

Table 2. Yin (2009) p.8 when to use each method.

According this matrix the case study fitted this study because it required not control of

behavioural events and it focused on contemporary events. “ ’how’ and ‘why’ questions are more explanatory… because such questions deal with operational links needing to be traced over time”

(Yin 2009, p.9)

Yin (2009) differs between explanatory(find explanations), descriptive(to describe), and exploratory(to explore), and also single case or multiple case.

Type of compositional structure

Explanatory Descriptive Exploratory

Linear analytic X x x

Comparative x x x

Chronological x x x

Theory-building x x

Suspense x

Unsequenced x

Table 3. Six structures and their application to different purposes of case studies. (Yin 2009) p.176

According to Yin(2009) the un-sequenced structure fits to the descriptive study, where the explanatory involves theory-building and suspense structure. “outcome is .. presented in the

(27)

25 | P a g e initial chapter or section…descriptive case study has no especially important outcome” (Yin 2009, p.178)

Päivärinta(2011) argues for the use of a systematic framework to perform organizational learning.

Ideas from this framework were used to support this study by comparing the context that existed during the development life cycle. Which methods were in use at the case study site. Were they the same during the process, or did they differ? What were the reasons for the differences in development context and what were the consequences of these differences? An example of context can be that vendors are working using an agile scrum methodology but the customer organization is using another methodology for system development. What was the consequence of the difference? Päivärinta(2011) discuss the concept of rationale which this study used by searching for an answer to what problem the organization solved with the gate meetings. What was the functional reason that the organization choose to implement the gate meetings?

4.2.1 Selection of research methodology

During this study the descriptive/explanatory single case study approach was chosen. The motivation to do so comes from that the case fitted well into the description that it was a

contemporary set of events which I have little or no control over. (Yin 2009) The motivation to choose a descriptive/explanatory single case study approach comes from an understanding of what was possible at the case site, where there was an option to have an approach to describe and explain events at the single case site, and the need for descriptions of case examples regarding control gates, with a case study that did not result in any special outcome, more than a description of the current state, but with some options to build explanations to the outcome. It would not been a good choice to choose multiple case sites because then the data collection would had suffered and the explanations been more on the surface. The context was complex within a big

organization where it was possible for me to describe what I could see through observations and read in emails and documentation and to report this in a scientific matter related to the master course. Main focus was descriptive but conclusions rely on explanation building as method for results. To have the angle with a more explanatory focus would make the study harder to perform in this complex organization so I found the best choice to be a single case descriptive study with explanation building to draw conclusions.

(28)

26 | P a g e I’m newly employed at this site so I had a good opportunity to make observations on site and collect data from company documentation. But I did not have very much control or power to influence the organization or processes which motivates the decision not to choose an action study. Still I was interested in understanding the case so surveys or an experiment wouldn’t give the answers I’m looking for because people working at the work site would not have accepted spending time on filling in long surveys or participating in experiments. Also it wouldn’t be practical possible to set up a scientific experiment in this site to get answers because it would stop production and interfere with the business.

Interviews

Interviews in this thesis has been based on six transcripts and six interviewed people which have been summarized in a scorecard that has been discussed in group form with the key informants to improve reliability. Interviewees have got the questions in good time before the meeting so they had time to understand the questions and to think them through. Scorecards are a good way to collect information like strategies and the visibility of the colour rating makes it easy to understand and communicate to the team which gives a triangulating effect. Kaplan, Norton(2002) highlights benefits with the balanced scorecard as an efficient tool to gather informal data like strategies and to communicate them within the organization. At the site was a good opportunity to gather key informants at one place at the same time and get valuable data.

“key informants are often critical to the success of a case study” (Yin 2009, p. 107) Members of the group can interact and decide on the grade for the question. All questions were directly related to the research question because they cover roles, responsibilities and risk/cost, benefit issues within the product queue meetings and therefore also well-grounded in information security theory and standards. Interviews have been done with a defined set of questions which has been constructed in dialogue with the organization to get valuable information from the product queue meetings. Execution of the interviews has been done in group form with key informants from the developing team. Discussing the questions in group form gave a triangulating effect, by giving key members opportunity to reflect and provoke discussions around the questions, which gave me valuable feedback and increased reliability by the group

(29)

27 | P a g e session. The group was able to review the information presented and reach consensus on the group answers on the questions.

4.2.2 Define unit of analysis

The unit of analyse was the organization. Empirical data has been collected at the organization integration centre, which was responsible for horizontal processes across the organization that could span IT-projects. Building focus on the control gates was motivated in earlier studies. One example was the stage gate model discussed by Cooper (2001) which defined stage gates which were related to what was referred as control gates in this study.

4.2.3 Data collection

Data has been collected from multiple sources on the research single case site. Information was collected using interviews, on site observations, and studying documentation. The collection strategy was to find and identify the SSCG’s (Information Security System development life cycle control gates) and to collect enough information from the unit of analysis to be able to study them in this case. Informants were mapped to be able sort out key informants and decide which informants were part of the SSCG’s. One way to start was to study the organizations documentation on the development process and use this information when doing interviews to verify which parts of the documented process were used and which parts were not used. And then use the documentation to drill more into detailed questions about what were used and what were unused related to documentation.

4.2.4 Collected data is interpreted using explanation building, linking data to the question

During data collection the theory was used as a template to compare the empirical result of the case study. (Yin 2009) Explanations were built as part of the analytic generalization.

(30)

28 | P a g e Conclusions from the study were based on the built explanations from theory in relation to collected data. Therefore data was linked to the research question using logical decision models to interpret ate the empirical information and build explanations that were triangulated in relation to the theory. Collected data were related to the NIST 800-64, 800-55, 800-12, 800-30 standards to find correlation between the case and the standard. Relations in the data that matched, or did not match were discussed and triangulated towards the theory and key informants at the case site.

4.2.5 Logical decision models for data processing

To be able to handle immediate decisions and interpretations during data collection the study included 3 logical decision models for processing the collected data. The model ensured that the collected data were linked to the research question at collection time, and therefore interference due to time between collection and documentation was minimized. Interpretation was done at collection time. ”You must be able to interpret ate the information as it is being collected” (Yin 2009, p.71)

Logical decision model I, Categorizing key informants

1. Is the informant part of SSCG? If not then find informant that is part of SSCG.

2. Is informant 100% clear over the process? If not document what informant is unclear over in relation to company documentation? If yes, what is the content of the steps in the process in relation to company documentation?

Logical decision model II, processing company documentation

1. Is the documentation possible to relate SSCG? If not, find documentation that is possible to relate.

2. Does the documentation answer how the SSCG is done in a real case? If not, use material to discuss with key informants.

Logical decision model III, processing observed data.

1. Is the observation possible to relate to a SSCG? If not, discard observation.

(31)

29 | P a g e 2. Is the actors in the observation part of a SSCG? If yes, prioritize the most important and use model I.

3. Is there an action within the observation that can describe some content of the SSCG, if yes, and then document the action.

4.3 Reliability and validity

Yin(2009) display the following matrix to test the quality of research design.

Tests Case study tactic Phase of research in which

tactic occurs Construct validity Use multiple sources of

evidence.

Establish chain of evidence Have key informants review draft case study report

Data collection Data collection Composition

Internal validity Do pattern matching Do explanation building Address rival explanations Use logic models

Data analysis Data analysis Data analysis Data analysis External validity Use theory in single case

studies

Use replication logic in multiple-case studies

Research design

Reliability Use case study protocol Develop case study database

Data collection Data collection Table 3. Case study tactics for four design tests. (Yin 2009, p.41)

The same matrix was used to verify how well the task was accomplished in this study.

Tests Case study tactic task accomplishment

Construct validity Use multiple sources of evidence.

Establish chain of evidence Have key informants review draft case study report

YES YES

YES, verified by CISO Internal validity Do pattern matching

Do explanation building Address rival explanations Use logic models

YES YES YES YES

(32)

30 | P a g e External validity Use theory in single case

studies

Use replication logic in multiple-case studies

YES - Reliability Use case study protocol

Develop case study database

YES YES Table 4. Task accomplishment of validity and reliability

4.3.1 Construct validity

Yin(2009) uses the following tests to verify construct validity:

 Multiple sources of evidence.

 Establish chain of evidence

 Have key informants review draft case study report

In this study interviews, company documentation and direct observations were used as data sources. A chain of evidence was maintained during the study using the research question and linking the collected data both to the question and to this report. During the study key informants have reviewed the interview material in the scorecard and discussed the questions and answers.

The Chief information officer has reviewed the draft of the empirical section. Yin(2009) discuss the weakness of interviews. “reflexity-interviewee gives what interviewer wants to hear”

Yin(2009, p.102)

To ensure construct validity (Yin 2009) highlights the importance of having multiple sources of evidence. In this study the data collection has covered many different sources of evidence using interview of actors in the meetings, company documentation, on site observations. The study has built chains of evidence and let key informants review the case study reports.

4.3.2 Internal validity

Yin(2009) uses the following tests to verify internal validity:

(33)

31 | P a g e

 Do pattern matching

 Do explanation building

 Address rival explanations

 Use logic models

During the study analysis data was matched according to pattern of answer and an explanation was built on the pattern. Also a rival explanation was considered when the explanation was built.

During data collection a logical model was used to ensure collected data quality.

During data collection the study has elaborated explanations and compared with the template theory in parallel with elaborating the rival explanations. Logical models have supported the thoughts to ensure the chain of evidence to minimize the effect of interference. Yin(2009) suggests an iterative manner to ensure the evidence which has been done during this study with frequent revisions during constructions of explanations and rebuilding the chains of logic.

4.3.3 External validity

Yin(2009) uses the following tests to verify external validity:

 Use theory in single case studies

Theory was used in this study.

Yin(2009) uses analytic generalization in this context and explains how a case study can strengthen the external validity for a theory. This case study has contributed to strengthen used theories. This case study is yet another example of the real world case of the theory.

4.3.4 Reliability

Yin(2009) uses the following tests to verify construct validity:

 Use case study protocol

 Develop case study database

(34)

32 | P a g e In this study a case study protocol has been used and there has been a case study database

developed. This case study has kept records of collected data and the steps that lead to these conclusions so it’s possible to trace and redo the conclusions at any time. The logical model has helped understand how the decisions were made and using which data. The results has been developed in an iterative/ triangulating manner so it was possible to trace the steps taken to the final conclusion and to see that there is multiple sources to the result.

Weakness in this study

During interviews many factors can affect the result. Yin(2009) mention situations where the interviewee gives what interviewer want to hear. This is a probable cause to subjective answers where the interviewee answers according to what fits the situation best in regard to the employer, or social factors. The final part of the interview was done in group form to conclude the scorecard which was a situation where one must also consider the peer pressure.

As Yin(2009) states there can also occur weaknesses due to selectivity. In this study the data is selected and reported which introduce many possible subjective selections.

Yin(2009) discusses weaknesses in relation to source of evidence where the author states that reporting biases can occur. In the learning context of this master thesis there are clear reporting requirements which help the students succeed within time. All information is filtered during the master thesis seminars to get the preferred result when the course ends. The filters are the researcher himself and the opponents, supervisors with both objective and subjective thoughts about how research reports should look like and be done which influence all research in the context therefore the results will be a result of the learning process participants including bias.

4.3.5 Review and approval

Information summarized in the empirical study (section 5) has been reviewed and approved by the chief information security officer at the organization. Reviews and approvals of data with key informants can increase reliability because the conclusions have been verified and can be redone to the same result. This is also discussed by Yin(2009).

(35)

33 | P a g e

5 Empirical study

5.1 Case site description

The case study was performed within an IT-team in the governmental sector. The team consisted of 7 people working internally and 15 people at the vendor. The department was new and was being established and there were still many roles to be filled during the time of this study.

Responsibilities covered among other tasks systems integration and Service Oriented

Architecture and I’m part of this team as an employee. In this study the data was collected in many forms like for example information from the company intranet and email dialogues.

Observations have been done on site and during internal courses, information sharing activities.

The team at integration centre has been interviewed by doing a scorecard rating of the product queue meetings. According to the methodology (4.25) interpretation was done at data collection time to reduce interference over time. Yin(2009) discusses the importance of collecting data and linking interpretations at collection time and compares it to a scene where the evidence is ‘fresh’

and needs to be handled fast to reduce interference over time. Collection time interpretations were marked with cursive text continuously in relation to the collected data. ”You must be able to interpret the information as it is being collected” (Yin 2009, p.71)

5.2 Results

Results have been summarized and the collected data has been interpreted at collection time.

Result summary

1. Risk is not understood.

2. Business line is not part of the prioritization at the product queue meetings.

3. Vendor and customer have difficulties to perform prioritizations.

4. A cost/benefit analysis is not done as basis for the prioritizing.

5. Lack of criteria to make prioritizations.

6. Lack of roles and routines for the product queue meeting.

7. Detailed documentation of roles and responsibilities exists in the SecSDLC.

(36)

34 | P a g e 8. Mix methods are used. Agile, ITIL, and local adaptions.

9. Prioritizations of elements are solved with help from experienced individuals.

10. Vendor and customer are sharing virtual and physical working space, but the internal business department is not actively sharing that information space.

11. Impact analysis is not done at all.

12. Incident and problem handling is working well and is prioritized by the business line.

Result one ‘risk is not understood’ came visible during the interviews of the team at the

integration centre where the team rated risk as not fully understood. Result two ‘Business line is not part of the prioritization at the product queue meetings.’ came visible during the interviews of the team at the integration centre where the team commented that the business department was not part of the prioritizations at the product queue meetings. Result three ‘Vendor and customer have difficulties to perform prioritizations’ is a result from email conversations between the vendor and the customer where the vendor answers the customer that there is difficulties in this direction. Result four ‘A cost/benefit analysis is not done as basis for the prioritizing.’ is a result from the interview section where the team rated this as not done. Result five ‘Lack of criteria to make prioritizations’ is taken from the company documentation which shows the criteria as non- existent. Result six ‘Lack of roles and routines for the product queue meeting’ is taken from the interviews where the team comments that there is a lack of roles and routines in this area. Result seven ‘Detailed documentation of roles and responsibilities exists in the SecSDLC’ is a result observed when comparing the NIST standard content with the content in the company

documentation. Result eight ‘Mix methods are used. Agile, ITIL, and local adaptions’ is

observed in the company documentation displaying mixed methods. Result nine ‘Prioritizations of elements are solved with help from experienced individuals.’ Was a direct observation at product queue meetings. Result ten ‘Vendor and customer are sharing virtual and physical working space, but the internal business department is not actively sharing that information space’ is a result from direct observation. Result eleven ‘Impact analysis is not done at all.’ Is a result taken from the comments made during interviews by the team at integration centre. Result twelve ‘Incident and problem handling is working well and is prioritized by the business line.’ is a result from the interviews where the team answered that its working well on all perspectives.

(37)

35 | P a g e

5.3 Interpretations of data.

As Yin(2009) suggest interpretations of data at collection time. First the company documentation is displayed and an interpretation was made during collection time. Next an email conversation is highlighted to be followed by important concepts in the development process like delivery

planning and SDLC documentation from the department to be complemented with observations on site.

5.3.1 Company documentation

When the administration discovered or got new requirements from the business department it was handled as a change request which was processed in the weekly change council. Interpretation at collection time: The weekly change council board was equal to the gate meetings described by cooper(2001) because they had the option to make go/no go decisions for change requests. This control gate meeting had both permanent participants and participants that were not permanent.

Among the roles participating were process leader change steering, process leader incident steering, process leader problem steering, process coordinator, section manager, people from the operation centre, service manager, system responsible, security manager, operation manager, area managers. The policy for the gate meeting stated that the change coordinator or the problem coordinator presents and explains the changes to be done for the council. These roles were filled with people from the development teams, for example if the integration centre had suggested a change/problem it could be the project manager and/or some individual from the integration centre that presented these problems or changes which needs implementation in the coming releases. Interpretation at collection time: If we assume it was the project manager from the integration centre that was presenting the information for the board he could only have collected the information about these issues from the product queue meetings at the integration centre, because it is the product queue meetings which shares the core information about the

prioritization of the elements to be implemented. One part of the policy says that at the gate meeting it’s required to present the cost/benefit, and risk/impact analysis. Interpretation at collection time: Cost/benefit and risk/impact analysis is clearly defined in NIST800-64 standard discussed by (Richard et al. 2008) as part of control gates in phase one , phase 2 which

strengthen the argument that this can be considered a gate.

5.3.2 Email conversation Company model for systems deliveries

(38)

36 | P a g e The internal system delivery methodology has defined a pre meeting before gate meetings in the process. The lowest level of gate meetings in the internal process was what’s internally called

‘product queue’ meetings.

Figure 5. Screenshot from intranet describing the existence of weekly gate meetings

Interpretation at collection time: In the screen shot is evidence of the weekly product queue meetings and it’s clearly stated who was responsible for the meetings and what the agenda for the meeting was. Still the team answers there is not anyone appointed responsible for the meetings during the scorecard rating. (appendix A)

Clear roles and responsibilities are discussed in NIST800-64 control gates. (Richard et al. 2008)

The internal company documentation described a pre- gate meeting form that was done once a week by key people in the team. Email conversation support that the customer used the vendor to be the driver for the meetings and perform the cost/benefit analysis at the meetings together with the customer, but the vendor experiences that they were not in the position to do so.

Figure 6. Screenshot from email conversation with the vendor. Vendors answers.

(39)

37 | P a g e Interpretation at collection time: The main idea with this lowest level of gate meeting was to have an updated queue of possible issues that can be included in a delivery. In other words a preparation for a project with the purpose to have a list of issues that can easily be estimated in time in order to sign agreements with vendors to implement the list of issues. This meeting form cannot be directly mapped to the gate meeting as cooper defines it with go/no go decision for the project; the product queue meeting is more like a pre meeting that serves for information sharing and decision about the content and cost/benefit prioritizing for the later gate meetings. The product queue meetings were the weekly meetings to collect the pieces that later can be puzzled into a delivery project. It was here the team learned about the core of the project, and which was later presented at the actual gate meetings for go/kill decisions.

As can be seen in the screenshot the criteria for prioritizing were not documented. Email conversation supported, that there were, difficulties in defining these in the IT-department because they needed to rely on business goals which were defined by another department in the organization. Observations showed that people working with business goals were not part of the product queue meetings. Email exchange supported that decision makers at the gate meetings based their decision on the information presented by key people from the teams, that had an understanding of the content of the product queue, and issues that were included. Observations display that during the product queue meeting a list of ‘things’, in other word reported issues were presented, and the idea was to arrange these in a prioritized order and include these in the next delivery. These issues could be of any kind from identified bugs or questions for a new architectural framework. Because the issues were yet not categorized in size of workhours they could span small and big work tasks.

5.3.3 Concepts in the development process

Evidence shows that languages were mixed in the system development tools and then also in the general method. Much of the English was kept with scrum terms like ‘sprints’ , ‘subtasks’ ,

‘stories’. Interpretation at collection time: It was obvious that vendors try to think agile methodology during development, but also tried adapting to the customers concept.

(40)

38 | P a g e Figure 7. Screenshoot of planning board displaying project sprints within SOA integrationcenter.

The term ‘virksomhetsko’ can best be translated to a business queue which has the purpose of catching more general business requirements and not yet detailed functional descriptions to be more processed.

Figure 8. Screenshot from vendors + customers mutual space for development process.

(41)

39 | P a g e

5.3.4 Delivery planning.

The IT-department followed predetermined dates for deliveries and the policy stated that the reason for this, among other things, were to be able to evaluate the risk and coupling to other systems.

Figure 9. Screenshot displays delivery planning for the IT-department.

5.3.5 Control gates in the SecSDLC

The organization had a SecSDLC document which described what and when security needed to be assured in the SDLC process. The document enlisted that system development documentation needed to be done and among other details it said “beskrivelse av gjennomført risikoanalyse”

which means that a description of the risk analysis is done.

Interpretation at collection time: This statement could be directly connected to the gate meetings by the statement “Etatens metode for håndtering av endringer skal benyttes. Ref.

melding i oppdragsdatabasen ODA samt RFC-skjemaet som brukes av IKT-drift for å melde

(42)

40 | P a g e endringsbehov (RFC: ”Request For Change”).” It means that to ensure security in the

administration the routines for the gate meetings earlier described at 5.1 needs to be followed.

The SecSDLC document said that the development process shall be followed and documented and highlighted that information on the product queue elements internal priority needed to be in place. Also there were roles with appointed responsibilities to ensure these processes and documentation. The document states that these role needed to collaborate with roles within the IT-department to ensure security in the process. During observation was noted that the role which had responsibility to ensure the SecSDLC processes were part of the business department which is located in another building. The SecSDLC document stated “Prosjektleder har et selvstendig ansvar for at relevante krav blir identifisert, samt å legge fram for prosjekteier evt. mangler man er kjent med.”

Interpretation at collection time: This statement proves the role of the ‘project owner’ which was located at the business department and putted the project manager as responsible to give roles higher up in the line correct information to make decisions. It meant the project manager needs to retrieve this information from the developing team, and/or by attending at product queue meetings. Roles and responsibilities is discussed in NIST800-64 Control gates by Richard(2008).

The IT- department had a special unit for ensuring security which according to the SecSDLC had responsibility to performs audit of core documents like requirement specifications for the

security, and these had to be delivered and approved by the unit for information security. It was a role within the business department who was responsible to create the information security requirements.

Interpretation at collection time: This statement in the SecSDLC results in a big challenge for this role located in the business department to ensure security due to limited option to

participation in the development process.

The SecSDLC stated “Det skal som del av arbeidet med utvikling av kravspesifikasjon for

sikkerhet gjennomføres minst en risikovurdering av den løsning som skal utvikles eller anskaffes.

Control Gates for Assuring Information Security during System Development: A Case Study

MASTER'S THESIS