An important step in the development of methods and knowledge meant for practical application, as was described in chapter 1, is the evaluation phase. In an evaluation, the advantages and limitations for example associated with suggested methods are highlighted. As such, an evaluation can be the first step towards modifying and improving a method, design or such. Below the three research themes will be addressed in turn and some general reflections will also be given.
6.2.1 Methods for vulnerability analysis of critical infrastructure networks
Evaluation of methods and models, as is the case in all evaluations of designed systems and artefacts, always has to be done in relation to the purpose and intended use of the method. Petersen (1994) argues that such an evaluation can be either user-oriented or scientific. Thus, there may be two types of reasons why an analysis based on a certain method or model does not lead to a successful outcome. First, the analysis may have been carried out in another way than what was intended or the purpose of the analysis may not be in accordance with the purpose of the method. These reasons are evaluated in what Petersen calls a user-oriented review.
The model and method may be formally and scientifically correct but lead to a bad result due to wrong application of the method. This, in turn, can depend on the user, such as the user having a lack of understanding of the methods; however, it can also depend on the characteristics of the method, such as it being “user-unfriendly”. Secondly, an analysis based on a method or model, although they are being used appropriately, may lead to an unsuccessful result due to fact that there are deficiencies and flaws in the method/model. These reasons can be evaluated in what Peterson calls a scientific review. Although both these perspectives definitely are important for method evaluation, the interest here will mainly be on the latter, since such a review is better suited to evaluate methods that still are in a developing stage. The methods presented in the present thesis are simply not yet adapted to suit the needs of potential users.
Conducting a thorough scientific review of a method can be a comprehensive task.
Here, only a rather overall evaluation will be made and reflections will be given concerning what features need to be incorporated in the methods and which tasks need to be performed in order to guarantee that the analysis result is of high quality, given an appropriate use of the method.
As mentioned previously, the evaluation of methods must be related to the purpose of the method. Both methods use the operational definition of vulnerability as a point of departure and the purpose of the methods is to suggest a practical approach for meeting the requirements of the definition, given in chapter 5.2.1. An effect of this is that the methods should provide a quantitative approach to vulnerability analysis. In addition, the methods aim to be applicable to large-scale technical infrastructures that are possible to model as networks, i.e. the methods aim to be quite broad approaches to vulnerability analysis. Another purpose is that it should be possible to consider large-scale perturbations by use of the methods.
Several of these purposes are clearly fulfilled; however, a number of areas need to be further addressed and reflected upon.
More concrete attack strategies (global vulnerability only)
Often very broad and generic attack strategies are used when analysing the global vulnerability of a system, such as random and directed removal of components.
The intention is that these should represent real perturbations to the systems.
Random removal is often said to represent natural phenomena, whereas removal directed at some specific components is said to represent a deliberate attack. These generic attack strategies can be very useful to get an overall picture of the vulnerability of a system, for example by comparing the vulnerability of the system to other systems or to some reference system; however, it may be difficult to draw concrete conclusions regarding the system’s vulnerability to real perturbations. A possible remediation is to develop attack strategies that represent real perturbations more appropriately. For example, if the interest is to analyse the vulnerability of an electric distribution system to hurricanes, a purely random removal may not be adequate. In that case edges representing long overhead lines would probably be more likely to fail than shorter lines or underground cables. There is a clear need for developing more concrete attack strategies that can be more practically attractive. By using existing knowledge (statistics, expert judgments etc.) regarding which types of components are more likely to fail in the face of a specific perturbation, it should be possible to develop more representative attack strategies.
Validation of the functional models
It has been argued that the methods proposed above are model agnostic, i.e.
basically any functional model can be used to model the consequences of perturbations. However, much is gained if the computation time can be kept low since the methods generally require a large number of calculations to be carried out. Therefore, it would be highly interesting to make detailed studies regarding the accuracy of different functional models. Of course, the required accuracy will depend on the purpose of the analysis, but if a coarser functional model correlates highly with more detailed models, in regards to the consequences that arise from perturbations, there are few reasons for using the more detailed models. What is very clear, however, is that the extremely rudimentary functional models, which is sometimes used in network analysis (for example when not distinguishing between different types of components), is not sufficient to capture the relevant features of most infrastructure systems.
Complementing the methods for analysis with methods for evaluation The methods described above have so far been concerned with analysis, i.e. to find an approach that is able to characterise/estimate the vulnerability of a system that is consistent with the suggested operational definition. An essential step, subsequent to the analysis, is evaluation of the analysis result. However, no clear guidelines or
methods exist to assist this process; therefore, there is a need for developing guidelines in order to take the analysis to the next step. Of course any evaluation carried out in practice must be related to the underlying values – what is seen as an
“acceptable vulnerability” by a specific person does not necessarily constitute an acceptable vulnerability from the view of another person.
Accounting for time-dependence
The methods described above are static since time is not explicitly modelled. The consequence of a perturbation is evaluated in terms of for example number of customers without power supply, or power loss. The duration of disruption and how fast the electric grid could be brought back to a normal state is however not considered, but certainly very important. This fact diminishes the practical value of the approach. By incorporating time-dependence in the analyses, the possibilities of incorporating human and organisational factors are also increased.
Applying the methods to other types of infrastructures
The methods have so far been applied to the electric power distribution system, although it is believed that the methods are possible to generalise to other levels of the electric power system and to other technical infrastructures as well. This must however be investigated in further detail. What primarily is needed in order to make generalisations possible is knowledge about the functioning of the particular system of interest in order to be able to suggest a functional model that captures the main aspects of the system.
Accounting for interdependencies between infrastructures
Previously, it was argued that the dependencies and interdependencies between infrastructure systems are increasing. There is a demand for methods that are able to take interdependencies into account. The network approach described above is believed to provide a platform for analysing interdependencies as well, since it can provide a common modelling basis for different types of infrastructure systems.
Taking interdependencies into account is believed to be a great challenge for future research.
6.2.2 Emergency response capabilities
The overall purpose of the operational definition of emergency response capabilities was to make the concept more concrete and also to suggest a structure for how the concept can be analysed. The definition provides an ideal way for determining/characterising the capability of an actor; however, before the operational definition of emergency response capabilities can be applied in practice, i.e. before the definition is extended into a method, a couple of issues need to be
addressed. One such issue is how a task should be defined, e.g. which level of detail is appropriate in a specific situation. The same holds for how to describe the context, such as how detailed the descriptions can be made. In principle, these concerns are analogue to how detailed a risk scenario should be described in a risk analysis. Another issue concerns how it is possible to gain knowledge about how well specific tasks can be performed in a future potential scenario, i.e. what sources of evidence can be used. Here, of course, persons in the system are important sources of knowledge; however, other sources of knowledge may complement this source, such as various modelling techniques, computer simulation, and statistics and so on.
It is important to distinguish between the two purposes an analysis can have (described in chapter 3.3), i.e. the process-oriented and the decision-oriented approach, when evaluating the approach. It is believed that analysing capabilities based on the proposed operational definition can help people learn about how a future emergency may evolve and which factors that influence the final outcome.
This can be accomplished by bringing people with different knowledge together and make them exchange ideas, create trust relations and hopefully also create synergies. Such analyses can also facilitate the creation of a common mental picture and mutual awareness of future possible emergencies. As such, it can be very useful for process-oriented purposes. However, an ambition here is also to be able to analyse capabilities from a decision-oriented perspective, i.e. to improve decisions regarding for example how capabilities can be enhanced. This ambition leads to higher requirements on the method for analysing capabilities, which must be addressed when such a method is suggested.
6.2.3 Value input to risk analysis and decision-making
The biggest limitation of the empirical study is that the sample used in the study consists of a quite homogenous group of students. It is important that the users of the results are aware of the composition of the sample so that the results are used with care when applied in other contexts. Of course, it is always preferable to elicit the values and preferences of the particular stakeholders that are relevant in the specific situation of interest; however, if this by any reason is not possible, one has to either somehow assume values or use values and preferences elicited in other contexts. The empirical study presented here can provide one input regarding which values that can be used as a basis for risk analyses and decision-making, but the study should be complemented with inputs from other studies.
Another reason for using the result from several studies as input to risk analyses and decision-making is that no methods are free of bias. Results from several methods can thus give insight into uncertainties regarding the values, which can be
propagated to the analysis. The presented empirical study made use of two fundamentally different methods, which increases the value of the study. However, by investigating the convergent validity further, the value of the research can increase even more, although the studies become more time-consuming and demanding for the participants.
The study showed that the cause of a disaster may be relevant for determining its seriousness. However, it is unclear whether this is an effect of people making inferences about indirect consequences. For example, in case of a disaster scenario caused by a terrorist act, people may have made inferences about increased airport security in the future, that other people are encouraged to perform acts of terrorism, consequences related to psychological distress etc. Thus, it may not be the cause per se that affects values, but the beliefs about the “consequences of the consequences” associated with terrorist acts. It is important that the users of these results are aware of the problems associated with the interpretation of the result.