REPORT D2.3 SCADA system architectures
PROJECT TITLE: Vital Infrastructure, Networks, Information and Control Systems Management
PROJECT ACRONYM: VIKING GRANT AGREEMENT
NUMBER:
225643
PROJECT START DATE: 01.11.2008 DURATION: 36 MONTHS
PROJECT CO-ORDINATOR: GUNNAR BJÖRKMAN,ABB (ABB) DE
PROJECT MEMBERS: ABB (1) (ABB) DE
EIDGENOESSISCHE TECHNISCHE HOCHSCHULE ZÜRICH (2) (ETHZ) CH
E.ON (4) (E.ON) DE
ASTRON INFORMATICS LTD. (5) (Astron) HU
KUNGLIGA TEKNISKA HÖGSKOLAN (6) (KTH) SE
UNIVERSITY OF MARYLAND FOUNDATION (8) (USMF) US
MMLANALYS &STRATEGI (9) (MML) SE
DOCUMENT IDENTIFIER: D2.3 ISSUE: 1.0
ISSUE DATE: 2010-05-19 PREPARED: 2010-05-19
APPROVED TECHNICAL LEAD/KTH
DISSEMINATION STATUS: PUBLIC
History Chart
Issue Date Changed Page (s) Cause of Change Implemented by 1.0 2010-05-19 All First version Teodor Sommestad
Gunnar Björkman
Authorization
No. Action Name Signature Date
1 Prepared Teodor Sommestad Gunnar Björkman
2010-05-19
3 Approved Technical Lead 2010-05-19
The information in this document is subject to change without notice.
All rights reserved.
The document is proprietary of the VIKING consortium members listed on the front page of this document. The document is supplied on the express understanding that it is to be treated as confidential and may not be used or disclosed to others in whole or in part for any purpose except as expressly authorized in terms of Grant Agreement number 225643.
The VIKING consortium makes no warranty for the information contained in this document; neither does it assume any legal liability or responsibility for the accuracy completeness or usefulness of this information.
Company or product names mentioned in this document may be trademarks or registered trademarks of their respective companies.
Distribution List
This document is distributed as below.
Additional copies held by unnamed recipients will not be updated.
Electronic Copy Number Name Address
1 European Commission EU, Brussels
2 Gunnar Björkman ABB AG, Mannheim
3 Pontus Johnson KTH, Stockholm
4 - 9 VIKING consortium members
Table of Contents
1 Preface ...9
1.1 Overview of the VIKING project ... 9
1.2 Purpose of deliverable 2.3 ... 12
1.3 Scope ... 13
1.4 Document overview... 13
2 Method and theoretical framework...14
2.1 Literature review ... 14
2.1.1 Security literature focusing on SCADA systems ... 14
2.1.2 SCADA and control system literature ... 15
2.1.3 Market surveys... 15
2.2 Metamodel... 16
2.2.1 Scope and definition... 16
2.2.2 Metamodel rationale... 17
2.3 Interviews with vendors... 17
2.3.1 Scope and limitations ... 18
2.3.2 Baseline models... 19
2.3.3 Respondent description... 19
2.3.4 Qualitative validation... 21
2.3.5 Quantitative estimates of commonality ... 21
3 Services and data flows...23
3.1 Services ... 24
3.1.1 Front-end ... 24
3.1.2 Phasor measurement unit ... 25
3.1.3 Phasor data concentrator ... 25
3.1.4 RTU/Gateway ... 26
3.1.5 SCADA server... 26
3.1.6 Inter Control centre Communication server... 27
3.1.7 Application server ... 27
3.1.8 Historian... 29
3.1.9 Operator HMI client... 30
3.1.10 HMI Server ... 31
3.1.11 Remote client (e.g. Critix) ... 31
3.1.12 Application software for the office ... 31
3.1.13 Office applications software ... 31
3.1.14 Web browser... 32
3.1.15 Office servers and power applications outside of the control room ... 32
3.1.16 Data engineering server... 32
3.1.17 Data engineering client ... 33
3.1.18 Remote engineers’ workstation for substations... 34
3.2 Data flows ... 34
3.2.1 PMU Measurements... 34
3.2.2 Process measurements to control center ... 35
3.2.3 SCADA sends commands to process ... 36
3.2.4 Front-end data to real-time database... 37
3.2.5 Phasor data sent to real-time database ... 37
3.2.6 SCADA server sends command to process ... 38
3.2.7 Data sent to emergency centre(s) ... 38
3.2.8 Commands and status ... 40
3.2.9 Process data to/from other control centres ... 40
3.2.10 Operator commands to process and manual changes and mark-ups of data42 3.2.11 Operator views and alarms sent to HMI... 43
3.2.12 A server hosts several HMIs ... 44
3.2.13 Terminal services... 44
3.2.14 Mail websites etc ... 45
3.2.15 Read operator views and change parameters of power applications ... 46
3.2.16 Process data to power applications ... 47
3.2.17 Set points to generators ... 47
3.2.18 Historian used to update application server data... 48
3.2.19 Historical data sent to historian... 49
3.2.20 External programs using API in the SCADA server ... 50
3.2.21 External programs using APIs in the historian ... 51
3.2.22 Operator HMIs interface the historian ... 52
3.2.23 Web browsers used to access the historian ... 53
3.2.24 The historians API is used by office desktops’ applications ... 54
3.2.25 External programs interface the application server directly... 55
3.2.26 Viewing and changing engineering data... 56
3.2.27 NIS/GIS applications updates engineering data ... 56
3.2.28 Data interchange with other DE systems ... 57
3.2.29 Define structure, views etc ... 58
3.2.30 Disturbance records sent for analysis... 59
3.2.31 Settings and parameters for RTU and substation equipment... 60
4 Control center architecture patterns...61
4.1 Zones and placement of services... 61
4.1.1 Demilitarized zone architecture ... 63
4.1.2 Single firewall architecture ... 66
4.1.3 Isolated zone architecture ... 68
4.1.4 Variants ... 69
4.2 Services added or subtracted... 73
4.2.1 Thin HMI client ... 74
4.2.2 HMI server gives access to operator client in the office... 74
4.2.3 No separate server for remote HMIs ... 75
4.2.4 SCADA replica in the DMZ ... 75
4.2.5 An interface server in the DMZ... 76
4.2.6 Separate inter control center communication front-end ... 77
4.2.7 External DMS system ... 78
4.3 Bundling of services ... 78
4.3.1 One HMI for Data engineering and Operators stations ... 80
4.3.2 Data engineering directly in the SCADA server ... 80
4.3.3 Phasor data concentrator and front-end coupled... 80
4.3.4 Distributed SCADA, power application and historian ... 81
5 Identified cross-boundary data flows...82
5.1 Substation communication ... 82
5.2 Inter control center communication ... 82
5.3 Data sent to emergency control center... 82
5.4 Historical data going out from the control center ... 83
5.5 Remote operator stations... 83
5.6 Suppliers dialling in ... 83
5.7 Office services updating control center historian ... 84
5.8 Office services writing to and reading from SCADA server ... 84
5.9 Office services updating control center power application server ... 84
5.10 Data engineering updated by external systems ... 84
5.11 Updates from quality assurance zones ... 85
5.12 Office and internet applications used in the control center... 85
6 Conclusions ...86
6.1 Clear patterns exists in SCADA system architectures... 86
6.2 The architecture is partly contingent on the type of utility... 86
6.3 The share of legacy systems are substantial ... 87
6.4 It is common that several types of data are exchanged with SCADA systems 87 6.5 The DMZ is not always a “real” DMZ ... 88
7 References ...89
8 Appendix A –Substation ICT architectures...92
8.1 The Single RTU architecture ... 92
8.2 The field bus/process bus based architectures... 93
8.3 Data flows and relevant communication protocols... 95
8.4 References... 97
9 Appendix B – Power System Communication Networks ...98
9.1 Communication media ... 98
9.2 Communication Topologies and Network scale... 99
9.3 Communication protocols ...101
9.4 Further reading ...103
10 Appendix C – Questionnaire... 104
Table of Figures
Figure 1 – Examples of SCADA System vulnerabilities... 10
Figure 2 – Models used in VIKING, this report focus on the SCADA system. ... 11
Figure 3 – Deliverable 2.3 in relation to other VIKING-deliverables. ... 12
Figure 4 – Architecture metamodel and annotated example... 16
Figure 5 – Focus of this report... 18
Figure 6 – The focus of chapter 3. ... 23
Figure 7 – Overview over services and data flows described in this section. ... 24
Figure 8 – Section 4.1 in relation to the architectural metamodel. ... 61
Figure 9 – The demilitarized zone architecture... 64
Figure 10 – The two zone architecture... 67
Figure 11 – The isolated architecture. ... 69
Figure 12 – The separate front-endzone architecture. ... 70
Figure 13 – The back-office zone. ... 71
Figure 14 – The quality assurance zone. ... 72
Figure 15 – Added or removed services and the implication this... 73
Figure 16 – Thin client in the architecture. ... 74
Figure 17 – HMI server hosts operator clients located in the control center. ... 74
Figure 18 – No separate server for remote HMIs. ... 75
Figure 19 – SCADA replica in the DMZ... 76
Figure 20 – Interface server in the DMZ. ... 77
Figure 21 – Inter control center front-endand its placement. Alternative a) is in the DMZ architecture, alternative b) is in the other architectures. ... 77
Figure 22 – External DMS system. ... 78
Figure 23 – The focus of this section – how services are coupled to each other... 79
Figure 24 – A single HMI application for data engineering and operators. ... 80
Figure 25 – A single server for for data engineering and SCADA. ... 80
Figure 26 – front-endimplementing the phasor data concentrator... 80
Figure 27 – Distributed server architecture. ... 81
Figure B.28 – Common Physical Communication Topologies (a) Star, (b) Ring, (c) Bus (d) Meshed 99 Figure B29 – An example network showing a control system, communication network.101 Figure 30: An example network showing protocols. ...102
1 Preface
The VIKING project is an EU financed Framework 7 Collaborative STREP Project and is part of themes 4, ICT, and 10, Security. VIKING stands for Vital Infrastructure, NetworKs, INformation and Control Systems ManaGement and will be executed between November 1, 2008 and October 31, 2011 by a consortium of industrial and academic partners.
1.1 Overview of the VIKING project
The main objectives of VIKING are:
To investigate the vulnerability of SCADA systems and the cost of cyber attacks on society
To propose and test strategies and technologies to mitigate these weaknesses
To increase the awareness for the importance of critical infrastructures and the need to protect them
Society is increasingly dependent on the proper functioning of the electric power system, which in turn supports most other critical infrastructures: water and sewage systems;
telecommunications, internet and computing services; air traffic, railroads and other transportation. Many of these other infrastructures are able to operate without power for shorter periods of time, but larger power outages may be difficult and time consuming to restore. Such outages might thus lead to situations of non-functioning societies with devastating economical and humanitarian consequences. For this reason, this consortium has decided to concentrate its research to the systems for transmission and distribution of electric power. We anticipate that most of the results will be applicable to the protection of other critical infrastructures.
The operation and management of the electric power system depend on computerized industrial control systems. Keeping these systems secure and resilient to external attacks as well as to internal operational errors is thus vital for uninterrupted service. However, this is challenging since the control systems are extremely complex. Yet, the systems are operating under stringent requirements on availability and performance: If control and supervision are not done in real-time, the power network may come to a collapse.
These computerized control system, normally called SCADA standing for System for Control And Data Acquisition, includes functions for remote collection of vast amounts of data from measurements placed in strategic points, e.g. power stations, in the geographically widely spread processes and for the remote control of process devices.
Many SCADA systems include computerized models of the process which enables simulation of alternatives process states and of optimization. Due to legal and environmental constraints, e.g. for building of new high voltage power lines or power stations, the primary process itself is difficult to expand which in its turn leads to higher and higher utilization of the existing transmission and generation resources. The process is, in other words, operated closer to its physical limits. Those the SCADA systems are becoming increasingly critical for the operation of the process and therefore are becoming a critical component for the availability and security of the supervised infrastructure.
The objective of the VIKING project is to develop, test and evaluate methodologies for the analysis, design and operation of resilient and secure industrial control systems for critical infrastructures. Methodologies will be developed with a particular focus on
increased robustness of the control system. As mentioned, the focus is on power transmission and distribution networks. The project combines a holistic management perspective—in order to counteract sub-optimization in the design—with in-depth analysis and development of security solutions adapted to the specific requirements of networked control systems.
The traditional approach to verify the security of SCADA systems has been ad-hoc testing of existing commercial SCADA system in laboratory environments. The systems to be examined have been installed in different labs, e.g. Idaho National Labs, and tested by skillful people searching for cyber attacks vulnerabilities. The focus in these tests has been on the protection of the central computer system of the SCADA system, since the central computer system has most connections to the external environment through office networks and Internet.
In the VIKING project we will take an alternative and complementary approach to SCADA system security. Firstly we will study the whole control system from the measurement points in the process itself over the communication network to the central computer system as illustrated in the following picture with the yellow exclamation marks indicating potential targets for cyber attacks.
Figure 1 – Examples of SCADA System vulnerabilities
Secondly, and more importantly, we take a model-based approach to investigating SCADA system vulnerability. Models are defined for the SCADA system, for the electrical process as well as of for the society that is dependent on the electricity supply. The society models are used to evaluate the economic consequences coming from disturbances in the electricity supply. The power system models are in turn used to evaluate the effects on the electricity supply by SCADA system misbehavior. Finally, SCADA system models are employed to assess the effect on SCADA system behavior by
cyber attacks. Based on analysis performed on these models, VIKING will propose mitigation actions to be taken to decrease or to eliminate these risks. The results of the project will be evaluated on a test-bed that can be configured to simulate the critical infrastructure of a power network and a range of attacks.
The modeling approach is indicated in the following picture.
Figure 2 – Models used in VIKING, this report focus on the SCADA system.
With this approach the project hopes to achieve the following research results.
Estimates of the security risk and consequences (in terms of monetary loss for the society) based on threats trees, graphical system architecture and society models
Comparable, quantitative results for IT security for different control system solutions and implementations
Use of existing model based application as application level Intrusion Detection Systems to detect manipulation of data
Use of innovative and existing communication solutions to secure power system communication
Help with identifying ”weak spots” and how to mitigate them
An environment for performing what-if analyses of the security risk impact of different architecture solutions.
This report, D2.3 of the VIKING project, address the SCADA system and models of it.
From security requirements to social costs
SCADA system
Power network
Societal cost Attack
1.2 Purpose of deliverable 2.3
The aim of deliverable 2.3 in the VIKING is to catalogue architecture patterns or reference architectures, i.e. commonly deployed solutions, for SCADA systems. These patterns are represented as a set of descriptions that capture the vast majority of SCADA systems’ architecture on a high level. The patterns developed in this report focus on:
Software services in SCADA systems and software services which SCADA systems exchange data with.
Data flows among these services.
How services are placed in different security zones (network zones).
The purpose of the SCADA architecture patterns is to clarify how SCADA systems are commonly designed by employing a stringent model framework. Internal in the project the SCADA patterns will be used to develop SCADA system design models that reflect some typical systems deployed in industry. These models will be used in other work packages and deliverables in the VIKING project. In particular, the patterns is an important input to deliverable 3.1 (Architecture based vulnerability assessment).
To support deliverable 3.1 the architecture patterns presented in this report should be a partial instantiation of the metamodel presented in deliverable 2. 1 (Threats and vulnerabilities – initial report). For a more detailed description of that metamodel, the reader is referred to deliverable 2.1 (Threats and vulnerabilities – initial report) of the VIKING project and the upcoming deliverable 2.2 (Threats and vulnerabilities – final report) which will complement deliverable 2.1 with probabilistic data on attacks’
likelihood of succeeding.
Figure 3 – Deliverable 2.3 in relation to other VIKING-deliverables.
Deliverable 2.1 will be used in task 3.1 (Architecture-based vulnerability assessment) to enable analysis of SCADA system vulnerabilities and risks. In task 3.1 the architecture descriptions described in this report will be analyzed using the probabilistic vulnerability/attack models described in deliverable 2.1 and 2.2.
1.3 Scope
This report will focus on describing the services in and around SCADA systems, the data flows and how services are placed in different zones. It will focus on the state-of-practice and only briefly discuss the difference to state-of-art. The reason for this is that the purpose of the VIKING project starts off with the motivation to effectively enhance security in an operating legacy system environment. A state of the art solution can thus only be achieved over a very long period of time by changing different parts of the existing solution, not through a green field approach. Furthermore, emphasis is placed on control centers and data exchange with other systems and zones. Substation communication, the internal workings of SCADA systems/EMS (Energy Management Systems)/DMS (Distribution Management Systems) and the internal workings of substations are not the focus of this report. This does not mean that these aspects are unimportant from a security perspective. Tentative diagrams for substation architectures and wide area communication architectures are given in Appendix A. Some issues of the internal workings of EMS are addressed in other reports of the VIKING project.
1.4 Document overview
This report gives a qualitative and quantitative overview over architectures used in the installed base of SCADA systems. Chapter two of this report will describe the theoretical framework and method used to survey these architectures. This includes a description of the metamodel used throughout the survey, literature studied on this topic and the interviews with SCADA system vendors.
Chapter three gives an overview of the reference model developed throughout this project. A set of services and data flows are described here. Chapter four continues with a describing how these services and data flows are instantiated in different architectures.
First it is described how services are placed in zones in the general case. Thereafter special solutions where services are added or subtracted are described. Lastly, in chapter four different ways to bundle services into software components are described.
Chapter five summarize the architectures from the viewpoint of cross-boundary data flows, i.e. data flows that cross the boundary to the SCADA system’s zone. Chapter six concludes the findings in this report.
This report focuses on services associated with the central part of the SCADA system, primarily those of the control center. In appendix A and B an overview of substation automation systems and control center to substation communication is given.
2 Method and theoretical framework
The architectures presented in this report have been identified through a literature study and number of interviews with SCADA system vendors. This chapter will describe the literature review, the metamodel used to document the architectures and the interview procedure as well as the respondents.
2.1 Literature review
This study was initiated with a review of literature where state-of-practice and state-of- art for SCADA system architectures where discussed. The literature found through this review is presented below under the categories:
Security literature
SCADA and control system literature
Market surveys
Literature found in all these categories describes SCADA systems from one viewpoint or another. In some cases there is also qualitative or quantitative statements indicating how common different variants are. Although this literature was used as input to the models described in this report they did not describe the state-of-practice to the amount of detail given in this report (hence the need for interviewing vendors). A more detailed description of the reference material found under these categories is given below.
2.1.1 Security literature focusing on SCADA systems
Architectures can be described from different viewpoints and with different views [1].
When security is to be assessed from the model some aspects are more important than others. Descriptions of SCADA system architectures given in security related literature can therefore be expected to be more relevant for the VIKING project than models described elsewhere.
A number of texts on SCADA security have been published in recent years. This includes standards and guidelines such as NERC CIP [2], NISTs special publication 800-82 [3], ISA-99 [4], IEC’s standards on secure substation and SCADA communication[5,6,7,8,9], NICSS practice guide for firewall deployment[10], the Secure Security Procurement language[11], and much more. Academic writing on this topic can be found in journals (e.g. [12,13,14,15,16,17,18,19,20]), and in greater quantity in conferences proceedings ( e.g. [21,22,23,24,25]). A number of textbooks (e.g. [26] and [27]) also exist on this topic.
Practically all of these provide schematic descriptions of SCADA systems and describe common vulnerabilities in these. For instance, in [3] NISTs special publication 800-82 and ISA-99 network diagrams over typical SCADA systems are given, and in [4] several aspects of SCADA communication is described.
While these references describe SCADA systems they do so to a limited extent. The descriptions included in these texts are intended to be general introductions to SCADA systems. Consequently these publications focus on some central aspects and disregard many variations that exist. Most commonly the architectures are described in terms of which physical nodes and links that exist. The logical distribution and division of services/functions and what data and data flow that is present are more implicitly addressed. It is also difficult to combine the descriptions into one coherent architecture
description. Both because the metamodels used are unclear and because the references contain a mixture of descriptions of typical architectures, vulnerable architectures, and best-practice solutions.
The work in this category that appears to be most closely related to the work presented in this report is the reference model presented in [28]. In this reference model four levels are identified: oversight entity, system and plant control centers, SCADA field equipment and infrastructure equipment. An object-role model over the services (objects) in these domains is outlined in the report and some security issues associated with the relations are discussed. In relation to [28] this report further detail the services, their relationship to each other (data flows in between these), and the levels (zones) related to these systems.
2.1.2 SCADA and control system literature
There are plenty of textbooks, articles and reports that describe SCADA systems in general. This includes:
Textbooks such as [29,30,31,32,33,34]
Articles such as [35], [36] and [37,38]
Reports such as [39], [40], [41] and [42].
These references include both schematic overviews of SCADA systems and detailed descriptions of their components. Some of them also provide empirical data on how common different solutions, services, or configurations are. None of the references found does however provide an encompassing description of common architectures. For example, the quantitative data given in [36] is limited to Finnish distribution utilities.
Moreover, many references describing SCADA systems are dated several years back.
Other are stated as state-of-practice descriptions or likely future architectures, see for example [38] and [42]. Their accuracy with respect to the SCADA systems now installed is thereby questionable and worth investigating.
The descriptions given in SCADA and control system literature have been used as an input to this work. The models presented in this report are also overall aligned with the descriptions given in literature. The interface reference model described in [41] was for example used as input when services where identified. From a particular viewpoint (described in section 2.2) this work has investigated how common different variants are.
2.1.3 Market surveys
Quantitative data on how common different architectures are is rarely described in academic literature or standards. It is however a common component in market surveys.
The market trend digests produced by Newton Evans Research Company1 does for example provide data on what security strategies utilities apply, what security measures that are being implemented, how common outage management functionality is and how common different protocols are in different regions. Also other actors produce market surveys, e.g. [43].
These market surveys have also been used as an input to this work, although to a limited extent. None of the surveys found has had the encompassing scope of this work, but
1 Official website: http://www.newton-evans.com/
rather surveyed what is a common solution for a certain components in the SCADA system.
2.2 Metamodel
In order to describe the SCADA reference architectures in this report an explicit metamodel has been developed. A metamodel is the model language and describes what the architecture model can, and should, contain. The vast majority of the literature found throughout the literature survey did not clearly state the scope and content of the metamodel used. This section describe the metamodel used for the survey used in this report and gives a rationale for it.
2.2.1 Scope and definition
The architecture models described in this report include a set of services that has been identified. A service in these models is realized by one or more software component(s).
The components as such are however not explicitly regarded as entities in their own right here. Examples of services include, “Engineering stations (HMI)” and “Data Engineering Server”. A service can be physically distributed on different machines, for example spread over multiple computers. It can also be replicated in the architecture; it is for example common that multiple Data Engineering Clients are instantiated in the architecture. Neither of computers is considered explicitly.
Figure 4 – Architecture metamodel and annotated example.
A service belongs to a zone. A zone in this metamodel has a close relationship to a network zone in computer networking. It should however rather be seen as a security domain as defined in [44], i.e. “...a set of security elements subject to a common security policy defined and enforced by a single security policy authority”. In the notation used a zone is represented as a frame around the elements (services) included in it. If there is a tunnel (e.g. a virtual private network, VPN) that makes it possible to join one zone from another zone this is also represented. In that case this represented by a block- arrow between the zones.
Data can be exchanged between services. The element data flow is included to represent this. A data flow goes between two services and data is written to either one or both
sides (services). A line connecting two services in the notation used represents a data flow that goes between them. If an arrow is at the end this end is written to. In this report focus is on data flows that hold an apparent and direct business value is included.
This means that data flows that only provide infrastructure services, e.g. the DNS and ARP, are not included.
2.2.2 Metamodel rationale
The metamodel used in this study only represent a subset of the information needed to assess security. VIKING deliverable 2.1 presents a metamodel for security analysis. As can be seen in Figure 4 the vast majority of the elements from this model are not covered here. Examples of elements that are excluded are authentication functionality, authorised users, persons, protocols, operating systems, hardware, physical zones and intrusion detection functionality. Consequently, the metamodel also excludes many concepts that are discussed in SCADA security literature.
The Purpose of this work is however to focus on what is specific for SCADA systems. That is foremost the division of services and corresponding dataflow. Of course, a larger scoped metamodel could have been used but due to limited time and resources in the project this was not prioritized. In addition, as technical details, and in particular details related to security solutions, are often confidential this type of information is more difficult to both get hold of and publish. Hence, a further argument for limiting the detail level was to avoid difficulties were when collecting empirical data on architectures.
The elements that are included in this metamodel is however central to security analysis.
Classical security models such as the Bell-LaPadula model [45] and Biba model [46]
focus on data flows between security levels. Security levels can be seen as the security zones used in this metamodel. Security domains are also described as a powerful concept in [44], and the implications of interactions and relationships between domains are explicitly discussed.
Modelling notations specifically developed to support security analysis also support the simplification of architecture models made here. In e.g. threat modeling procedures described in [47] and [48] actors, data flows, processes, and trust boundaries are included which is closely related to the data flows, services and zones used in this report.
The elements included here are also included in deliverable D2.1 “Vulnerability models”
of the VIKING project. A zone in this model is in D2.1 a “network zone”.
A further argument for the metamodel used here is that it is similar to other metamodels with a closely related purpose. In [49] the “Unified logical architecture for the smart grid”
is depicted through a set of boxes (services) and arrows (data flows) over different domains.
2.3 Interviews with vendors
The reviewed literature was in combination with the experience possessed by consortium members used to create a set of draft models over SCADA systems. These models complied with the metamodel presented in section 2.2 and outlined the software services, the data flows between these, and the services their placement in different zones. As this baseline model was developed from the consortiums experience it was expected to be biased towards architectures and solutions commonly implemented by ABB. To validate that these architectures are prevalent in the industry, identify other
architectures and collect data on how common different architectures are a number of SCADA system vendors where contacted.
This section will describe:
the scope and delimitations of this survey,
the baseline models,
the respondents interviewed,
how the qualitative descriptions of architectures were validated,
how quantitative estimates of how common these architectures are was collected
Each of these is addressed in a subsection below.
2.3.1 Scope and limitations
The architectures and architecture-variants presented in report comply with the metamodel described in section 2.2. The data flows and services covered in these architecture diagrams are those that offer SCADA system functionality or services closely related to this functionality. In simpler terms, this study covers the services located in SCADA system control centers and the services that exchange data with these services.
This means for instance, that services located in the office are included if they exchange data with the SCADA system; other services exchanging data with the services located in the office are however not included.
A key feature of most control systems is that they are implemented with some degree of redundancy. Redundancy is often implemented for both infrastructure, e.g. as network connections and network devices, and for central services such as the SCADA server. The redundancy typically implemented for services in SCADA systems are briefly discussed in chapter 3.Redundant services is however not included in the architecture descriptions and not explicitly covered during the interviews with vendors. Chapter 3 also indicates which services share infrastructure software (i.e. machine and operating system) although it is not included in the metamodel.
Figure 5 – Focus of this report.
The focus of this report and the interviews with suppliers has been placed on services located in control centers and their relationship to other services. This means that substation automation systems, e.g. the communication between intelligent electronic devices, are not included. The focus on the control center and the absence of technical details in the metamodel also means that the technical realization of the communication between substations and the control center falls out of scope of this report. While these
two parts of the SCADA system is not fully detailed in this report an overview of them is given in appendices. An overview of substation automation systems is described in Appendix A and an overview of substation communication solutions is given in Appendix B. The architectures described in these appendices are based on a literature review and has not been empirically validated. This is not the case for the descriptions given in chapters 3, 4 and 5. The content of these chapters has been validated.
2.3.2 Baseline models
Prior to the interviews with SCADA system vendors a set of architecture models over SCADA systems were developed. These where drawn from descriptions given in literature and processed by members within the consortium.
The literature used as input to this process is described in section 2.1. Consortium members from the Royal Institute of Technology (KTH) and ABB was involved in creating these baseline models. The intention was to create generic models over SCADA system architectures. However, as ABB was the only SCADA system vendor involved in this process the baseline was expected to be biased towards the architectures and solutions that ABB offers.
The baseline models were used to produce a questionnaire used throughout the interviews with SCADA system vendors. During the interviews this baseline model was presented to the respondents and deviations, amendments, and comments to this baseline model were noted. The baseline model can be found in appendix C of this report. In appendix D the questionnaire used throughout the interviews, is shown.
2.3.3 Respondent description
A substantial effort has been placed on finding suitable respondents. The consortium contacted the following vendors with a request to meet for an interview: ABB, Siemens, PSI, Netcontrol, Areva and General Electric. When this report is issued the first four has accepted to meet and also been interviewed. In Table 1 an overview of these vendors and their typical customers is given.
Table 1 –Overview of vendors in this study.
Vendor Product Typical customer
ABB ABB Network manager Transmission utilities or larger distribution utilities worldwide.
ABB MicroScada Small or medium sized distribution utilities in Europe, the Middle East or Asia.
Siemens About 10 (including legacy
systems) Transmission and distribution utilities worldwide.
PSI PSIcontrol Transmission and distribution utilities in the German region or Russia.
Netcontrol Netcon 3000 Small or medium sized distribution utilities in Sweden or Finland
ABB have a long history in the market of SCADA systems. The product Network Manager and its ancestors ABB Ranger and ABB Spider are installed in about 500 control centers around the world. Network manager is primarily used on the transmission level or in
larger distribution utilities. Ranger-based systems are mainly installed in the US and Australian markets and Spider-based systems are mainly installed in the European, African, and Asian markets.
ABB MicroSCADA is another ABB-product. ABB MicroScada is targeting distribution utilities and is mainly installed to operate power networks on the distribution level. More than 1000 MicroScada systems are installed in control centers in the world today, primarily in the European and Asian market.
Siemens are also seasoned in this market. Around ten different products or product versions are included in the installed base of Siemens system. Their systems are installed on all voltage levels throughout the world, but the respondents interviewed in this study do not have great experience from the US market.
PSI is a SCADA system vendor with the base in the German market. Their SCADA system solution is in use on both the transmission level, sub-transmission level and on the distribution level. While the vast majority of PSI’s systems are installed in Germany, some installations also exist in other countries and in other continents, e.g. in France, Switzerland, Rwanda and Russia.
Netcontrol is as ABB MicroScada focusing on the distribution market. Their customer base is small to medium sized utilities in Sweden and Finland. Around 110 Netcontrol systems are running in control centers in these countries.
Some vendors market multiple products and have a clear division between different regional markets. Therefore multiple interviews were held with different respondents at ABB and Siemens. An overview of the respondents, their experience and the focus of the interview with them is given in Table 2.
Table 2 –Respondents in interviews.
Vendor Respondent Experience
working with SCADA systems
Experience
working with the vendor and/or systems
Focus of interview
ABB Neela Mayur - - Network
Manager, US market ABB Gunnar
Bjoerkman 34 34 Network
Manager, rest of the world
ABB Mikael Molander 21 21 ABB MicroSCADA
Siemens Hans-Joachim
Diehl 34 35 All
Gilbert Suter 38 10
(38 with Landis &
Gyr)
Distribution
Erich Wuregler Distribution
Netcontrol Mats Elmgren 41 7 Distribution
Lars-Gunnar Lif 33 15 Distribution
PSI Wolfgang Fischer 22+ 22 All
Joachim 18+ 18 All
ABB has gone to the market with two solutions: ABB Network Manager and ABB MicroSCADA. Neela Mayur represented ABB Network Manager for the American market and Australian market. This also includes the previous installations of the product RANGER. Gunnar Bjoerkman represented ABB Network Manager for Europe and the rest of the world, which includes previous installations of the products known as SPIDER.
Mikael Molander represented ABB for the product MicroSCADA.
Siemens also market multiple SCADA system products. To get an overview of these three respondents were interviewed. Gilbert Suter and Erich Wuregler are based in Switzerland and the interview focused on Siemens solutions for power distribution (lower voltage levels). Hans-Joachim Diehl confirmed the output from the interview with Gilbert Suter and Eric Wuregler. Hans-Joachim Diehl also complemented this interview with variants and data applicable to Siemens solutions on transmission and sub-transmission.
Netcontrol was represented by two respondents who gave an overview of the vendor’s architectures as a whole. As Netcontrol’s customers are power distribution utilities this was the focus of the interview.
Also PSI was represented by two respondents who gave an overview of the vendor’s architectures as a whole.
2.3.4 Qualitative validation
As outlined by the questionnaire in Appendix D the interviews were initiated by an overview of the services and services included in the baseline model. This was followed by a presentation of three different network zone models and how services are placed in these zones. From this follows that certain data flows will cross over zones and some will not.
Throughout this procedure respondents were encouraged to give comment to these architecture descriptions and comment unfamiliar or missing components. This includes the services included in the descriptions, the data flows that go between these, the zones included in the descriptions and how the services were placed in different zones.
This part of the interview lasted between one to three hours. Three hours was spent by PSI whose SCADA system has, in some aspect, a different architecture2 than the baseline architecture and approximately one hour for the other respondents.
All comments and deviations where noted throughout the interviews. Once documented in an electronic and readable format the respondents has had the opportunity to review and comment these interview protocols.
2.3.5 Quantitative estimates of commonality
The second part of the interviews was dedicated to quantifying how common different elements are in the installed base of systems. As outlined by the questionnaire in Appendix D this includes quantifying how common different zone models are, how common different data flows are and quantifying how common it is that services share one machine.
Although archival records (e.g. specifications from delivered projects) might be obtainable for these quantities they were all made based on the respondents’ own
2 PSI’s SCADA system is “distributed” unlike the other vendor’s SCADA systems.
experience. The respondents in this study have a substantial experience from both SCADA systems in general and the products offered by the vendor they represented.
This, and the difficulty to persuade respondents to retrieve archival data for these quantities, are the primary reasons for relying on the respondents own experience and judgment.
It should be noted that many of the estimates obtained where in the form “about half of the systems”, “the vast majority of systems” or “only some installation has that”. These estimates are also, according to the respondents themselves, associated with a substantial uncertainty in some cases. In this report these estimates are presented together with the components and variants they are associated to, see chapter 3 and chapter 4. When they are presented, they are presented as they were expressed by the respondents.
These estimates are also presented independently of each other. There is however often a clear dependency between the architectures. For example, modern SCADA system architectures often have the demilitarized zone architecture. Also, it is modern SCADA system architectures that use quality assurance zones and the modern SCADA systems exchange data with office systems more often than old legacy systems. Although these dependencies apply to many of the variants they are not surveyed in this work and is described in this report. Further work could explore the relationship between different patterns and how common different combinations of architecture variants are.
3 Services and data flows
This chapter will present a number of services and data flows present in SCADA system environments. As described in section 2.3.1 this work aims to identify the services that located within the control center, or is directly interfacing services in the control center.
In relation to the metamodel (cf. Figure 6) this chapter will first describe the services in the reference model used throughout this study, thereafter the most common data flows are describe
Chapter 3 thus provides a list with descriptions of the building blocks services and data flows. The actual architectural patterns describing how the services are placed in zones and how this influence the data flows is described in chapter 0. Chapter 0 also describes replicas of these services and any special solutions identified in this study.
Writes to
Service
Zone Dataflow
1..*
1..1 2..2 1..2
Goes between
Belong to 0..* 0..*
0..*
0..*
Can join
Figure 6 – The focus of chapter 3.
Figure 7 gives an overview of the services and data flows described in this chapter.
Data Engineering server
[29] Define data structure, views etc [27] NIS/GIS applications updates engineering data
Engineering stations (HMI) [26] Viewing and changing engineering data [28] Data interchange
with other DE systems [29] Define data structure, views etc
Office Servers and power applications located outside the control center
SCADA Server
Control center application Server
DMS (Application
Server) GMS
(Application Server)
[16] Process data to power applications EMS (Application
Server)
[17] Set points to generators Historian
[20] External programs using APIs in the SCADA server [21] External services using APIs in the historian
[19] Historical data sent to historian [23] Web browsers used
to access historian
[22] Operator HMIs interface the historian [24] The historians API
is used by office desktops’
applications
[25] External programs interface application server directly
Application software for the office (e.g. browsers, mail
clients)
[10] Operator commands to process and manual changes and markups of data
Operator HMI Client
[11] Operator views and alarms
sent to HMI Remote client
(e.g. Citrix client)
[14] Mail, websites etc
HMI Server (e.g. VNC, Terminal
Services or Citrix) [13] Terminal services
[12] A server hosts several HMIs Office application
software (e.g. spreadsheets)
[3] Front end sends commands to process Front End
RTU/SCS [6] SCADA server sends commands to
front end
[4] Front end data to real-time database
[2] Process measurements to
control center Phasor data
concentrator
Phasor measurement unit [1] PMU Measurements
[5] Phasor data sent to real-time database Inter control
center communication
server [9] Process data to/from
other control centers
[8] Data shared with inter control center server
[7] Data sent to emergency center(s)
[15] Read operator views and change parameters of power applications
[29] Define data structure, views etc
[29] Disturbance records
[30] Settings and parameters Web browser
[18] Historian is used to update application server data
Remote engineers’
workstations
Figure 7 – Overview over services and data flows described in this section.
3.1 Services
This section will describe the services (boxes) depicted in Figure 7. How these services are placed in zones, replicas of these services and services only used by some vendors are described in chapter 4. Data flows (arrows) are described in section 3.2.
3.1.1 Front-end
The front-end servers are responsible for the communication with the Remote Terminal Units (RTU) and Substations Control Systems (SCS) systems. They receive process data in raw format using standardized RTU communication protocols like IEC 60870-5- 101/104, DNP3.0 or MODBUS. However, a big variety of proprietary protocols are also still in use in operational system due to the very long life time of RTUs. In addition to data acquisition, the Front ends receive commands and setpoints from SCADA servers
and send these to the RTUs and SCS for execution. Traditionally point-to-point connections to the RTU/SA systems have been used but today there is clear trend to move to TCP/IP based protocols like 60870-5-104 and DNP over TCP/IP. For TCP/IP based protocols the front-end application is sometimes implemented in the SCADA servers and commercial routers are used to interface the SCADA zone to the Wide Area Network.
In some cases, like for the collection of local disturbance data, file transfer types of formats are received and sent to applications in the SCADA servers.
Normally the front-ends perform pre-processing of process data for the SCADA servers like conversion of raw digital formats to engineering values and translation of hardware addresses from the RTUs/SCSes to logical database addresses in the SCADA servers. The converted process data is transmitted to the SCADA servers (both online and standby) with very short time delays.
Front-ends are also responsible for the supervision of RTU communication lines and reconfiguration to redundant access paths if available. Statistics for the communication lines are collected like bit errors, number of retries and time out errors. Fronts Ends are normally redundant and support line-by-line failovers for point-to-point connections or complete failover between the two redundant front-end servers. The most common configuration is that the communication load is shared between the two Front Ends in a pair on a line-by-line basis but a single front-end is dimensioned to handle the whole traffic in case of failure of the other server. An alternative configuration is that one front- end handles the whole traffic and the standby is prepared to take over if the online server fails.
In some SCADA configurations data concentrators are used. These are servers that are placed geographically outside of the central control system in the communication network structure and are used to concentrate low speed RTU traffic to higher capacity communication lines. In many cases protocol conversion and front-end functionality are performed in the data concentrators.
3.1.2 Phasor measurement unit
In the recent years a new generation of measurement devices have been introduced.
These are called Phasor Measurement Units or PMUs. Through the use of GPS a very accurate time measurement can be done and process values previously not possible to measure are now attainable. One example of this if phase angle measurement, especially phase angle difference over power lines, which is a measure of the electrical stability of the network. Measurement are stored locally in the PMUs with very accurate time stamps and sent to phasor data concentrators (see next section) where special applications to calculate and present network stability exist.
PMU generate substantially higher volumes of data and required more bandwidth than traditional measurements.
3.1.3 Phasor data concentrator
The phasor data concentrator collects data from PMUs using PMU protocols IEEE 1344- 1995 or C37.118 and sends this data either to special applications in the SCADA servers or to dedicated applications servers for evaluation of network stability. The phasor data concentrator is normally implemented in the same computer as the front-ends. Some vendors implement the phasor applications directly in the front-ends.
