IN
DEGREE PROJECT ELECTRICAL ENGINEERING, SECOND CYCLE, 30 CREDITS
STOCKHOLM SWEDEN 2017,
Time Synchronization for Ethernet Communication based Line
Differential Protection
SURYA SEETHARAMAN
KTH ROYAL INSTITUTE OF TECHNOLOGY SCHOOL OF ELECTRICAL ENGINEERING
Abstract
Most of the industrial manufactures and designers of Power Substation Automation Protec- tion Control products are under pressure regarding the existing communication architecture of the Line Differential Protection Application. Currently the widely used technologies include SONET/SDH which are circuit switched networks, that are no longer predominant. They are been replaced with Packet Switched Networks everywhere. Moreover dedicated network equip- ment is required to be maintained for using SDH/SONET technologies. Hence it is becoming increasingly hard for the utilities to get access to and maintain the SDH/SONET networks.
The main objective of this thesis work is to develop an alternative communication solution in order to migrate away from these SDH/SONET networks. The project work is done such that the requirements for the Line Differential Protection Communication are carefully analysed and a solution is designed such that it fulfils those requirements. Following that the imple- mentation and evaluation of the solution is performed. In the end, a testing of this solution is also done on a real-time WAN network running the Line Differential Protection function, in order to verify the working of the solution and study the challenges of a WAN deployment scenario in future.
Abstrakt
De flesta industriella tillverkare och designers till produkter av system f¨or stationsautomation ¨ar under press ang˚aende den existerande kommunikationsarkitekturen f¨or Linjedifferentialskydds- till¨ampningar. De f¨or n¨arvarande utbredda teknologierna som anv¨ands, som SONET/SDH, ¨ar kretskopplade n¨atverk vilket inte l¨angre ¨ar den dominerande n¨atverksprincipen. Dessa ¨ar p˚av¨ag att bli ersatta av paketf¨ormedlade n¨atverk. Dessutom kr¨avs det att dedikerad n¨atverksutrust- ning m˚aste underh˚allas f¨or att anv¨andas med SONET/SDH-teknologier. D¨arf¨or b¨orjar det bli kr¨avande och sv˚art f¨or verktygen att ges ˚atkomst samt underh˚alla SONET/SDH-n¨atverk.
Huvudm˚alet med denna uppsats ¨ar att utveckla en alternativ kommunikationsl¨osning i syfte att byta ut dessa SDH/SONET-n¨atverk. Arbetet ¨ar utf¨ort genom att f¨orst noggrant analysera kraven f¨or Linjedifferentialskydds-kommunikation f¨or att sedan finna och designa en l¨osning som uppfyller dessa krav. D¨arefter f¨oljer en implementation och utv¨ardering av den funna l¨osningen. I sista fasen genomf¨ors en testning av l¨osningen i ett realtids WAN-n¨atverk som k¨or Linjedifferentialskydds-funktionen, detta f¨or att verifiera hur den funna l¨osningen fungerar samt att studera utmaningar f¨or en framtida WAN implementation.
Acknowledgement
This thesis work has been conducted in collaboration with ABB Substation Automation Prod- ucts in V¨aster˚as as well as with the Industrial Information and Control Systems department at KTH in Stockholm. This project would not have been possible without a lot of people for whom I hold a great amount of respect. Hence I would like to express my gratitude towards them and acknowledge their roles in making this thesis work happen.
Johan S¨alj has been my supervisor at ABB. He has been a pillar of great support throughout the thesis project specially when it came to technical design and guidance. Bjorn Lexelius from ABB has also been very helpful and skilled when it comes to the IED communication area.
I am very thankful to both of them for sharing their broad expertise and for supervising my thesis in a systematic and dedicated manner. Moreover, it has been a great pleasure, working in the team of Henrik B¨ackstrand. In his capacity as a line manager he has helped me with all administration matters as well as with the arranging of the required technical equipment.
His timely advice has always kept me motivated.
Furthermore, my supervisor from KTH Fabian Hohn has been really impressive and of note- worthy help to me. During the course of my thesis work, he has always been there - giving suggestions for improvisation, clarifying doubts in case I got stuck on technical aspects, en- suring I stay on track with the project. Most importantly he has constantly encouraged me to do well right from the beginning. Professor Lars Nordstr¨om who is head of the department of Electric Power and Energy Systems at KTH was acting as the responsible examiner of the thesis. I would like to thank him for having given me this great opportunity to conduct the thesis with ABB.
Lastly, I would like to thank all my colleagues from ABB, friends from KTH and my lov- ing family in India for supporting and encouraging me throughout.
Contents
Abstract i
Abstrakt ii
Acknowledgement iii
List of Figures viii
List of Tables x
Nomenclature xi
1 Introduction 1
1.1 Problem Statement . . . 1
1.2 Problem Description . . . 1
1.3 Project Goals and Scope . . . 2
1.4 Project Outline . . . 2
2 Background 4 2.1 Line Differential Protection . . . 4
2.1.1 Working Principle . . . 4
2.1.2 Requirements of the Protection Interface . . . 5
2.2 Existing Industrial Solutions : SDH/SONET . . . 5
2.3 Motivation for IP-based Ethernet Networks . . . 6
2.4 Time Synchronization . . . 7
2.4.1 Inter-Range Instrumentation Group - B . . . 8
2.4.2 Network Time Protocol . . . 8
2.4.3 SyncE . . . 9
2.5 Precision Time Protocol . . . 9
2.5.1 Protocol Details . . . 10
2.5.1.1 PTP devices . . . 10
2.5.1.2 PTP Messages . . . 11
2.5.1.3 PTP Communication Path . . . 12
2.5.1.4 PTP Delay Mechanisms . . . 12
2.5.2 Working Principle . . . 17
2.5.2.1 Best Master Clock Algorithm . . . 18
2.5.2.2 Synchronization . . . 19
2.5.3 PTP Profiles . . . 20
2.5.3.1 PC37.238 IEEE-1588 : PTP Power Profile . . . 20
2.5.3.2 IEC/IEEE 61850-9-3:2016 : PTP Power Utility Profile . . . . 21
2.5.3.3 IEEE G.8275.1/G.8275.2 : PTP Telecom Profile . . . 21
2.6 PTP versus existing methods . . . 22
3 Methodology 24 3.1 Proposed Solution . . . 24
3.2 Satisfying the Requirements of the Protection Interface . . . 24
3.3 Approach . . . 25
3.3.1 Communication : Packet Switched Networks . . . 25
3.3.2 Synchronization : Precision Time Protocol . . . 25
4 Implementation 26 4.1 Resources Used . . . 26
4.1.1 Hardware Network Components . . . 26
4.1.2 Software Tools . . . 32
4.2 Network Topologies . . . 35
4.2.1 N1: Point to Point Connection between two IEDs . . . 35
4.2.2 N2: Introducing L2 switches between the IED’s . . . 37
4.2.3 N3: Three Ended Application - GPS to IED . . . 38
4.2.4 N4: Three Ended Application - GPS to switch . . . 39
4.2.5 N5: Hardware-in-the-Loop Trial with OMNET++ . . . 40
4.2.6 N6: Unlocked from GPS time reference . . . 41
4.3 Measurement Conditions . . . 42
4.3.1 Network Device Settings . . . 42
4.3.1.1 IED Configuration . . . 42
4.3.1.2 Switch Configuration . . . 43
4.3.1.3 GPS Clock Configuration . . . 43
4.3.2 Oscilloscope Settings . . . 43
4.3.3 Differential Current Settings . . . 43
5 Evaluation 44 5.1 E1: Functionality Testing . . . 44
5.1.1 Time Synchronization using PTP . . . 44
5.1.2 Line Differential Protection Communication over Ethernet . . . 44
5.2 E2: Performance Testing . . . 46
5.2.1 Time to Synchronize at Start-Up . . . 46
5.2.2 Time to Re-Synchronize . . . 46
5.2.3 Time synchronization Accuracy . . . 48
5.2.3.1 N1 : For the point to point Set-Up . . . 48
5.2.3.2 N2 : For introduction of L2 switches Set-Up . . . 49
5.2.3.3 N3 : Three-ended Application Set-Up . . . 50
5.2.3.4 N4 : Three-ended Application Set-Up with GPS to switch . . 51
5.2.3.5 N6 : Unlocked to UTC . . . 52
5.3 E3: Stress Testing . . . 54
5.3.1 Effect on Application Communication . . . 56
5.3.2 Effect on PTP Communication . . . 57
6 Feasibility for WAN Deployment 61 6.1 WAN Description and Set-up . . . 61
6.2 Testing and Results . . . 62
6.2.1 Case 1 : With the ABB v1.2 IEDs . . . 63
6.2.2 Case 2 : With the ABB v2.2 IEDs . . . 64
6.2.2.1 Case 2.1 : Data and Sync messages over Primary route . . . 66
6.2.2.2 Case 2.2 : Data over Secondary and Sync over Primary routes 66 6.2.2.3 Case 2.3 : Data and Sync messages over Secondary Route . . 67
6.3 Challenges during Testing . . . 69
6.4 Customer Requirements after the Testing . . . 70
6.5 General Industry Challenges for WAN Deployment . . . 70
7 Conclusion and Discussion 72 7.1 Conclusion . . . 72
7.2 Future Work . . . 72
Bibliography 73
Appendix 73
List of Figures
2.1 Line Current Differential Protection [3] . . . 4
2.2 PTP Clocks [16] . . . 11
2.3 PTP End-to-End Delay Mechanism [19] . . . 13
2.4 Hardware Timestamping during the transport of a Sync message [19] . . . 14
2.5 Working of peer-delay messages in Peer-to-Peer Delay Mechanism [20] . . . 15
2.6 PTP Peer-to-Peer Delay Mechanism [20] . . . 15
2.7 PTP Working Principle [16] . . . 17
2.8 PTP BMC Working . . . 19
4.1 RED 670 IEDs . . . 27
4.2 LDCM, IRIG-B and GTM Modules (left to right order) . . . 28
4.3 AFS 660 switch . . . 29
4.4 GPS clock and AFS677 switch . . . 29
4.5 GPS Antenna . . . 29
4.6 GPS clock signal relied to all network nodes in the lab . . . 30
4.7 Oscilloscope . . . 30
4.8 Communication Media . . . 31
4.9 Single Phase Relay Tester . . . 32
4.10 ABB AFS Finder 2.2.0.7 . . . 33
4.11 Wireshark capturing PTP sync messages . . . 34
4.12 Sender Application . . . 35
4.13 Point to Point set-up between two IEDs . . . 36
4.14 Introducing PTP compatible L2 switches . . . 37
4.15 Three ended application . . . 38
4.16 Three ended application with the GPS signal provided directly to the network . 39 4.17 HiL Network Setup . . . 40
4.18 Unlocked to UTC with IED acting as a PTP Master . . . 41
5.1 Functionality Testing . . . 45
5.2 Time required for Re-synchronization . . . 47
5.3 Density Plot for the PTP Slave’s deviation or time sync accuracy from PTP
Master . . . 49
5.4 Density Plot for the PTP Slave’s deviation or time sync accuracy from PTP Master . . . 50
5.5 Density Plot for both the PTP Slaves’ deviation or time sync accuracy from PTP Master . . . 51
5.6 Density Plot for the three PTP Slaves’ deviation or time sync accuracy from PTP Master . . . 52
5.7 Density Plot for the two PTP Slaves’ deviation or time sync accuracy from PTP Master . . . 53
5.8 Topology for Stress Testing . . . 54
5.9 Measuring Line Differential Current during Stress Testing . . . 57
5.10 Measuring Time Sync Accuracy during Stress Testing . . . 58
5.11 Time Synchronization Accuracy measured during stress testing . . . 60
6.1 WAN test network . . . 61
6.2 WAN set-up for v1.2 IEDs testing . . . 63
6.3 WAN set-up for v2.2 IEDs testing . . . 65
6.4 WAN set-up for v2.2 IEDs testing . . . 69
List of Tables
2.1 PTP Message Types . . . 12
2.2 Comparison of Time Sync Mechanisms . . . 22
5.1 Diff and Bias currents measured from N4 set-up. . . 45
5.2 Time to synchronize to 1 µs accuracy with the GPS signal after boot up of the network devices. . . 46
5.3 Re-synchronization time for the three Slaves with respect to the deviation en- countered with the GPS signal was removed for various time periods. . . 47
5.4 Configuration Parameter Values for Performance Testing . . . 48
5.5 Time Sync Accuracy (ns) for PTP Slave from PTP Master . . . 49
5.6 Time Sync Accuracy (ns) for PTP Slave from PTP Master . . . 50
5.7 Time Sync Accuracy (ns) for both the PTP Slaves from PTP Master . . . 51
5.8 Time Sync Accuracy (ns) for the three PTP Slaves from GPS clock . . . 52
5.9 Time Sync Accuracy (ns) for both the PTP Slaves from PTP Master . . . 53
5.10 Configuration Parameter Values for one instance of Sender Application . . . 55
5.11 Configuration Parameter Values for Stress Testing . . . 56
5.12 Diff and Bias currents measured from N4 set-up during Stress Testing. . . 57
5.13 Time Sync Accuracy (ns) Results for the three Slaves with variation in Addi- tional Traffic load . . . 59
6.1 Results for differential current from v1.2 IEDs over the primary route . . . 64
6.2 Results for differential current from v2.2 IEDs . . . 66
6.3 Results for differential current from v2.2 IEDs - part 2 . . . 67
6.4 Results for differential current from v2.2 IEDs - part 3 . . . 67
6.5 Results for differential current from v2.2 IEDs - part 4 . . . 68
7.1 Communication and PTP Configuration for RED670 IED . . . 77
7.2 Global Configuration Parameter Values for AFS665 L2 switch . . . 77
7.3 BC Configuration Parameter Values for AFS665 L2 switch . . . 78
7.4 BCv2 port configuration parameters . . . 79
7.5 TC Configuration Parameter Values for AFS665 L2 switch . . . 80
7.6 TCv2 port configuration parameters . . . 80
7.7 PTP Configuration for Meinberg LANTIME M3000 . . . 83 7.8 Oscilloscope configuration parameters . . . 84
Nomenclature
AFS ABB FOX Switch AP Access Point BC Boundary Clock
BMCA Best Master Clock Algorithm CPU Central Processing Unit
CSN Circuit Switched Network
FPGA Field Programmable Gate Array GPS Global Positioning System
GTM GPS Time Synchronization Module GM Grandmaster
HiL Hardware in the Loop HMI Human Machine Interface
IEC International Electrotechnical Commission IED Intelligent Electronic Device
IEEE Institute of Electrical and Electronics Engineers IP Internet Protocol
IRIG Inter-Range Instrumentation Group LAN Local Area Network
LDCM Line Data Communication Module LDP Line Differential Protection
LibPTP A Library for PTP Simulation
MAC Media Access Control MII Media Independent Interface
MT-RJ Mechanical Transfer Registered Jack NTP Network Time Protocol
OC Ordinary Clock
OMNeT++ Objective Modular Network Testbed in C++
OS Operating System PC Personal Computer
PCM Protection and control IED manager PPS Pulse Per Second
PSN Packet Switched Network PTP Precision Time Protocol SDH Synchronous Digital Hierarchy SFP Small Form-factor Pluggable SNTP Simple Network Time Protocol SONET Synchronous Optical Networking SyncE Synchronous Ethernet
TC Transparent Clock
UDP User Datagram Protocol UTC Universal Time Coordinate WAN Wide Area Network
Chapter 1 Introduction
1.1 Problem Statement
The main problem that this thesis work is trying to solve is in the area of Line Differential Protection (LDP) :
1. Most of the LDP relays currently communicate over SONET/SDH.
2. This requires dedicated communication equipment that is a wastage of resource.
3. Furthermore SONET/SDH are now becoming outdated and are being replaced by new technologies.
4. Hence there is a need to find an alternative method for the LDP relays to communicate.
1.2 Problem Description
Today most of the Line differential protection relays communicate over SONET (Synchronous optical Networking) or SDH (Synchronous Digital Hierarchy). These technologies provide a symmetric communication delay in both directions since they use the Time Division Mul- tiplexing mechanism. The symmetric delay is utilized to achieve time synchronization for measurement data between the IEDs. One drawback is that dedicated communication equip- ment is needed both inside and outside the IEDs.
Furthermore, SONET/SDH network equipment is being replaced by new technologies (like Ethernet/IP based networks). Thus it is hard for the utilities to get access to SONET/SDH networks. In addition, these IP networks can be shared for other communication requirements in power systems unlike the CSNs.
1.3 Project Goals and Scope
The main goals of this project are to : 1. Find a solution to the stated problem
2. Make pilot implementation set-ups using the ABB IEDs and get the proposed solution for LDP communication working :
(a) Get LDP working on a point to point set-up
(b) See the effect of adding asymmetry (switches) to the network (c) Use a simulator to test on a larger network
(d) Try a WAN testing of the solution
3. Perform an evaluation of the important parameters that are required to study the feasi- bility of the solution.
4. Validate the solution on a WAN network and find out the challenges in using this solution for a future WAN deployment.
Scope: This thesis mainly revolves around developing a modern-day solution for the com- munication of LDP Application, implementing it, and doing a laboratory based testing of the solution. The scope of the solution’s implementation and evaluation do not go beyond pilot laboratory set-ups and measurements.
1.4 Project Outline
The thesis project work is divided into Design, Implementation and Evaluation phases. The total duration of the thesis is twenty two weeks. The first four weeks are used to perform lit- erature study and propose a solution. The next nine weeks are used to design and implement the solution while the remaining nine weeks are used for the evaluation and testing.
Design Phase : This phase is divided into two parts - first is to do a literature study and come up with a feasible solution to the problem and the second is to perform a background study and get acquainted with the ABB internal tools which are required for the work.
Implementation Phase : This phase mainly involves working with the IEDs and other hard- ware/software equipments to set up pilot network models to test the LDP over the proposed solution, keeping in mind the requirements as per the standards. The main goal of this phase is to try out variants of the solution and ensure the working and monitoring of the behaviour.
Evaluation Phase : This phase involves the measurement of some of the performance metrics
as well as a full on testing in a WAN deployment scenario. Possible future work is also analysed which is mentioned in the Conclusion Chapter.
Chapter 2 Background
2.1 Line Differential Protection
Due to the quick and inherently selective fault detection, Line Differential Protection is an im- portant protection scheme. According to ANSI/IEEE C37.2-2008 [1], the communication and its corresponding time synchronization are two of the most important parts of this protection scheme.
2.1.1 Working Principle
The working principle behind Line Differential Protection is based on Kirchhoff’s Current Law.
This law can be stated as; “The sum of the current flowing into a node has to be equal to the current flowing out of this node” [2]. The operating principle of differential protection scheme is shown in Figure 2.1.
Figure 2.1: Line Current Differential Protection [3]
Figure 2.1 shows the simplified version of the working principle of the fault/delta current by summing up the measured inflow/outflow current values. If the inflow current equals the out-
flow current, they cancel each other and on the other hand a fault occurs if the line differential current delta is not zero.
In order to transfer and evaluate the measurement data, two different methods can be used.
One is to calculate phasors for each period or a predefined number of periods and only trans- mit this phasor value. Second is to transfer all the sampled values instead of a single value.
The second method has an advantage over the first, since transfer of all the sampled values would contribute towards providing more options for the future algorithms that detect faults [4]. Hence having a high bandwidth channel and a precisely accurate time synchronization mode are two of the most important requirements to be able to transfer all the sampled values within the stipulated time accuracy.
2.1.2 Requirements of the Protection Interface
The communication between the relays is the most essential factor for the working of the line differential protection. Only if the data from the remote relay is communicated correctly along with a precise timestamp, the relay can calculate the difference in current.
Taking the communication architecture into consideration, there are two very important phe- nomenons [2] - the path delay, also called the latency and the path delay variation (PDV), also called as jitter. Latency can be defined as a delay occurring in the communication of the data information to the remote relay. This can cause a variation in the tripping time (based on the amount of delay) and hence is an important point that has to be overcome or taken care of in the fault detection algorithms that are designed. The recommended latency values are stated in the IEC standard 61850-90-1 [5]. Based on the amount of applied voltage, usual range requirement for the maximum latency is between 5 ms and 10 ms.
Like latency has an influence on the communication of data through the channel, jitter has an influence on the time synchronization’s accuracy. The recommended jitter values are stated in the IEC standard 61850-90-1 [5]. Based on the required fault current sensitivity, usual range requirement for the synchronization accuracy is less than 10 µs. Hence lesser the jitter, higher the time synchronization accuracy and consequently quicker and more precise detection of faults is possible.
2.2 Existing Industrial Solutions : SDH/SONET
A traditional and widespread means of communication, in the field of teleprotection is SDH.
In simple terms, SDH can be defined as a standard technology used for data transmission in a synchronous manner using optical communication media [6]. SONET is a similar protocol standard like SDH, which is used in the US. Before these technologies, there was the tradi-
tional PDH (Plesiochronous Digital Hierarchy) technology that was used. The SDH/SONET technologies were invented in order to provide faster communication with lesser costs than PDH.
SDH communication network is based on TDM (Time Division Multiplexing) which, using fixed time interval slots, blocks chunks of bandwidth for the various services running in the network that are in need of them. Due to the fact that TDM needs a fixed bandwidth, which is reserved for the needed timeslots, it provides a symmetric communication delay in both direc- tions. This symmetric delay is utilized to achieve time synchronization between the protection devices [2]. A major drawback of SDH is that it does not provide effective bandwidth utilization.
The SDH protocol has a complex structure due to the interweaving of the header part across the data part. However this feature allows for the data that is encapsulated to have its own freedom in terms of the frame structure and rate [6]. The interweaving is also advantageous since it allows only very less latency for the encapsulated data. The delay for the data passing through is atmost 32 µs, compared to a frame rate of 125 µs. Other complementary protocols usually buffer the data during transits for at least one frame or packet before sending it on unlike SDH. The data and frame rates are different due to which extra padding is provided for the data so that it can glide within the frame. Although the complexity increases with the allowance of this extra padding, the performance is also enhanced. While SDH is still used for protection applications, the network providers discontinue native SDH services successively and change over to the PSN technology.
2.3 Motivation for IP-based Ethernet Networks
IP-based Ethernet networks offer more advantages than CSNs and also remove the existing problems caused by the outdated SDH networks. In the case of PSNs, the available bandwidth can be utilized more efficiently thus requiring fewer network devices and connections for the same network load. This further leads to lesser floor space utilization for the network deploy- ment as well as a reduced usage of electricity. Ciena Corporation compares the legacy SDH equipment with the PSN technology in a white paper [7]. The comparison and case study done in the paper reveals the following figures - 95% floor saving and 90% energy saving. Hence, it can be concluded that it is advantageous to communicate natively over PSN technology and discontinue SDH for LDP application.
From the Line Protection point of view, different approaches have already been tested in the past which use PSN technologies. Some laboratory based tests have been performed in the past using Wi-Fi and WiMAX technologies [8], [9]. When using Wi-Fi technologies, usage of a centrally controlled radio communication can be a disastrous disadvantage since it is po- tentially liable to a single point of failure. Also the range of distance that a Wi-Fi signal can
span plays and important role. The 87L protection scheme done, based on Ethernet communi- cation using echo mode for round trip measurement in order to establish time synchronization was published by Fukushima et al. in [10]. The stated method in the paper uses a PSN network and this academic research work shows that using PSNs for LDP is also a plausible solution that can be considered.
Quality of Service for prioritizing traffic and routing of packets are some of the major perks of building networks on top of the IP Layer 3 and upwards in the ISO/OSI model. One can also route traffic through different ports in case multiple applications are running at the same time on the network. Also, the same Ethernet network can be used for connecting the LDP protec- tion relay device with the Personal Computer in use. This option of using an IP based network has been tested upon in [2] and it is the approach most closely related to the methodology of this thesis work.
Unlike CSNs, using the PSNs for LDP will require the usage of a time synchronization mecha- nism that can get the clocks in the network synchronized. As mentioned in Section 2.1.2, a very precise time synchronization is needed for the transfer of data samples. Using devices having GPS receiver and then synchronizing them through the GPS signal is an alternative solution.
However the aim here is to develop an independent solution and also, it may not be possible to have GPS antenna and receivers in several situations since it becomes an added requirement on the hardware. Besides here, the focus is on synchronization via the communication path.
2.4 Time Synchronization
In modern power systems, precise time synchronization has become an essential phenomenon.
Although there are several methods available today to achieve synchronization between the intelligent electronic devices (IEDs), recently there has been a great deal of interest in trying to provide the time synchronization using the same infrastructure channel through which the data messages are communicated.
Most of the power system applications involving IEDs usually require that the IEDs have an accurate time source, using which they operate in sync with each other. Examples of such applications requiring micro and nano second accuracy include travelling wave fault location, synchrophasors and sampled values. In order to distribute such accurate time stamp values, the traditional methods that have been popularly used in the past include IRIG-B (which needs an extra cable different from the cable used for the data communication) and NTP (which does not provide sub micro-second range accuracy). In modern times, Ethernet based communication technologies are becoming more prevalent in substation automation industrial environments. When using PSN networks, the challenge lies in retaining “high reliability, deter- minism, and availability” [11]. A protocol used for time synchronization over the Ethernet that
satisfies these qualities is the PTP (Precision Time Protocol or IEEE 1588). This is becoming a dominant and standard technology for synchronization over the same data communication path. It provides an accuracy of 1 µs or lesser depending on the implementation.
The upcoming sections discuss the existing time synchronization solutions like NTP, IRIG- B and Synchronous Ethernet followed by a detailed protocol analysis of the Precision Time protocol. It includes information regarding the protocol’s working, designing and special ter- minologies used by it in the network. PTP provides accuracy within the sub-microsecond range. It has also been defined as several different profiles in the standard out of which the Power profile, Power Utility profile and Telecom profile for power system applications are being mentioned here.
2.4.1 Inter-Range Instrumentation Group - B
IRIG-B technology, developed by the Inter-Range Instrumentation Group (IRIG), is one of the most widely used time codes in the electrical industry. It is often used to provide time synchro- nization solutions in the power systems domain for phasor measurement units, protection relays or similar utilities. Each IRIG-B signal is a frame, 100 bits in size having precise information regarding the date, time and sometimes the control functionalities [12].
IRIG-B can provide accuracies in the order of 1 µs or lesser depending on the implemen- tation. However there are a lot of details like method of connecting the network devices to the time source, correct usage the transceivers, receivers and repeaters etc. to which care- ful attention has to be given during the implementation phase. Unfortunately, due to the requirements of these specifics it makes the overall design, implementation, installation and maintenance of these IRIG-B networks costly [12]. Another disadvantage is that the IRIG-B signals are not compatible with the existing data networks. Apart from these issues, due to a technological migration happening now, towards PSN based technologies in the industry, IRIG-B is also slowly being substituted by Ethernet based solutions for time sync that use the same communication channel as the data.
2.4.2 Network Time Protocol
The NTP protocol was created around the 1980s and since then has been a widely accepted protocol used for offering time synchronization solutions in several domains which use Ethernet based networks [12]. This protocol, instead of just copying and pasting the timestamp received into the device’s internal clock, makes use of an algorithm that controls the time frequency in order to keep the timestamping error to a minimum and hence ensures there is no disruption caused. However this leads to a slight delay in the conjunction of time. NTP transmits time messages through the Ethernet medium along with the other data and by taking the time mea- sured for sending and receiving these messages in the network into account, it balances the
possible network latency. Apart from this NTP also calculates an approximate clock deviation and provides a steady time sync even if the signal to the time source is lost.
SNTP is another subset protocol to the NTP, which has lesser algorithm constraints and does not provide all the features like NTP does. However the IEDs using Ethernet commu- nication for synchronization usually use SNTP instead of NTP due to its simplicity and less complexity. In fact, the IEC 61850 standard itself recommended this protocol for the IEDs at first. Unfortunately, the SNTP (neither the NTP) does not provide the required time synchro- nization accuracy wanted by the LDP application and hence does not satisfy the protection interface requirements. It provides an accuracy at a range of 1 ms or lesser. Under higher traffic loads, in practice it attains only an accuracy between 2 ms and 10 ms.
2.4.3 SyncE
Synchronous Ethernet is another technology that is gaining popularity amongst the time syn- chronization solutions. It usually operates along the physical layer of the OSI layer structure in order to distribute the time sync messages from the clocks to the nodes in the network [13].
Similar to SDH/SONET technologies, SyncE also uses a local clock for every network device to learn the clocking rate from each interface in the network device. Usually the system clock is obtained from one of these interfaces in the network device depending on the accuracy. The architectural design of SDH and SyncE are quite similar. Its capability is said to be similar to that of SDH since it does not get affected by other higher traffic load in the network. Also, SyncE is mainly bothered only about the time signal’s precision between the devices.
The synchronization architecture for SyncE is such that any network device should have two local clocks at least and the network interfaces on the devices should be able to produce their own time synchronization signal if the reference clock’s signal fails. This state of the network device is called “holdover” [13]. In order to ensure that this signal generated does not fade away in the communication medium, the signal should be subjected to a filtering at regular intervals and revived.
2.5 Precision Time Protocol
Precision Time Protocol (PTPv2) is defined in the IEEE 1588-2008 standard [14] as a “Pre- cise Clock Synchronization protocol for Networked Measurements and Control Systems”. The PTPv1 was published in 2002 [15] and then its second version PTPv2 was released in 2008.
Both these versions are not compatible and thus cannot be used interchangeably in the net- work. Since PTPv2 has more enhanced features and is more popularly used, that is the version referred to and used in this thesis work.
The aim behind the origin of PTP is to synchronize the different types of clocks varying in precision and stability, in the PSNs. PTP does not need much bandwidth and neither does it cause much of an overhead in the communication aspect. Hence it can be a perfect choice for a time synchronization solution in the distributed network systems. Based on the type of hardware clocking in the network devices and the kind of PTP stack implementation in them, this protocol can provide an accuracy on a sub-microsecond level.
2.5.1 Protocol Details
The PTP standard includes details regarding the principles and operation sequences that hap- pen to make all the clocks in the network synchronize with each other in a real-time run. All the clocks in the network work on the basis of a “master-slave synchronization hierarchy”.
The main clock in the network that acts as the time source for the whole system is called a grandmaster. By sending timing messages to other devices from this clock, the whole network synchronizes to the time advertised by this grandmaster. This protocol works within a subnet scope which is termed as a “PTP domain”.
2.5.1.1 PTP devices
A network running PTP can have both PTP compatible devices called as PTP devices and PTP incompatible devices which are known as non-PTP devices. The non-PTP devices are the ordinary switches and routers and other network gear. Some of the PTP devices include Grandmaster Clocks, Ordinary Clocks, Boundary Clocks and Transparent Clocks. These are described in more detail below [16]:
1. PTP Grandmaster Clock: The role of the grandmaster clock within a PTP network is to act as the time reference or time source to which the rest of the clocks in the network would synchronize too. Hence this clock needs to have a very accurate time feed, which is usually a GPS signal feed or an internal atomic clock. So basically this is an external source of time reference.
2. Ordinary Clock (OC) : A PTP clocking device having a single port is called an ordinary clock. Such a clock can act as either a master i.e become a time source or as a slave i.e synchronize to a master clock. The BMC algorithm (explained in Section 2.5.2.1) is what decides whether the OC becomes a master or a slave for configuration. These are the most common PTP nodes in the network since they act as end devices and also connect to various other external devices.
3. Boundary Clock (BC) : A PTP clocking device that has multiple ports with each port connecting the clock to a different communication channel is called a boundary clock. It can be thought of as the equivalent of a standard router or switch in the PTP network.
A BC passes all ordinary traffic along with the PTP traffic. Such a clock can act as
either a master i.e become a time source or as a slave i.e synchronize to a master clock.
A slave BC synchronises itself to a master and also transmits the time signal to the other master ports in the device. It also uses the BMC algorithm to decide which port becomes master and which one becomes the slave. The master ports in BC synchronize the clock ports in a downstream manner while the slave ports in BC synchronize the clock ports in an upstream manner.
4. Transparent Clock (TC) : The most important manner in which PTP counteracts the network delay and compensates for it, is by using the transparent clock. Its role is to keep updating the time interval field of the PTP event messages. However this update then recompenses for the network delay thus providing an accuracy of 1 picosecond. A TC conveys the transit time of the PTP event message to the receiver device which helps in calculating the propagation delay. There are two types of TCs :
(a) Peer-to-peer transparent clock (P2P TC) : A P2P TC, apart from calculating the propagation delay, also measures and corrects the propagation information as well as the port link delay of the receiver port. With the Peer delay measurement mechanism described in Section 2.5.1.4, the delay between slave TCs, slave-master TCs are calculated.
(b) End-to-end transparent clock (E2E TC) : An E2E TC only calculates the transit time for the PTP event message and does not provide any corrections for the propagation delay like P2P TC. As opposed to P2P TC, it uses the Delay Request-Response mechanism which is described in Section 2.5.1.4.
Figure 2.2: PTP Clocks [16]
2.5.1.2 PTP Messages
This section describes the different PTP messages that are exchanged between the PTP devices for achieving time synchronization [16]. The are mainly two types of PTP Messages : (1) Event Messages which are time critical and they include the first four rows in Table 5.10; (2) General Messages which are not time dependent and they comprise the rest of the rows in Table 2.1.
Hex message type
00 Sync
01 Delay Req
02 Pdelay Req
03 Pdelay Resp
08 Follow Up
09 Delay Resp
0a Pdelay Resp Follow Up
0b Announce
0c Signaling
0d Management
Table 2.1: PTP Message Types
2.5.1.3 PTP Communication Path
In simple terms, a network cable that has a master element at one end and a slave element at the other is called a PTP communication path [16]. To this network cable if a TC is added in between, then the connection from the master element could be branched out to several slave elements. However an important point to remember here is that a PTP communication path may have a TC in between but not a BC. This is because while a TC splits the path, a BC ends it. For instance, if a GM talks to a BC and this BC talks to an OC slave clock, there are two cables used here and the GM is not directly talking with the OC slave clock. In fact all the PTP messages received by the OC slave will have the BC in its address.
2.5.1.4 PTP Delay Mechanisms
In order to perform clock synchronization in a network, it is essential to know the delay cause in communicating the time messages between them. Such a delay in communication is other- wise called path delay or latency. In PTP terminology, this path/network delay measurement mechanism is what is known as the PTP delay mechanism. The delay that is of importance here is the port-to-port delay on a path instead of the end to end delay between two distant devices [17]. This Delay Mechanism concept is what is behind the reason for the accuracy of the PTP protocol functionality. In other words the Hardware Timestamping mechanism is
what makes the IEEE1588 protocol so trustworthy [18]. There are mainly two types of delay mechanisms :
1. End-to-End Delay Measurement Mechanism : The standard also calls this the delay request-response mechanism. This delay measurement mechanism uses message types Sync, Follow-up (optional), Delay Req, and Delay Resp as shown in Figure 2.3. An exchange of messages between the master and slave is portrayed. The time at which a message is sent and received on both ends is recorded and stored as four timestamps - t1, t2, t3 and t4. The Follow Up and Delay Response messages take the timestamps to the slave clock from the master. Using this timestamp data, the slave clock adjusts its time according to the time of the master which it calculates by using the following formula that gives the time deviation between itself and the master :
offset = (t2 + t3–t1–t4)/2
Figure 2.3: PTP End-to-End Delay Mechanism [19]
Unfortunately, there is one shortcoming in this equation. It presumes the fact that the time taken for the messages transmission from master to slave is the same as the time taken for the messages transmission from slave to master, i.e it presumes a symmetrical delay in both the directions. However these are not symmetrical. Any difference in the delay between both the paths - which could happen due to the queues in the routers, switches or even in the network stacks at the end devices - results in an error in deter- mining the time difference between the master and the slave. This is taken care of by the hardware timestamping. How this works is shown in Figure 2.4.
Figure 2.4: Hardware Timestamping during the transport of a Sync message [19]
The Media Independent Interface (MII) between the data link layer and the physical layer in the network device helps in the hardware timestamping mechanism of PTP. Whenever a message leaves or arrives at a network port, special “PTP hardware” in the device creates a timestamp from the internal clock in the MII. This wipes out any uncertainty regarding the response time of the OS. Besides this, the switches supporting PTP, better known as TCs, also timestamp the PTP messages. One such TC is shown in Figure 2.4, which rectifies the time spent by updating the field in the PTP messages. The BCs use the PTP messages to adjust their own internal clock and then send it to the other slave which need to be synchronized. This delay mechanism focuses the PTP load on the master since it does most of the work. Due to this it is not so scalable with an increase in the number of network devices.
2. Peer-to-Peer Delay Measurement Mechanism : This mechanism calculates the delay between two connected network ports, like for instance the ports of two OCs that are connected, with no middle router or switch. The P2P delay mechanism scales with the increase in the size of the network since the network load is not focused on the master port alone like is the case with the E2E delay mechanism.
Like in E2E mechanism, in the P2P mechanism also the master and slave exchange Sync and Follow Up messages and the slave calculates its deviation from the master according to the following formula.
slave time = master time + network delay.
Unlike in E2E mechanism where the slave sends delay measurement messages to the master, in the P2P mechanism all the devices in the network send peer-delay measure- ment messages to each other. Hence there is no requirement for the four timestamp calculation that was seen in E2E mechanism. Using the peer delay messages, all the de- vices can calculate their offset with respect to the neighbouring devices in the network.
Figure 2.5 shows how this works.
Figure 2.5: Working of peer-delay messages in Peer-to-Peer Delay Mechanism [20]
In all the ports on every device, a periodic prompt for sending peer delay messages happens. Next, when the Sync messages go into the device, the peer delay factor is discarded by resetting the fields in either the Sync or the Follow Up message. However in case the device is a switch, the peer-delay is not present in the exit port since the next device that is connected to the switch will do this. This way the resetting being done twice can be avoided. The order of peer-delay messages is shown in Figure 2.6.
Figure 2.6: PTP Peer-to-Peer Delay Mechanism [20]
In this scenario, Clock A is trying to find its offset from Clock B. A sends a Pdelay Req message to B at time t1 and this message reaches B at time t2. A stores t1 and B stores t2 which is sends at time t3 through the PDelay Resp message to A that reaches at t4 which now receives t2. Following this message a Pdelay Resp Follow Up message is also sent by B to A carrying information about the timestamp t3. So finally A ends up having t1, t2, t3, t4 timestamps and it calculates its deviation from B as described in Section 2.5.1.4, point 1 under end-to-end delay mechanism. Next B can do the same with A to calculate its offset from A and in this manner each device can calculate its offset from the neighbouring devices.
Another presumption made is that the communication delay for these messages to reach from A to B and vice versa are the same which is not the case in reality. However this is different from the E2E case since this presumption is done only for a point to point direct cable. So since there will be no queues, this presumption will hold. Here, as with the end-to-end mechanism, the assumption is made that the time it takes for the peer-delay messages to get from one clock to the other is the same in each direction. If supposing the device at the end of this direct cable is a switch, then this presumption can fail. However that is the reason why P2P mechanism only works well with TCs or BCs; since TCs and BCs can take care of the delay. OCs cannot be used for P2P delay mechanism since they cannot handle the peer-delay messages.
An E2E TC supports and takes part only in the end-to-end delay mechanism while a P2P TC supports and takes part only in the peer-to-peer delay mechanism. A BC on the other hand can have several master ports and slave ports. Hence both end-to-end delay mechanism as well a peer-to-peer delay mechanism can be configured on those ports. An OC that has only a single port can either have peer-to-peer delay mechanism or end-to-end delay mechanism. Any one PTP communication path cannot have a combination of both mechanisms. It can either run peer-to-peer delay or run end-to-end delay at any given time. A peer-to-peer communication path can have exactly two ports while an end-to-end communication path can have atleast two ports.
End-to-End versus Peer-to-Peer : E2E delay mechanism is more user friendly since OCs, routers, switches etc.. can also manage this mechanism. However only TCs and BCs usually work well with the P2P delay mechanism. Inspite of that P2P delay mechanism can be more useful in some scenarios where it is more advantageous :
1. When the networks are dynamically changing, even if a network path has changed, this periodic outburst of peer-delay messages from the devices, ensures that all the devices are aware of their own offsets from the neighbouring devices. Also, these peer-delay messages are transmitted in such a way that there are no loops (by using the Spanning Tree protocol).
2. Unlike in E2E mechanism where the Sync message and the Delay Request message can end up being on different paths, in P2P it is not a problem as there is no Delay Request message.
3. As in P2P, one has to send only the Sync and Follow Up messages the overhead is less and there is more simplicity during congestion times which makes it more scalable while for E2E mechanism, it has to deal with the Delay Request messages responses.
2.5.2 Working Principle
The IEEE 1588v2 standard describes a “hierarchical master-slave architecture” for PTP time synchronization. Under this architecture, the devices in the network act as either a master (thus relaying time to other devices) or a slave (thus synchronizing to the time relayed by the master devices). These master and slave devices then exchange a set of messages as explained under the delay mechanisms to synchronize with each other. A sample PTP Network in the power domain containing grandmaster clock, BC switches, IEDs etc. is shown in Figure 2.7 which also shows the working of PTP time synchronization. Here if the Master 1 looses its GPS signal, then all the devices start synchronizing to Master 2 which is the second best clock to take upon the role of a grandmaster based on its time quality [16].
Figure 2.7: PTP Working Principle [16]
The working principle mainly revolves around two steps for synchronizing the devices using IEEE 1588 protocol : (1) Check and decide which device will become the grandmaster clock and which will become master/slave in the network, and (2) perform delay measurement calculations to find the deviations of the internal clocks in devices with respect to the other network devices and correct this time difference in order to synchronize with each other. In
order to execute the first step, PTP uses the BMC algorithm which is described in the section section.
2.5.2.1 Best Master Clock Algorithm
The Best Master Clock algorithm is an important part of PTP. This algorithm determins which node in the network will become the GM. The algorithm runs at regular intervals in the network and updates itself in case there is another better GM available [21].
In order to determine which node has a better time quality, the following parameters are chosen by BMC and evaluated for each node :
1. Priority One Field : It is a user definable value with possible ranging from 0 to 255.
The default value is 128 for a device that needs to compete for the master role. For the slaves its set to 255. So basically lower the number, higher the priority of it becoming the master.
2. Clock Class : The class of a clock is determined by its current state. A clock having a stable source of time which is really accurate is said to have a higher class than a local clock running wildly. For instance a clock connected to UTC, has more class than a clock connected to a local reference. Hence higher a clock class, more probability that the clock becomes the master.
3. Clock Accuracy : This field specifies the accuracy level of a clock and more the accuracy more the chances are of the clock becoming the master. Usual range for PTP is between 20ns to 100 ns.
4. Clock Variance : This field indicates how stable the clock is or how much it is affected by the jitter and latency in the network. Lesser the variance the better it is.
5. Priority 2 Field : This is similar to the Priority 1 Field that can also be set by the user.
It is just a back up option for providing a second clock as a choice in case for some reason the first clock is not able to become the master.
6. Source Port ID : It is the MAC address of the source port in the clocking device. It is also acting as a tie breaker here in case all the above mentioned values end up being equal in two or more nodes fighting to become the master.
7. Steps Removed : If two nodes are having the same GM in the network, then whichever node is connected to the GM through least hops has better chances of becoming a master (in case the GM looses its GPS signal).
According to the BMC algorithm, when the existing GM looses its GPS signal, another clock with the next best time quality takes upon the role of the GM according to the above rules.
The set of possible states that any clock can go into are shown in Figure 2.8.
When the clock is switched on, or restarted, it goes into the listening state where it lis- tens to announce messages from other clocks around it. An announce message contains the above mentioned fields and properties of the clock sending it. So when a clock receives an announce message from another clock, it can compares the remote clock’s properties with itself and determine if the announce messages was a better one or a worse one. If that announce message was a better one, which means the remote clock has a better time quality, this clock goes into the slave state making the remote clock as its master. If the announce message was worse one which means the remote clock has worse time quality than this clock, then it ignores this messages and waits. The wait time for getting a better announce message than itself is called the announce time out interval. Within this interval the clock should get a better quality master, else it takes over the role of a GM in the network. This algorithm runs continuously in the network and also it is important to note that the announce time out interval should be greater than the announce interval for better results.
Figure 2.8: PTP BMC Working
2.5.2.2 Synchronization
Like mentioned under the working principle, there are two steps for PTP synchronization out of which the first step is to run the BMC and determine the master-slave architecture for the network. The second stage is where each device calculates its offset from another device using the delay mechanisms in order to achieve time synchronization in the overall network. So this second stage also known as the synchronization process is divided into two phases [22]. First phase is the “offset correction” phase that has been described in Section 2.5.1.4 by calculating
the delay measurements. Second phase is the actual network synchronization where the clocks using the delay measurements and offset calculation done, update their hardware and software clocks and synchronize themselves to their respective masters. Hence all the devices in the network get synchronized in this manner.
2.5.3 PTP Profiles
A PTP profile as defined by the IEE-1588 standard is “the set of allowed PTP features ap- plicable to a device”. PTP has different profiles like the power profile, power utility profile, telecom profile etc. to satisfy the needs of different kinds of applications. However all these profiles are just variations of the PTP standard or restrictions imposed of several features in PTP in order to enable the usage of PTP in various domains like the power domain. The IEEE standard and the description of PTP given in Section 2.5 all refer to the standard profile also known as the default profile. This profile is the most widely used on in a lot of applications.
Almost all manufactures and vendors manufacture PTP devices based on this default profile, since it also provides an easier standard way of comparison and testing. All other profiles are created by modifying this default profile by various other standards like the IEC for their own purposes.
A PTP profile is created by modifying one or more of the following features from the de- fault PTP profile usually [16].
• BMCA parameters
• PTP delay mechanism configuration parameters
• PTP management features
• default values of all the variable parameters
• PTP device types
• extra PTP features
• communication rules for PTP
Some of such profiles that are relevant in this thesis work are : 2.5.3.1 PC37.238 IEEE-1588 : PTP Power Profile
The IEEE Power Profile [23] defines “specific or allowed values for PTP networks used in power substations”. Some of these values that are tampered with from the default profile include the BMCA parameters, higher layer protocols in the power communication domain for running PTP messages etc. This profile is configured with the aim of providing accurate time synchronization using PTP over country wide substation automation networks over WANs.
The profile also defines the usage of FPGA and PHY interface for the network node operations on a physical layer. In this profile mode, the devices uses the configuration values defined in the IEEE-1588 Power Profile standard [23]. The PTP messages are transported over layer 2 using P2P delay mechanism in the TCs.
2.5.3.2 IEC/IEEE 61850-9-3:2016 : PTP Power Utility Profile
The IEC/IEEE 61850-9-3 Standard [24] defines the PTP Power Utility Profile (PUP) for the power utility automation field. It is derived from the default profile and is a subset of it apart from the redundancy link method that is a new addition in this profile. The reason behind adding this redundancy method is because of the high demands of time sync from the substa- tion automation environment. The IEC 61850 defines “Parallel Redundancy Protocol) (PRP) and High-availability seamless redundancy (HSR)”. PRP is used in cases where the network devices are from a combination of two separate LANs. Since the two nodes could be in differ- ent LANs, the delay principle in the default profile is compromised as the delay calculation in different LANs could change differently. To avoid this, the IEC 62439-3 Annex A changes this 1588 principle only for the PTP messages. The same change is done for the HSR protocol as well. More about this can be read in [25].
Another difference from the default profile is that although the default profile defines a lot of parameters like the P2P, E2E, synchronization modes etc. it does not pay attention to the performance factors or parameters. This PTP PUP profile takes such performance character- istics like time accuracy for each PTP device for instance.
This PTP PUP profile along with the modification regarding PRP and HSR protocols has benefited the substation automation section in the power industry. It has enabled an increase in the performance of the time synchronization solution which is very important in specially the protection function applications.
2.5.3.3 IEEE G.8275.1/G.8275.2 : PTP Telecom Profile
The IEEE G.8275.2 is the PTP Telecom profile [26] which is created when “phase or time- of-day synchronization is required”. The previous telecom profile version is the G.8275.1 in which every node in the system may not participate in the PTP synchronization. The G.8275.2 version uses unicast mode with IPv4/v6 protocols running on layer three in the OSI model.
The network devices running this v2 of the protocol need not be connected with a direct fibre since they use a “partial timing support” for synchronization. Some applications of this pro- file are Wi-Fi systems and LTE technology running networks that need synchronization in the telecommunication sector.
The kind of PTP devices used in this profile include OCs (GM and partial time support-
ing slaves) and BCs (GM BC, slave BC) [26]. Another interesting difference in this profile is the ABMCA (alternate best master clock algorithm) that is used instead of the BMCA algorithm which is the common algorithm used by all the other profiles.
Some other features of this profile include [27]; usage of both hybrid and non-hybrid modes of PTP, usage of redundant time sources for each device although at any given time the device is synchronized only to one of those sources. This profile does not give much of importance to performance metrics.
2.6 PTP versus existing methods
PTP delivers a clock synchronization accuracy of 1 µs which is somewhere in the middle of the accuracy provided by NTP protocol and GPS based synchronization systems [11]. Instead of using GPS in order to synchronize the whole system, it is easier to use PTP because only one GPS signal is needed in the PTP system for the GM and it is easy to maintain than ensuring every device in the network having a GPS antenna. It also makes it cost friendly. Also this way the whole system does not become a predator to a bad GPS signal; only the GM would loose its signal which would be easy to figure out. Also PTP system can stay independent of GPS. More about this will be covered in the upcoming Chapters. An approximate comparison of the sync accuracies are given in Table 2.2.
Technology Accuracy
atomic clock 0.1 - 1 ns per day
GPS (expensive) 10 ns
PTP (inexpensive) 100 ns - 1 µs
NTP 0.2 ms - 10 ms
IRIG-B 1 µs
Table 2.2: Comparison of Time Sync Mechanisms
While PTP emphasizes on day time based synchronization like the UTC, SyncE method em- phasizes on only frequency based synchronization. However due to the similarity with SDH, it is still a huge competition for PTP in some domains like telecommunication. Just like SDH uses TDM at the physical layer, Sync-E also works on a similar principle as was explained in Section 2.4.3. According to [17], PTP supports upto 1GigE while SyncE supports upto 10GigE. However Sync-E is not so compatible as it requires hardware changes in the network
devices. PTP is cheaper in this aspect since it requires only minor hardware changes. It is due to this main reason that PTP is more cost efficient and popular.
Chapter 3
Methodology
3.1 Proposed Solution
From the literature study phase of this thesis work which is summarized in Chapter 2, it is evident that using PTP with PSNs can be a great alternative method for SDH/SONET technologies in order to run LDP application. So the proposed solution here it to use Packet switched networks for communication of LDP data and use PTP for time synchronization just like its outlined in [2]. A detailed description of the technical aspects of PSN and PTP are discussed in Sections 2.3 and 2.5.
3.2 Satisfying the Requirements of the Protection In- terface
As discussed in Section 2.1.2, the latency and jitter should be minimum while higher bandwidth and accurate synchronization are absolutely necessary for the LDP communication at the protection interface. In fact according to the requirements specified in the IEC 61850 standards, a time synchronization solution of the order of 10 µs or less is recommended. Hence this solution of using PTP for time synchronization, is beneficial as it fulfils the requirements of the LDP protection interface. Also the networks no longer need to be symmetrical like for SDH/SONET networks. Apart from the time synchronization requirements, PSNs allow routing and flexibility through the QoS features that can ensure a reduced latency and jitter depending on the implementation conditions and network settings.
3.3 Approach
3.3.1 Communication : Packet Switched Networks
Since running LDP communication over PSN technologies would mean working at the different ISO/OSI model layers of the network, in this thesis work, it has been decided to run the application over the UDP protocol (at the Transport Layer), IP (at the Network Layer) and Ethernet 802.3 (at the Link Layer). This option of using an IP based network has been tested upon in [2] and it is this approach that is most closely related to the methodology of this thesis work. The academic evaluation done in that paper has been looked upon from an industrial perspective in this thesis project.
3.3.2 Synchronization : Precision Time Protocol
Synchronization over Ethernet using the same medium as that used by the data messages is the main criterion that was considered for choosing PTP as the time sync solution. The approach on using PTP for implementation was again by installing the PTP stack on all the network devices.
A detailed description of the implementation of this solution using the above mentioned ap- proach is done in the next Chapter.
Chapter 4
Implementation
The implementation part of this project consists of creating pilot network set-ups in the com- munication laboratory of ABB, using various resources. These set-ups are mainly used, to implement line differential protection over UDP(L4)/IP(L3)/Ethernet(L2) based networks and achieve time synchronization using PTP protocol. This Chapter explains in detail all the re- sources that are used and how the different network topologies are implemented. It also covers details regarding the measurement conditions like the configuration settings in the various network devices.
4.1 Resources Used
A combination of hardware and software tools have been used for implementation and evalua- tion purposes. Before going into the implementation set-ups, a briefing of the resources used is done below.
4.1.1 Hardware Network Components
The various hardware equipments that are used to create the different network topologies and to perform measurements include :
– Intelligent Electronic Device (IED) : An IED is generally defined as a microprocessor- based controlling device that can control power system equipments such as tap changers, switches, drives, valves, transformers, circuit breakers or capacitor banks [28]. Some of the IEDs are also the protective relaying types using the microprocessor to perform several protective and control functions. A typical IED usually has around 5-12 protection functions, 5-8 control functions controlling separate devices, an auto-reclose function, self monitoring function, communication functions etc. That is why they are named as Intelligent Electronic Devices. With respect to the communication functions that they can perform, some of the latest IEDs are designed to support the IEC61850 standard
for substation automation thus providing interoperability and advanced communication capabilities.
One such IED called the RED670 [29], which belongs to the Relion [30] protection and control product family is what is used for line differential protection application, as shown in Figure 4.1. The RED670 IED is designed for protection, monitoring and control of overhead lines and cables in all types of networks. In addition, this IED is capable of handling transformer feeders, generator and transformer blocks. It provides an extensive functionality with configuration opportunities and expandable hardware to meet our specific requirements. For this thesis work, three RED670 pilot version 2.2 (laboratory models) IEDs that support PTP and have Ethernet optical ports are used.
Figure 4.1: RED 670 IEDs
Within the IEDs, there are several slots into which different modules can be fit as per our requirements. Each module (mainly containing FPGAs) is used to bring about a specific functionality in the IEDs. The modules that are explicitly used in this project are the following. They can also be seen in Figure 4.2.
– Line Differential Communication Module (LDCM) : This module is used for the communication of the LDP’s data packets in the form of C37.94 frames [31].
LDCM uses the C37.94 as the protocol for optical fiber communication. It is these C37.94 frame formats carrying the LDP communication data, that are then sent over UDP packets.
– Inter-Range Instrumentation Group (IRIG-B) : This module is mainly used for achieving time synchronization. The message in the IRIG-B will give an indication when a new second is present, with the pattern “on-time”. This pattern then
triggers a storage of time within the module and generates a signal for a capture register. An optical PPS input used by itself or in combination with the IRIG-B input can synchronize other equipments in a station if such a device is present.
This method is what is used at first before using PTP; during the initial phase of the project.
– GPS Time Synchronization Module (GTM) : This module as the name sug- gests is also used for establishing time synchronization between the IEDs. The main notable hardware I/O of this module that is used is the PPS input and output like in IRIG-B module. The probes are connected between this module and the oscil- loscope, and using the PPS out signals that are generated, several measurements are done during the evaluation phase.
Figure 4.2: LDCM, IRIG-B and GTM Modules (left to right order)
– L2 Switch : Layer 2 switches are generally used for storing and forwarding L2 packets in the TCP/IP suite. The L2 AFS switches from ABB are optimized for the data communication of electrical utilities. With their robust hardware and powerful operating system they are able to withstand extremely harsh environmental conditions as mentioned in [32]. The integration of the redundancy protocols PRP and HSR allows uninterrupted data communication and therefore guarantees the reliability and interoperability of the system. Precision time synchronization in accordance with IEEE 1588v2, synchronizes sensors, drives and measuring equipment. Gigabit Ethernet enables a fast connection to
the backbone, while connections to terminal equipment are made via 100 BASE-TX – either alone or in combination with 100 BASE-FX.
For this project, two PTP compatible L2 AFS665 LAN switches with 11 ports each are used. These switches are mainly used as BCs or TCs thus allowing them to be syn- chronized to a precision of approximately 100ns. However these switches only support Multicast PTP thus making the load on the network due to PTP sync messages negli- gible. An AFS660 switch is shown in Figure 4.3, which after slight modification makes the AFS665 switch.
Figure 4.3: AFS 660 switch
– GPS Clock and Receiver : A Meinberg LANTIME M3000 GPS clock [33] is used as the PTP grandmaster for all the implementation set-ups. For some of the set-ups at the evaluation stage, the GPS bulb shaped antenna is also used. The front view of the GPS Clock and Antenna used along with the AFS677 switch are shown in Figures 4.4 and 4.5. The Clock is connected to an AFS677 switch (that acts as a BC) [34] and then the GPS signal is relied to all the IEDs in the lab as shown in Figure 4.6.
Figure 4.4: GPS clock and AFS677 switch Figure 4.5: GPS Antenna
– Personal Computer : Two desktop PCs (with CPUs having 8 cores) are used to create and configure the network topologies. One of these PCs is running Windows OS while the other is running Linux Ubuntu OS. The Software tools described in the following section are also installed and run on these machines.
Figure 4.6: GPS clock signal relied to all network nodes in the lab
– Oscilloscope : In order to measure the performance metrics like the time sync accuracy, an Agilent Technologies’ DSO1024A oscilloscope [35] is also used as shown in Figure 4.7.
A PPS signal is obtained from the GTM boards of the IEDs to perform the measurements.
Figure 4.7: Oscilloscope
– Connection Media :There are mainly three types of connection media that are used and they are shown in Figure 4.8:
– CAT5e UTP Cable with an RJ 45 Connector : The Ethernet over twisted pair cables are used for connecting the L2 switches with each other and to connect the switches to the PCs as well. The ones used for this project are 100BASE-TX, supporting speeds of 100Mbit/s in full-duplex fashion.
– MTRJ, Duplex Multimode Fiber Optic Cable and Patch Cord : These fiber optic cables are used to connect two IEDs together and also to connect the IEDs to the switches.
– SFP Transceiver : Small Form-Factor Pluggable Transceiver are basically used to standardise the transformation from the optical domain of the communication medium to the electrical domain of the integrated circuit. This transceiver is com- pact and hot-pluggable. It is widely used in the field of Data Communication and Networking. It mainly acts as an interface between a networking device and its interconnecting cable. In the course of this project it is was mainly plugged into the ports of the IEDs, thus acting as an interface to the fiber optic cable.
(a) Ethernet RJ 45 Cable (b) SFP Transceiver
(c) MTRJ, Fiber Optic Cable (d) MTRJ, Fiber Optic Patch Cord
Figure 4.8: Communication Media
– Single Phase Relay Tester : The SVERKER650 [36] relay tester as shown in Figure 4.9, is used to supply a single phase current in order to test if LDP application works well or not during the evaluation phase. The tester is directly connected to the IEDs and the differential current and bias current are measured using remote IED access clients.
