Advanced Ethernet Clock Synchronization based on Round Trip Time Protocol

(1)

Advanced Ethernet Clock Synchronization based on Round Trip Time Protocol

GRANVILLE MANVEL GOES

K T H R O Y A L I N S T I T U T E O F T E C H N O L O G Y

E L E C T R I C A L E N G I N E E R I N G A N D C O M P U T E R S C I E N C E DEGREE PROJECT IN EMBEDDED SYSTEMS, SECOND LEVEL STOCKHOLM, SWEDEN 2020

(2)

Advanced Ethernet Clock

Synchronization based on Round Trip Time Protocol

Granville Manvel Goes

2020-04-29

Master’s Thesis

Examiner Johnny Öberg

Academic adviser Kalle Ngo

Industrial adviser Ulf Parkholm

KTH Royal Institute of Technology

School of Electrical Engineering and Computer Science (EECS) Department of Electronics

SE-100 44 Stockholm, Sweden

(3)

Abstract | i

Abstract

In this master thesis project, a new protocol called the Round Trip Time (RTT) protocol is implemented and verified. It helps determine the Ethernet clock frequency offset between two communicating nodes. The detection of this offset between nodes is a way to reduce the clock synchronization error. Ethernet is the basis on which a large amount of communication takes place in the world. Either it is used for exchanging data from one device to another or to connect devices to the internet. Due to the absence of clocks being exchanged between the various Ethernet communicating nodes, clock phase and frequency offsets can be present which leads to clock de-synchronization between the various nodes and results in lower system throughput. In the telecommunication industry, synchronization error between base stations can lead to lower throughput, performance degradation and packet loss.

Also, with the introduction of 5G, stringent requirements will be placed on the clock synchronization errors.

Currently, the Precision Time Protocol (PTP) is used to detect and correct clock synchronization errors. The PTP implementation reduces the clock synchronization error but it is still quite large. Hence, it is necessary to find a protocol which can work together with the PTP protocol to reduce this error. This thesis will introduce a new way to determine the clock frequency offset between nodes through the implementation of the RTT protocol. Through the course of this project, the clock frequency offset was determined by the RTT protocol. By comparing the expected and the theoretical clock offsets, it was concluded that the two values were very similar. The error between the offsets was in the range of 2.349-15.687 parts per billion (ppb) of the link frequency. Thus, the RTT protocol accurately and precisely determined the clock frequency offset between two Ethernet communicating nodes.

This protocol is also extended to determine the clock frequency offset between two nodes transmitting periodic signals. For future works, this protocol can be combined with the PTP protocol and a way to determine the clock phase offset will be investigated.

Keywords

Networks, Ethernet, Clock synchronization, Synchronization error, Precision Time Protocol (PTP), Round Trip Time (RTT) protocol

(4)

(5)

Sammanfattning | iii

Sammanfattning

I detta examensarbete implementerades och verifierades ett nytt protokoll, kallat Round Trip Time (RTT)-protokollet, som hjälper till att bestämma Ethernets klockfrekvensförskjutning mellan två kommunicerande noder. Denna fastställda förskjutning mellan de två noderna är ett sätt att reducera klocksynkroniseringsfelet.

Ethernet är grunden i en stor del av dagens kommunikation i världen. Antingen används det för informationsutbyte mellan två enheter, eller för att ansluta till internet. Då det saknas ett utbyte av referensklocka mellan de olika kommunikationsnoderna på Ethernet, kan det uppstå klockfas- och frekvensförskjutning som leder till att klockan desynkroniseras mellan de olika noderna och därmed ger ett minskat dataflöde. I telekommunikationsindustrin kan ett synkronisationsfel mellan basstationer leda till minskat dataflöde, sämre prestanda och paketförlust. I och med introduktionen av 5G kommer stränga krav att ställas på klocksynkronisationsfelen.

För närvarande används Precision Time Protocol (PTP) för att upptäcka och korrigera klocksynkroniseringsfelen. Implementationen av PTP reducerar klocksynkroniseringsfelet, men det är fortfarande relativt stort. Därav är det nödvändigt att hitta ett protokoll som kan arbeta tillsammans med PTP för att reducera detta fel. Detta arbete kommer att introducera ett nytt sätt att bestämma klockfrekvensförskjutningen genom implementation av RTT-protokollet. I detta arbete bestämdes klockfrekvensförskjutningen av RTT-protokollet. Genom att jämföra det förväntade och faktiska värdet på klockförskjutningen kunde slutsatsen dras att de två värdena var väldigt lika. Felet var i storleksordningen av 2,349-15,687 parts per billion (ppb) i linkfrekvensen. Således bestämmer RTT-protokollet korrekt och exakt klockfrekvensförskjutningen mellan de två kommunikationsnoderna i Ethernet. Protokollet utökas också för att bestämma klockfrekvensförskjutningen mellan två noder som sänder en periodisk signal. För framtida arbete kan detta protokoll kombineras med PTP-protokollet, och det ska även undersökas ett sätt för att bestämma klockfasförskjutningen.

Nyckelord

Nätverk, Ehternet, Klocksynkronisering, Synkroniseringsfel, Precision Time Protocol (PTP), Round Trip Time (RTT) protocol

(6)

(7)

Acknowledgments | v

Acknowledgments

I want to take this opportunity to thank Ulf Parkholm who has been an incredible supervisor. He has guided me throughout the course of the project and patiently answered all my questions. I am glad to have him as my supervisor since he has such a vast experience and I was able to learn quite a lot under his guidance. I extend my sincere gratitude towards Pierre Rohdin G for helping me throughout the onboarding process at Ericsson AB and helping me get accustomed to the processes within the company. I like to thank Ericsson AB for giving me this opportunity to pursue my Master Thesis project at their company.

At the University, I like to thank Johnny Öberg for his encouragement towards my master thesis and for providing intellectual ideas. I appreciate all the help received from my supervisor Kalle Ngo in tracking my thesis status every week and providing constant valuable feedback. I like to especially praise Parmiss Fallah and Ben Olayinka for reviewing my document as my opponents and suggesting

meaningful updates which made a huge difference.

Finally, I thank my mom, dad and sister for their constant support,

encouragement and for believing in me. Also, I thank all my colleagues, professors and friends at Ericsson AB and Kungliga Tekniska Högskolan for helping me morally and academically during each step of my Master Thesis project.

Stockholm, February 2020 Granville Manvel Goes

(8)

(9)

Table of contents | vii

List of Figures

Figure 1-1: Ethernet communication ... 2

Figure 2-1: Clock Skew ... 6

Figure 2-2: Clock Jitter ... 6

Figure 2-3: Ethernet frame and packet format ... 7

Figure 2-4: OSI Model for Clause 108 RS-FEC Ethernet ... 9

Figure 2-5: RS-FEC sublayer ... 10

Figure 2-6: PTP protocol ... 11

Figure 2-7: RTT protocol ... 12

Figure 2-8: Initial analysis to generate and plot RTT values ... 14

Figure 2-9: Compare clock synchronization error ... 16

Figure 2-10: RTT using PS signals ... 17

Figure 2-11: RTT using SoF signals ... 18

Figure 3-1: PS detection and packet size ... 21

Figure 3-2: Periodic PS detection ... 21

Figure 3-3: Modified way of acquiring the RTT values ... 22

Figure 3-4: No clock offset - generate RTT values from PS ... 23

Figure 3-5: Clock frequency offset present - generate RTT values from PS ... 25

Figure 4-1: Algorithm for injecting clock offset ... 31

Figure 4-2: Algorithm for calculating the RTT values ... 32

Figure 4-3: Simulation ... 35

Figure 5-1: Plot for +/- 45 ppm offset RTT values ... 38

Figure 5-4: Test 1: Link 1 +45 ppm offset and link 2 -45 ppm offset ... 44

Figure 5-5: Test 2: Link 1 -45 ppm offset and link 2 +45 ppm offset ... 44

Figure 5-7: Test 4: Link 1 -75 ppm offset and link 2 +75 ppm offset... 45

Figure 5-9: Test 6: Link 1 -90 ppm offset and link 2 +90 ppm offset ... 46

(12)

(13)

List of Tables | xi

List of Tables

Table 4-1: Offset values ... 34 Table 4-2: Reversing offsets in links ... 34 Table 5-1: Comparison between expected and actual clock frequency offset ... 42

(14)

(15)

List of acronyms and abbreviations | xiii

List of acronyms and abbreviations

AM Alignment Marker CSV Comma Separated Value CWM Codeword Marker

DUT Design Under Test

fs femtosecond

Gbps Gigabits per second

GM Grand Master

GMII Gigabit Media Independent Interface GPS Global Positioning System

Hz Hertz

ICT Information and Communication Technology IEEE Institute of Electrical and Electronics Engineers kHz kilohertz

MATLAB matrix laboratory

MHz MegaHertz

ns nanoseconds

OSI Open Systems Interconnection PCPs Periodogram and Correlation Peaks PHY Physical

ppb parts per billion ppm parts per million

ps picosecond

PS Parity Symbol

PTP Precision Time Protocol

RS-FEC Reed-Solomon Forward Error Correction RTL Register Transfer Level

RTT Round Trip Time

RX Receive

SFD Start Frame Delimiter SFD

SoF_RX Start of Frame signal for the Receiver SoF_TX Start of Frame signal for the Transmitter SV System Verilog

TX Transmit

ULS Unwrapped Least Squares

µs microsecond

VIP Verification Intellectual Property WLS Weighted Least Squares

WR White Rabbit

WWW World Wide Web

(16)

(17)

Introduction | 1

1 Introduction

The Master Thesis project mainly focuses on the implementation of a new protocol to determine the clock frequency offset between Ethernet communicating nodes. The determination of this offset can be used to reduce the clock synchronization error.

This error is caused due to the clock frequency and phase offset between the nodes [4]. If this clock offset is present between base stations of telecommunication networks, it will lead to lower throughput. The goal of the project is to determine the clock frequency offset through the implementation of the Round Trip Time (RTT) protocol. It will be carried out by implementing the RTT protocol in the existing design and simulating it in Cadence Xcelium.

1.1 Background

Ethernet is a useful medium for the exchange of data between various devices. It can also be used for connecting devices to the Internet. Nowadays the speed of Ethernet has reached a staggering 25 Gbps, a very high rate for data transmission [1]. In the case of Ethernet communication, only the Transmit (TX) and Receive (RX) data lines are exchanged between the communicating nodes as seen in Figure 1-1. The TX of Ethernet node 1 connected to the RX of Ethernet node 2 forms Link 1 and the TX of Ethernet Node 2 connected to the RX of Ethernet Node 1 forms Link 2. Each of these nodes have their clocks and these clocks are not exchanged between the nodes. Some of the various ways to distribute and synchronize Ethernet clocks are listed in [5].

This may lead to clocks drifting apart and eventually leading to de-synchronization between the nodes which may lead to issues when tighter clock synchronization is necessary. Various other protocols are used for clock synchronization. One such protocol is the Precision Time Protocol (PTP). It helps to determine the clock offset between Ethernet communicating nodes but this protocol has a finite amount of clock synchronization error [6]. Thus, it is of utmost importance to find an alternative way to determine the clock offset which can be combined with the PTP to achieve better performance.

(18)

2 | Introduction

1.2 Problem

De-synchronization between Ethernet communicating nodes leads to a clock synchronization error. A small error in a base station leads to lower throughput, performance degradation and packet loss [10]. These can cause problems in precisely determining a user within the coverage area of a base station antenna. This may also lead to noise reception at the end-user which will cause a disturbance.

The Precision Time Protocol (PTP) protocol determines the clock offset by collecting timestamps of packets when they are transmitted and received. In the case of PTP, there exists a finite amount of clock synchronization error which cannot be corrected by the PTP. The question that will be answered through the implementation of this project:

Does the protocol implemented accurately and precisely determines the clock frequency offset between two Ethernet communicating nodes so that this value could be used to synchronize the nodes?

1.3 Purpose

By developing a new protocol which determines the clock frequency offset, the offset between the nodes can be corrected and thus the clock synchronization error will be reduced. This will ensure that all the nodes will be synchronized, and the base stations will be able to achieve a larger throughput. The implementation of this Master Thesis project will benefit Ericsson AB who will be able to provide very high- quality telecommunication services to the end-user which will ensure customer satisfaction.

Since this Master Thesis project will only be involved with the simulation of designs and collecting simulation data, it will follow all the ethical requirements.

Figure 1-1: Ethernet communication

(19)

| 3

Also, through this solution, it may be possible to increase the throughput and efficiency of systems and hence be able to save a large amount of energy which will eventually contribute to a sustainable environment. This project may ensure better connectivity between people through better communication services.

1.4 Goals

This project will mainly focus on the implementation and testing of the RTT protocol that will help to determine the clock frequency offset. This value can be used to reduce the clock synchronization error between Ethernet communicating nodes by correcting the offset between the communicating nodes. The goal of this thesis is as follow:

Implementing protocol using existing design in Cadence Xcelium

• Use existing Ethernet Design Under Test (DUT) connected to a Verification Intellectual Property (VIP)

• Implement the RTT protocol using existing Ethernet designs

• Inject clock offsets in the Ethernet links

• Simulate the design in Cadence Xcelium

• Keep track of packets being sent and received at the Ethernet node

• Calculate the RTT value

• Repeat the process for various packets

• Plot the RTT measurements

• Use a mathematical model for determining the clock frequency offset

• Compare the computed offset to the one injected into the system

1.5 Research Methodology

The Master Thesis will be carried out by using existing designs. A simulation approach is chosen over hardware. It was carried out by implementing the RTT protocol and simulating the entire design. In the simulation approach, the DUT is connected to a VIP and will be simulated in Cadence Xcelium. It is assumed that the DUT and VIP work reliably and perfectly which implies that it does not require any sort of verification.

The RTT protocol is implemented in the existing design. A known clock frequency offset is injected between the Ethernet links. The design is simulated and the RTT values are determined. These values are exported and plotted on a graph. The clock

(20)

4 |

frequency offset is determined from this graph by applying mathematical models.

Finally, the actual and expected clock offsets are compared to determine if the protocol works perfectly.

Communication between only two Ethernet nodes will be considered. One node will act as the master and the other will be the slave. The clock of the master node will be injected with a positive offset, whereas the clock of the slave node will be injected with an equal and opposite offset as compared to that injected in the master.

1.6 Delimitations

The focus will be on determining the clock frequency offset between Ethernet nodes through the RTT protocol. Determination of the clock phase offset will be investigated as part of the future works. The implementation of PTP combined with RTT will not be carried out as part of this project. It will only be carried out through simulations and will not involve any sort of hardware. The result may differ in hardware as compared to simulations and hence the implementation of this protocol in hardware will be part of the future works. Only the clock frequency offset is calculated and a way to correct the clock based on the determined offset is not investigated. Hence, the clock synchronization error is not determined after correcting the clocks, but it is calculated by comparing the injected offset and the calculated offset from the RTT protocol.

1.7 Structure of the thesis

Chapter 2 presents relevant background information about PTP and RTT. Chapter 3 presents the methodology and method used to solve the problem. Chapter 4 describes the RTT implementation carried out through this thesis. In chapter 5 the results based on the implementation is discussed and an analysis is performed.

Chapter 6 summarizes the entire thesis along with additional works to be carried out by future teams.

(21)

Background | 5

2 Background

This chapter provides basic background information about Ethernet clock synchronization error, Ethernet packet and the Ethernet OSI model. Additionally, this chapter also provides information about the PTP which helps to improve the accuracy and precision of clock synchronization to some extent. The implementation of 5G technology enforces more stringent requirements on networks which necessitates even lower clock synchronization error. This leads to stringent requirements being placed on clock skew and clock jitter tolerance values [12]. Hence a protocol called the RTT will be implemented to determine the clock frequency offset, which can be used to reduce the clock synchronization error by combining it with the PTP protocol. Some background information about the PTP and RTT protocol is provided in Section 2.2.

2.1 Ethernet

In the case of devices communicating through Ethernet, clock exchange is absent between nodes for synchronization. The frequency synchronization aspects of an Ethernet packet network are defined in [13]. In Ethernet, communication synchronization takes place through a field called the Preamble which is part of the Ethernet packet. As Ethernet speeds have reached a staggering 25 Gigabits per second (Gbps) this kind of synchronization is not enough. When the synchronization takes place only through the preamble field, there tend to be clock offsets creeping into the system. There are two kinds of clock offset:

1. Clock skew (Clock phase offset):

In this case, the clocks have the same frequency but the phase of one clock is shifted as compared to the other. This offset is depicted in Figure 2-1. In this figure, clock 1 is the reference clock, and the first rising edge of clock 2 is shifted to the right by some amount as compared to the first rising edge of clock 1. The difference between the rising edges of the two clocks forms the clock skew or clock phase offset as marked in the figure. This is usually caused due to mesochronous clocking synchronization.

(22)

6 | Background

Figure 2-1: Clock Skew

2. Clock jitter (Clock frequency offset):

Clock jitter occurs when reflections and unbalanced loads in the transfer functions of the clock wires make the edge jump back and forth in time. In this case, the frequency of one clock differs from the frequency of the other clock [11]. This is visible in Figure 2-2. In this figure, the period T2 maybe sometimes smaller than T1 and sometimes larger, but on the average the same. If Clock 1 is the reference clock, then Clock 2 has a jittered clock or phase offset jitters. Two frequencies that are almost the same but have a little frequency difference is called "Plesiochronous clocking". This offset is usually caused due to improper clock generator circuitry, noise or interference.

Figure 2-2: Clock Jitter

The jitter and wander parameters for a slave clock are defined in [12]. In the following subsections 2.1.1 and 2.1.2, a brief overview is provided of the Ethernet frame and packet structure. This is followed by an explanation of the Ethernet Open

(23)

| 7

Systems Interconnection (OSI) model. The information provided in these sections is very crucial for the implementation of the RTT protocol.

2.1.1 Frame and Packet

The Ethernet packet consists of various fields such as the Preamble, Start Frame Delimiter (SFD), destination address, source address, a length or type field indicating the characteristics of the user data that follow, a field with padding bits if necessary and finally a Frame Check Sequence (FCS) which is used for detecting errors during packet reception [1]. The FCS consists of an error checking code for error detection and correction if any errors arise during transmission of a packet.

The relationship between an Ethernet frame and packet can be seen in Figure 2-3.

Figure 2-3: Ethernet frame and packet format

Most of the Ethernet communication is asynchronous in nature. Due to the rapid digital bits transitioning from a 0 to 1 or 1 to 0, the transitions are not sharp and leads to rounding around the transitions which lead to an eye like formation between the transitions. A clock and data recovery unit are used to recover the bits by sampling in the middle of the eye [14][15]. As no direct exchange of clocks takes place between the nodes, the preamble field helps to synchronize the various Ethernet communicating nodes. The Preamble is made up of 8 bytes, the first seven bytes are the same: 10101010 and the last byte is 10101011. The preamble wakes the receiver and informs it to pay attention to the incoming information. It also helps in clock synchronization by using the 10101010 patterns. To synchronize, the receiver needs

(24)

8 | Background

to determine the exact time when a bit transition from 1 to 0 and back to 1 and so on.

By doing so, the receiver can set its clock speed to the frequency of the incoming message. This ensures that the receiver’s timing is synchronized to that of the transmitter by having the receiver’s PLS circuitry reach its steady-state synchronization. In this way, when a packet is received, the Ethernet node always synchronizes the clocks to that of the transmitting node which ensures that the nodes have their clocks synchronized. But this sort of synchronization is not always perfect and leads to clock synchronization errors.

2.1.2 OSI Model

The speed of Ethernet communication has reached 25 Gbps and it supports the full- duplex operation. In the OSI model for the 25 Gigabit Ethernet depicted in Figure 2-4 [1], the MAC layer is connected to the physical layer through the 25 Gigabit Media Independent Interface (GMII). As seen in the figure, the Physical (PHY) layer implements the Reed-Solomon Forward Error Correction (RS-FEC) clause 108 of the Institute of Electrical and Electronics Engineers (IEEE) standard for Ethernet.

This sublayer in the OSI model assists in error detection and error correction [1].

Through this project, a new protocol to determine the clock frequency offset will be implemented in the Physical (PHY) layer of the OSI model with the help of the RS- FEC sublayer.

(25)

| 9

Figure 2-4: OSI Model for Clause 108 RS-FEC Ethernet

The RS-FEC sublayer is further expanded to understand more about its internal blocks. The expanded RS-FEC sublayer is visible in Figure 2-5. Among all these blocks, the most important blocks are the Reed-Solomon Encoder, Reed-Solomon Decoder, codeword marker insertion and the codeword marker synchronization. The Reed-Solomon Encoder block uses the RS (528,514) code to encode the information bits for error detection and correction. This block converts the 514 information symbols to a codeword containing 528 symbols by appending parity symbols. It consists of 514 information symbols and 14 Parity Symbol (PS). Each symbol is made up of 10-bits. In case of the Reed-Solomon Decoder, the PS is used for correctly decoding the received codeword to extract the information symbols and correcting them if necessary. For 25 Gbps Ethernet, the time between the beginning of successive PS is 204.794843 nanosecond (ns) or 4.882935463 megahertz (MHz).

The PS is added to the codeword according to clause 108.5.2.5 and they are decoded

(26)

10 | Background

according to clause 108.5.3.2. This block can correct errors in a combination of up to 7 symbol errors [1].

In the codeword marker insertion block, Codeword Marker (CWM) is inserted into the transcoded blocks [1]. These markers form the first 257-bits of every 1024^th RS-FEC codeword. In the case of 25 Gbps Ethernet, the time between the beginning of successive codeword makers is 209.709919 microsecond (µs) or 4.768492 kilohertz (kHz). These CWMs are very useful for the receiver since the codeword marker synchronization block uses these alignment markers to obtain a lock to the incoming bitstream and assure codeword alignment. The CWM is inserted in the sender codeword which is specified in clause 108.5.2.4 and they are removed according to clause 108.5.3.4. The insertion and removal function of RS-FEC sublayer is part of the PHY layer.

Figure 2-5: RS-FEC sublayer

(27)

| 11

2.2 PTP and RTT

This section provides a basic overview of the working of the PTP which is currently used to determine the clock offset between Ethernet communicating nodes. Also, some description is provided for the RTT protocol.

2.2.1 PTP

The PTP protocol is mainly concerned about the transmission and reception timestamps of packets at both the master and slave node. Consider Figure 2-6, assume that the slave has a clock offset as compared to that of the master. The timestamps are collected when the Ethernet packets are transmitted and received at a node. The PTP protocol uses messages such as SYNC, DELAY_REQ and DELAY_RESP for determining the timestamps. The way the protocol works is illustrated in Figure 2-6:

Figure 2-6: PTP protocol

• Master begins by transmitting the SYNC message to the slave at time t1 and appends this timestamp to the message.

• Slave receives this SYNC message at time t2 and stores this timestamp.

(28)

12 | Background

• Slave transmits DELAY_REQ message at time t3 and appends timestamps t1, t2 and t3 to this message.

• Master receives the message at time t4 and collects all the timestamps t1, t2, t3 and t4, and responds by sending DELAY_RESP to slave.

Mathematical equations will be applied to these timestamps to determine the clock phase and frequency offset of the slave [8][9]. These values will help to correct the clock at the slave thus achieving a slave clock which is synchronized with the master clock. In this manner, this protocol reduces clock synchronization error. But with new technology and telecommunication networks progressing from 4G to 5G, stringent requirements are placed on networks, thus necessitating tighter synchronization in communication networks.

2.2.2 RTT

To reduce the clock synchronization error, the RTT protocol will be investigated.

Some of the ways to measure the RTT values are illustrated in [7]. This protocol will use the round-trip time of a transmitted packet between two Ethernet communicating nodes. One node act as the master and other as the slave. The working of this protocol is depicted in Figure 2-7.

Figure 2-7: RTT protocol

(29)

| 13

• Master transmits an Ethernet packet and a timer is started at that instant

• Slave receives the packet

• After some pre-defined period, the slave transmits the packet to the master

• When the master receives this packet, the timer is stopped

• The value of the timer is the round-trip time of that packet

This process is repeated for many packets and hence generates many RTT values.

Mathematical equations are applied to these RTT values which will help to generate the clock phase and frequency offset values of the slave as compared to that of the master.

2.3 Related work

The RTT protocol for Ethernet communication is new and has been suggested in some of the papers [2] [3]. In one of the major paper [2], the round-trip time is measured for wireless communication networks to determine the clock offset parameters. Whereas in the other [3], the clock synchronization error of the PTP is compared to that of PTP and RTT combined for a wireless communication network.

2.3.1 Initial RTT proposal for wireless communication

The way to synchronize clocks in wireless communication networks is much easier and efficient using the RTT protocol since only the round-trip time of a packet is to be determined and simple hardware is used for this purpose [2]. The master and slave nodes require to detect the exact time when the packets are transmitted and received. In the case of PTP protocol, these timestamps will have to be appended to the packets which will consume a portion of the communication channel resources.

Whereas, in the case of the RTT protocol it does not use channel resources for transmission of packet timestamps which makes it more attractive. In this paper, the RTT protocol achieves an accuracy in the timescale of a few nanoseconds and determines the clock phase difference which will prove to be very essential for communication systems as the speeds are increased higher and higher.

(30)

14 | Background

The paper provides an insight into the way an RTT measurement can be captured between two communicating nodes (one acting as the master and the other as a slave) in a wireless network. Figure 2-8 depicts an initial analysis of measuring the RTT values from transactions between a master a slave, where the slave clock has a phase and frequency offset. The RTT values gathered are then plotted to generate a sawtooth-like waveform.

Figure 2-8: Initial analysis to generate and plot RTT values

The RTT values need to be further processed to calculate the clock phase and frequency offset. To accomplish this task three algorithms were considered:

Unwrapped Least Squares (ULS), Periodogram and Correlation Peaks (PCPs) and Weighted Least Squares (WLS). The performance of these algorithms to detect the clock phase and frequency offset were compared to determine which among them is the better algorithm.

The experiment was carried out by using two Xilinx Virtex-5 FPGAs part of ML505 evaluation boards communicating wirelessly along with an Agilent 53230 A frequency counter, and a host computer to store the RTT values to process them offline. The three algorithms were used to determine the actual clock phase and frequency and compared to the expected values. From the comparison, it was found that the WLS performed better among the three algorithms.

2.3.2 Combining PTP and RTT

The previous systems used the incoming data rate to synchronize the internal clocks with the sender’s clock. But as advances in communication networks have taken place, it is no longer possible to extract that information. Even new technology such as the Global Positioning System (GPS) receivers is being used to distribute the clocks. But it is not possible to have multiple GPS clock receivers due to cost and accessibility issues. In recent time, people are moving towards the PTP protocol

(31)

| 15

which helps to achieve clock synchronization within a few hundred nanoseconds, but as newer and newer technologies are being invented precision is moving towards sub-nanosecond synchronization. This led to CERN striving to develop a sub- nanosecond clock synchronization protocol called the White Rabbit (WR) [16]. But to implement this protocol it required replacing the existing switches with SYNC switches which were quite expensive. Hence, there was a need to find a more suitable alternative. Thus, it was decided to explore the option of combining the PTP with RTT protocol [3].

Through the implementation of either the PTP or the RTT protocol, it is very important to determine the clock phase and frequency offset of the slave as compared to the master for nodes which are not connected to GPS. The determined offset will be used to correct the slave clock so that it is synchronized with the master clock.

In [3], the experiment is carried out by modelling two clock sources connected by a 1 Gb Ethernet link and ensuring various packet transactions between the two nodes termed as master and slave. By using the PTP, timestamps are collected of transactions between the master and slave. To determine the clock phase and frequency offset of the slave, equations are applied to these timestamps. At the same time, RTT values are sampled at a rate greater or at least twice the assumed frequency difference between the two clocks. This ensures that the method follows the Nyquist criteria. Due to this way of sampling the RTT values, there can be folding in some jitter sidebands which can cause issues, hence the jitter needs to be controlled or filtered [12]. Since the values are captured at a rate of at least twice the frequency difference between the master and the slave clocks, it leads to the generation of a sawtooth waveform. From this waveform, it is possible to determine the clock phase and frequency offset of the slave.

Both the PTT and RTT provide clock phase and frequency offset parameters which are vital for synchronizing the slave clock with that of the master clock. But when the PTP protocol is implemented as a standalone protocol, the clock synchronization error has a large magnitude. To reduce this error, the experiment combined the clock offset parameters generated from the PTP and RTT implementations and based on the resultant values corrected the slave clock. In doing so, the clock synchronization error reduced to a value less than only when the PTP protocol is implemented as can be seen in Figure 2-9 [3]. In the case of PTP, the accuracy of the timestamp samples is very important for accurately determining the clock synchronization error. Since the timestamping clocks are discrete, the actual

(32)

16 | Background

timestamp event may take place between the clock edges and hence may lead to a finite uncertainty. The PTP protocol does not detect phase difference and since the amount of uncertainty depends on the clock phase and frequency difference, this error will be visible in the timestamps and hence it will have a finite amount of clock synchronization error. But in the case of the RTT protocol, it is possible to determine the phase difference and when combined with the PTP measurements it is possible to reduce the error.

Figure 2-9: Compare clock synchronization error

The implementation of the RTT protocol has several advantages over other methods. One of the key aspects of this protocol is that it can be implemented with the existing hardware, not requiring any change. Also, it does not require an exchange of any timestamps for computing the clock offset values but only uses the transmission of Ethernet packets between the nodes to compute the offset. The combination of PTP and RTT protocol does not require any complex changes to the system.

2.3.3 Information on available Ethernet designs

An existing DUT and a VIP are used for the implementation. The DUT has numerous communication channels consisting of individual RX and TX data lanes connected to the VIP. This implementation will make use of only one channel consisting of one DUT TX connected to the VIP RX and the VIP TX connected to the DUT RX. Ethernet packets are transmitted between the DUT and the VIP blocks. To make things

(33)

| 17

simpler and relate to the previous work, we consider the DUT to be the master and the VIP block to be the slave.

The design also has a testbench to initiate different tests between the DUT and the VIP blocks. Different parameters such as the number of packets and enabling RS-FEC encoding/decoding can be controlled through this testbench. Also, all the design files are available thus allowing changes to be made to the source code.

Both the blocks use the 25 Gbps IEEE 802.3 clause 108 RS-FEC encoding which is part of the PHY layer in the OSI model. This clause defines PS for codewords which help in error detection and correction in the received codeword. It also defines CWM which is a special alignment marker for receiver synchronization. In the case of PS, when the PS is detected in the transmit channel of the master node a Detect Parity_Transmit (DP_TX) signal is asserted for one clock period. In the receiver channel of the master node, when a PS is detected a Detect Parity_Receive (DP_RX) signal is asserted for one clock period. The DP_TX and DP_RX have a periodicity of 204.794843 ns or 4.882935463 MHz. This is depicted in Figure 2-10. These two signals are available during simulations.

Figure 2-10: RTT using PS signals

In the case of CWM, when a packet is transmitted from the Master node, the CWM is detected on the transmitting channel and a Start of Frame signal for the Transmitter (SoF_TX) is asserted for one clock period in the master node. Whereas, when the CWM is detected on the receiving channel of the master Ethernet node a Start of Frame signal for the Receiver (SoF_RX) is asserted for one clock period.

These two signals are available while simulating the designs. A depiction is presented in Figure 2-11. These SoF_TX and SoF_RX have a periodicity of 209.709919 µs or

(34)

18 | Background

4.768492 kHz. These signals can also be used for the implementation of the RTT protocol.

Figure 2-11: RTT using SoF signals

2.4 Summary

This section provides all the necessary information about Ethernet communication and the OSI model. It also highlights the working of the PTP and RTT protocol. It is visible that the RTT protocol is simple and easy to implement. It has been depicted that through the implementation of the RTT protocol it is possible to determine the clock offset parameters. Also, when this protocol is combined with the existing PTP, the clock synchronization error is lower than the PTP implementation alone. In the case of the related works, it mentions about only using the round-trip time of the packet to determine the clock phase and frequency offset. A new way to generate the RTT values will be explored which will be used to calculate the clock frequency offset.

(35)

The RTT Protocol | 19

3 The RTT Protocol

The purpose of this chapter is to provide an overview of the research method used.

Section 3.1 details the research paradigm. Section 3.2 focuses on the data collection techniques used for this research. In Section 3.3 the experimental design is described. Section 3.4 explains the techniques used to evaluate the reliability and validity of the data collected. Section 3.5 describes the method used for data analysis.

Finally, Section 3.6 describes the framework selected to evaluate the RTT protocol.

3.1 Research Paradigm

There are two ways of evaluating the design, one was either through simulation or through hardware implementation. In the case of simulation, the Ethernet design files were available for the DUT and the VIP blocks. Also, the link between the DUT and the VIP was tested previously and confirmed as working. A simulation of the design would allow a quick debug and fix if any errors were encountered. But the simulation approach would lack the accuracy of implementing the protocol in hardware since it would be difficult to model real-life parameter variations. Various parameters such as transmission line jitter and wander, and physical anomalies would be harder to implement in simulation and hence would not be able to mimic the exact hardware behavior.

In the case of hardware implementation, real-life parameters will be monitored as compared to simulating a design in software. But a hardware implementation will require an ample amount of time to get acquainted with the new hardware and to get the setup ready. The amount of time is a crucial parameter since it should be completed in a span of around 21 weeks. Also, another decisive parameter is the cost of the implementation. In the case of hardware, it will incur a larger cost due to the procurement of hardware resources. But in the case of simulation, it would only require providing access to the simulation tools which is a very low-cost process.

Once all the pro and cons of the hardware and simulation implementations were listed down, they were compared to decide which method is to be followed. It was concluded that due to time and cost constraints, the simulation approach will be used as the implementation method and would be most suited to carry out the project.

The project supervisor also agreed with the same and supported the claim of using the simulation approach.

(36)

20 | The RTT Protocol

To generate the RTT values, either the CWM or PS could be used. In the case of the PS, it was possible to achieve a sampling frequency of 4.882935463 MHz. This sampling frequency is very large and would lead to capturing many RTT values in a given time frame. Thus, when these RTT values are plotted on a graph it will lead to higher precision. Whereas in the case of the CWM, it has a sampling frequency of 4.768492 kHz. In this case, fewer RTT values are captured in a given time frame leading to lower precision of the graph. Thus, it was concluded to use the PS for the generation of the RTT values due to its higher precision and sampling frequency.

3.2 Research direction

Some of the decisions that were made before beginning the implementation phase are listed:

• It highly possible to generate the required sawtooth waveform from the output of Ethernet design simulations.

• The sawtooth waveform will be generated by transmission of Ethernet packets in Ethernet designs employing Clause 108 Ethernet standard.

• The Ethernet packets will be transmitted between a DUT and a VIP block.

• The PS in the packets will be detected by the master on the transmission and reception channels.

• The time between the detection of the parity symbols on the two lanes at the master will make up one RTT value.

• The RTT protocol will be implemented only in the master node.

The modified way of generating the RTT value uses the PS from Clause 108. The protocol calculates this value from the time difference between the transmission and reception of Ethernet packets at the master. In the case of Clause 108 RS-FEC, the codeword is 5280 bit long which includes 5140 information bits and 140 parity bits.

The parity bits form the PS are detected in the TX and RX directions. Each packet which is transmitted on the TX and RX lines are 40-bit packets as seen in Figure 3-1.

Thus, the codeword will be transmitted in multiple packets. The bits following parity bits of the first codeword including the parity bits of the second codeword forms one codeword. As can be seen from the Figure 3-2, whenever the PS is detected on the TX direction in successive codewords the DP_TX signal is asserted once per RS-FEC codeword in a specific position and the same is true for the RX direction. The PS detection in the transmit and receive direction does not require to be from the same

(37)

| 21

codeword but can be from different codewords. The detection of the PS on the TX and RX are periodic as can be seen from Figure 3-2. Thus when there exists a clock frequency offset between the master and slave clocks, the time between the assertion of the DP_TX and DP_RX signals are constantly varying. This value is considered as the RTT measurement.

Figure 3-1: PS detection and packet size

Figure 3-2: Periodic PS detection

For some basic understanding, an initial (hand-sketched) waveform is depicted in Figure 3-3 for calculating the RTT values. This is a modified way to generate the RTT values than the one presented in Figure 2-8. The explanation for the waveforms is presented below:

(38)

Figure 3-3: Modified way of acquiring the RTT values

• In Figure 2-8, the transaction illustrates the normal RTT (As seen in [2]). In this case, the RTT value is calculated between the time when a packet is transmitted from a master to the slave to the time when the same packet is received by the master from the slave.

• The waveform 1 in Figure 3-3, illustrates the modified way to generate the RTT based on the available Ethernet designs. The Ethernet packet exchange takes place between the Ethernet DUT node (master) and the VIP node (slave). In this case, both the master and slave clocks are synchronized. The master begins transmitting at its first clock edge, whereas the slave begins at its second. Both the nodes begin transmitting periodically at every 4^th clock edge. At the master, the PS is detected when a packet is transmitted and received. The time difference between the detection of the PS on the transmission and reception lanes form the RTT value. As can be seen from the RTT values plotted below the transactions, the time between detection of the PS on the TX and RX lanes at the master is the same for all the intervals from 1-22. Hence, we get a straight line from the plotted RTT values.

• The waveform 2 in Figure 3-3, illustrates the modified way to generate the RTT but with a de-synchronized slave clock. In this case, we consider the slave clock frequency to be offset from that of the master. It is assumed that the frequency offset is varying (the clock frequency is changing multiple times) since if we considered a fixed frequency offset it would take many transactions to display a hand-drawn sawtooth waveform from the RTT values. As can be seen from the plotted RTT values, they first decrease until the 21^sttransmission from the master end and then increases at the 22^nd transmission. Thus, generating a sawtooth waveform.

Waveform 1

Waveform 2

(39)

| 23

To explain the concept of how the PS will be able to generate the RTT values and in turn, a sawtooth waveform will be clear from Figure 3-4 and Figure 3-5. Figure 3-4, displays three transactions between the master and the slave when there is no clock frequency offset. On the left side, we have the master with its positive clock edge marked as a dash and on the right side, we have the slave with its positive clock edge marked as a dash. When there is no clock frequency offset, the master transmits the packet and a timer is started at the master end when the PS of the packet is detected. The timer is stopped when the PS of the receiving packet is detected. This value of the timer forms the RTT value. This process is repeated for many packets.

The detection of the PS in the packets takes place after every 204.794843 ns (4.882935463 MHz). The first, second and third RTT values are the same since there is no clock frequency offset between the two nodes. As can be seen from Figure 3-4, the three RTT values are the same for the Ethernet packets.

Figure 3-4: No clock offset - generate RTT values from PS

(40)

Figure 3-5 displays three transactions between the master and the slave when there is a clock frequency offset present. When the master clock is slower as compared to the slave clock, the transmission of the packet from the receiver takes place at a time earlier than the expected time and hence the reception of the packet at the master takes place at an earlier time. This causes the time difference between detection of the PS to be lower than when no frequency offset was present. As can be from the figure as the slave clock is faster, the slave will transmit at a faster time thus reducing each RTT value as the transactions are progressing. The first RTT value is larger than the second, which is larger than the third and so on. This decrease in RTT values continues until the value reduces to the lowest possible value. Once the RTT value reaches the lowest possible value, the next RTT value returns to its maximum possible value. Plotting these RTT values will lead to a sawtooth waveform.

(41)

| 25

Figure 3-5: Clock frequency offset present - generate RTT values from PS

3.3 Data Collection

The DUT and the VIP will be simulated to transmit Ethernet packets back and forth between the blocks. Based on the transmission between the two nodes, the RTT values for packets will be determined from the simulation. The RTT values will be sampled at a sampling frequency of 4.882935463 MHz. These values will be exported as a Comma Separated Value (CSV) format file. Once exported, this file is directly read in Microsoft Excel 2016. Since only simulation data is dealt with, it will follow all the social and ethical concerns.

(42)

3.3.1 Sampling

The simulation signals sampled are:

1. Number of clock cycles between the assertion of the DP_TX and DP_RX signals.

2. RTT values

3. Time when the RTT values are captured

3.3.2 Sample Size

The number of samples required is 27490 RTT values to generate the sawtooth waveform which will help to determine the clock frequency offset. A large number of samples are chosen so that it will be possible to generate a periodic sawtooth waveform.

3.4 Experimental design/Planned Measurements

To mimic the same setup as implemented in this project, a few steps will have to be followed which are listed in this section. Also, access to some of the software and Ethernet design files will be required. The following steps will have to be followed to reciprocate the same test environment the project used:

• Get access to Cadence Xcelium 18.12.001

• Get access to design files for an Ethernet communicating DUT and VIP block

• Open the design files in Cadence Xcelium

• Inject clock frequency offsets in the design

• Update the design code and use the PS detection signals to generate the RTT values

• Export the RTT values

• Plot the RTT values

• Implement a mathematical model to process the RTT values to determine the clock frequency offset

3.5 Assessing reliability and validity of the data collected

Reliability and validity of the acquired data are very important aspects of any project.

Hence it is of prime importance to verify these two parameters. The reliability and validity of the data may vary if implemented in hardware.

(43)

| 27

3.5.1 Reliability

The existing Ethernet designs have been undergoing rigorous testing for a long duration. Some of the designs are currently implemented in various devices and have no issues while working in the field. The existing test benches are currently in use and considered as very reliable. The RTT values are exported from the simulation are reliable. This proves that the entire process of data gathering is very reliable.

3.5.2 Validity

The RTT values exported from the simulation are valid since it can measure the time difference between the assertion of DP_TX and DP_RX very accurately. The exported RTT values are compared to the actual time difference between the assertion of the signals from the simulation and these values are equal.

3.6 Planned Data Analysis

Once the RTT values are exported from the simulation, they need to be further analyzed to gather additional information from these values. The data analysis technique will be explained along with the software tools required to analyze the data in this section.

3.6.1 Data Analysis Technique

The RTT values from the simulation will be exported as a .csv file. This file will be imported into Excel. Once imported, these values will be plotted against the sample index to check if the plotted values result in a periodic sawtooth waveform. The frequency of the sawtooth waveform is determined, and mathematical models will be used to determine the clock frequency offset.

3.6.2 Software Tools

The software tools used are:

• Cadence Xcelium version 19.03.001

• Gvim version 7.4.417

• All simulations were done a Hewlett-Packard workstation running Linux OS

(44)

3.7 Evaluation framework

The results will be evaluated by comparing the expected clock frequency offset to the actual clock frequency offset. The expected clock frequency offset is the offset injected in the system before the start of the simulation, whereas the actual clock frequency offset is the value calculated from the RTT protocol. If the two values are similar it will prove that the RTT protocol can successfully determine the clock frequency offset between two Ethernet communicating nodes. If they do not match, then the RTT protocol cannot detect the clock frequency offset.

(45)

Implementation | 29

4 Implementation

In this project, codes were implemented to inject clock frequency offsets in the clocks of the Ethernet link and to capture the RTT values. These designs were simulated and the RTT values and the time when they are sampled were exported. These values are imported into Excel and a graph is plotted for further analysis.

4.1 Code Implementation

The two System Verilog (SV) Register Transfer Level (RTL) codes implemented to mimic the requirements will be explained in detail in this section. These RTL codes will implement two functionalities:

• Inject clock frequency offset

• Calculate the RTT value

4.1.1 Inject clock frequency offset

The TX and RX Ethernet clocks are susceptible to variation in their operating clock frequency. The TX and RX clocks can tolerate frequency variations in the range of +/-100 parts per million (ppm). We consider the two links between the DUT and the VIP. The nomenclature used for the links are:

• Link 1: DUT TX connected to VIP RX

• Link 2: DUT RX connected to VIP TX

Each of these two links works at a frequency of 644.595709571 MHz. To verify the effect of the frequency offset in the Ethernet links, a positive frequency offset is injected in one link and a negative offset in the other. It is also possible to consider one link has a perfect clock and inject the offset in the other and carry out the experiment. But the implementation focused on an extreme case of injecting offsets on both the links. Even if both the links are offset, it would be possible to visualize one as fixed and calculate the total offset between the two links. Based on the frequency offset to be injected in the links, calculate the minimum and maximum frequencies from the expected frequency. The calculation for the minimum and maximum frequencies is as follows:

(46)

30 | Implementation

For negative offset,

𝐹𝑚𝑖𝑛 = 𝐹𝑒𝑥𝑝𝑒𝑐𝑡𝑒𝑑 − 𝐹𝑒𝑥𝑝𝑒𝑐𝑡𝑒𝑑 ∗ 𝑜𝑓𝑓𝑠𝑒𝑡 𝑝𝑝𝑚 (1) For positive offset,

𝐹𝑚𝑎𝑥 = 𝐹𝑒𝑥𝑝𝑒𝑐𝑡𝑒𝑑 + 𝐹𝑒𝑥𝑝𝑒𝑐𝑡𝑒𝑑 ∗ 𝑜𝑓𝑓𝑠𝑒𝑡 𝑝𝑝𝑚 (2)

From these values of Fmin and Fmax the values of clock periods can be calculated as:

𝑇𝑚𝑖𝑛 = 1

𝐹𝑚𝑎𝑥 (3)

𝑇𝑚𝑎𝑥 = 1

𝐹𝑚𝑖𝑛 (4)

The algorithm for injected clock frequency offsets in the Ethernet links is visible in Figure 4-1. First, create two processes to generate the clock for links 1 and 2. In this example, a negative clock frequency offset is injected in the clocks of link 1 and a positive clock frequency offset is injected in clocks of link 2. Begin by assigning Tmax and Tmin to parameters Clock_delay1 and Clock_delay2 respectively. These parameters will be used to continuously toggle the clocks after every Clock1_delay/2 for link 1 clocks and Clock2_delay/2 for link 2 clocks. This will lead to the link 1 clocks namely DUT_TX_Clock and VIP_RX_Clock having the same period as Tmax.

Also, link 2 clocks, namely VIP_TX_Clock and DUT_RX_Clock, will have the same clock period as Tmin. In this way, the clocks of link 1 will have a negative clock frequency offset and the clocks of link 2 will have a positive clock frequency offset.

(47)

| 31

Figure 4-1: Algorithm for injecting clock offset

4.1.2 Calculate the RTT values

As mentioned in Section 3.3, to calculate the RTT value the DP_TX and DP_RX signals of the DUT are used. The time difference between the assertion of these signals will provide a single RTT value. To check when these signals will be asserted a third clock called the radio clock is used. This clock has a very high frequency of 983.119832476 MHz as compared to the Ethernet TX and RX clocks (644.595709571 MHz). This clock was available in the current design and was used for some other purpose. The high-frequency clock used might reduce the power efficiency of a system if power was the main concern. But a clock with a lower frequency could also be used but might result in lower precision in determining the clock frequency offset.

The algorithm for capturing the RTT value is depicted in Figure 4-2.

(48)

32 | Implementation

Figure 4-2: Algorithm for calculating the RTT values

(49)

| 33

At the start, the existing System Verilog RTL codes are updated to inject clock frequency offset in the Ethernet links using the GVIM tool. Next, simulate the design in Cadence Xcelium, and packets are transmitted between the master and the slave.

In the next step, the radio clock samples the DP_TX signal and checks if it is asserted, this signal indicates the time when the measurement of the RTT value should begin.

If the DP_TX signal is asserted, a timer is started when the DP_TX assertion is detected and a timer_trigger signal is asserted. This signal ensures that the timer continues counting until the DP_RX signal is asserted, and this timer value is stored in a variable called timer_value. When the DP_RX signal is asserted, the timer stops counting and de-asserts the timer_trigger, the RTT value is stored in a variable called RTT_value and the timer is reset. The timer value before being reset depicts the total time between the assertion of DP_TX and DP_RX. A mathematical model is applied to the acquired RTT values to determine the clock frequency offset between the links.

4.2 List of Test

To get a good confidence in the simulation data and to make suitable inferences from the project, the design was tested by injecting different clock frequency offset values as listed in Table 4-1. The design is simulated with three different offsets: +/- 45 ppm, +/- 75 ppm and +/- 90 ppm. The time resolution of the available design was 1 picosecond (ps) and thus when a positive and negative offset was injected in the clocks a slight deviation was noticed from the required offset. Hence, all the parameters in Table 4-1 are calculated from the simulation. In each of these cases, a negative clock frequency offset is injected in clocks of link 1 and a positive frequency offset is injected in the clocks of link 2. The link frequency offset is the difference between the expected link frequency when no offset is present and the link frequency when an offset is applied (either positive or negative) which is called the actual link frequency. The total frequency offset is the combined link frequency offset of link 1 and link 2. For example, when a - 45 ppm offset is injected in link 1, the frequency of the link reduces by 29.086566 kHz and when a + 45 ppm offset is injected in link 2 the frequency of the link increases by 29.086566 kHz as seen in Table 4-1.

𝐿𝑖𝑛𝑘 𝑥 𝑓𝑟𝑒𝑞𝑢𝑒𝑛𝑐𝑦 𝑜𝑓𝑓𝑠𝑒𝑡

= 𝑎𝑏𝑠(𝐸𝑥𝑝𝑒𝑐𝑡𝑒𝑑 𝑙𝑖𝑛𝑘 𝑓𝑟𝑒𝑞𝑢𝑒𝑛𝑐𝑦 − 𝐴𝑐𝑡𝑢𝑎𝑙 𝑙𝑖𝑛𝑘 𝑓𝑟𝑒𝑞𝑢𝑒𝑛𝑐𝑦)

(5)

𝑇ℎ𝑒𝑜𝑟𝑒𝑡𝑖𝑐𝑎𝑙 𝑓𝑟𝑒𝑞𝑢𝑒𝑛𝑐𝑦 𝑜𝑓𝑓𝑠𝑒𝑡 𝑏𝑒𝑡𝑤𝑒𝑒𝑛 𝑙𝑖𝑛𝑘𝑠

= 𝐿𝑖𝑛𝑘 1 𝑓𝑟𝑒𝑞𝑢𝑒𝑛𝑐𝑦 𝑜𝑓𝑓𝑠𝑒𝑡 + 𝐿𝑖𝑛𝑘 2 𝑓𝑟𝑒𝑞𝑢𝑒𝑛𝑐𝑦 𝑜𝑓𝑓𝑠𝑒𝑡

(6)