Automating Network System Configurations for Vendor-Specific Network Elements

(1)

(2)

Automating Network System Configurations for Vendor-Specific Network Elements

Golinski, Anne.

June 21, 2017

(3)

Abstract

In present day, telecommunications stakeholders have not yet fully automated hardware configurations. Network configuration and reconfiguration is a repetitive, time consuming, and error prone process. To address this problem this bachelor thesis is going to shed light on the benefits of an automated configuration and topology verification process. To this end, a proof of concept system, Enna, has been developed in a case study together with an ISP stakeholder. Enna reads the current network state, applies predetermined configurations loaded from text-files, and automatically verifies the network state. The goals of this thesis are as follows: development of Enna to illustrate the simplicity in the implementation, compare the automated network reconfiguration to a fully manual one, and finally discuss potential benefits or problems in switching to an automated network configuration. Since this thesis is carried out in collaboration with an ISP working with Cisco IOS XR devices, Enna is only tested for basic network protocols common in such environment, the conclusions drawn in this paper are therefore only applicable to such context.

Keywords. Automation, Network Configuration, Cisco IOS XR, Python, Netmiko, pyIOSXR.

(4)

Sammanfattning

I dagens läge tillämpar inte telekommunikationsintressenter automatiserade processer som standard. Nätverkskonfigurering och omkonfigurering är en repetitiv, tidskrävande och felbenägen process. För att adressera detta problem kommer denna kandidatavhandling belysa fördelarna med en automatisk nätverkskonfiguration och verifieringsprocess. För detta ändamål har ett system, Enna, framtagits som ett bevis på konceptets ändamålsen- lighet. Enna läser aktuell nätverksstatus, tillämpar förutbestämda konfigurationer som laddas in från textfiler, och verifierar nätverkstatus automatiskt. Målet för detta examen- sarbete lyder: utveckling av Enna för att illustreara simplisteten i framtagandet, jämföra en automatisk nätverkskonfiguration med en manuell , och slutligen diskutera fördelar och problem vid byte till en automatiserad närverskonfiguration. Eftersom att detta exam- ensarbete genomförts i samarbete med en ISP som arbetar med Cisco IOS XR-enheter, och Enna endast testats med grundläggande närverksprotokoll vanliga för dessa miljöer, kommer slutsatserna i detta dokument endast vara tillämpbara i dylikt sammanhang.

Nyckelord. Automation, Nätverkskonfiguration, Cisco IOS XR, Python, Netmiko, pyIOSXR

(5)

Acknowledgements

I first of all like to thank Com Hem and its employees, Anders Holm-Melin for trusting me with the development of Enna, and especially Alexander Wågberg, for without his professional help, the system would not have been built.

I also like to thank my supervisor, licentiate graduate Georgios Katsikas, for without his patience and constructive criticism the thesis would not have been possible, and Professor Dejan Kostic for his inspiration.

Finally, I would like to thank Felix Gustavsson for his excellent knowledge in Python programming.

Stockholm, May 2017, Anne Golinski

(6)

Introduction

This Chapter serves as an overview of the problem, purpose and goal of the thesis as well as the challenges and the delimitations that follow. Section1.1offers background information about automation, Sections1.2 states the general problem tackled by this thesis. Section 1.3 describe the purpose and goals of this thesis. Section 1.4 presents the methodology of the research as well as the thesis work, and Section 1.5 contains the challenges of this thesis. Finally, Section1.6outlines the remaining thesis.

1.1 Background

To keep the style of text conform and not influenced by different handwriting, the type- writer was a common text handler in late 19th centenary. The process is outlines by keys dipped in inc, are manually pressed to push the inc onto a piece of paper. In the early versions of this device, the document could not be fully inspected until the paper was taken out of the machine, controls would often jam, and if a typing error was present, the whole process had to be redone. These problems led to the development of new electric typewrit- ers and is nowadays totally replaced by the automatic process of computers and printers.

This shift emphasizes that a manual process that lead to bad results, will eventually be replaces by a process that not necessarily lead to fewer mistakes, but provides the user with the ability to correct these mistakes before producing the final result.

In 1998, NASA’s Mars Climate Orbiter was lost in space on its mission to Mars [1]. This failure resulted in a financial loss of $125 millions, because the trajectory software used an incorrect unit to compute the trajectory of the spacecraft. This example shows that automated processes can still be error prone, when their functionality is not systematically verified.

"Automatic" is defined as: Acting or operating in a manner essentially independent of external influence [2]. One can therefore claim that the process of automation can be implicitly related with benefits such as saved labor and cost. When automation is combines with testing and verification processes, the number of errors can be substantially reduced.

A manual process, such as cleaning teeth with a manual tooth brush, is repetitive.

Instead, using an automated toothbrush, appears to be more efficient and produces a better result than its hand operated counterpart. The repetitiveness in a manual process

(10)

implies that the process itself remains unchanged, however a manual execution of the process might lead to results that deviate from one execution to another. This deviation is possible because during the manual execution of a process’s internal steps, some of these steps might not be executed correctly. This is where automation can offer important benefits, by undertaking to accurately execute these steps.

In the area of networked systems, automation is related to the configuration of network elements that are in charge of realizing the communication between different users and services. This is challenging because not only one needs to verify the correctness of a single device’s configuration, but also the entire network configuration that results from the interconnection of all the devices. When working with automated network configuration it is therefore important to detect buggy software, leading to for instance delay in committed configurations, in order to avoid the potential effects of that bug on the stability or correctness of the network state. Canini et al. in [3] stated that: The centralized programming model, where a single controller program manages the network, seems to reduce the likelihood of bugs. However, the system is inherently distributed and asynchronous, with events happening at different switches and end hosts, and inevitable delays affecting communication with the controller. This statement emphasizes on the importance of combining automation with systematic verification techniques.

In 2016, the Accenture manager directors Marshall Wells and Miha Kralj published an article in DevOps [4] on why automation is crucial for IT maturity. Wells and Kralj stated that today’s systems must be resilient by detecting errors and being able to auto- tune themselves such that they restore the system’s stability. To detect such deviations, automation capabilities include the understanding of a "steady state" of where the system fulfills the previous mentioned requirement of system "health". In some cases, avoidance of an error could be possible, hence the system does not need a recovery process. This means that the system should be able to "predict" potential errors and act accordingly in advance. In the concept of a network system, the steady state would be the basic configurations required for a functioning topology. In essence, a steady state requires that (i) interfaces and ports are properly configured, (ii) desired protocols are enabled on devices, and (iii), desired protocols interplay in a functioning manner.

Established that automation introduce room for error correcting, saved labor and costs, and is in fact crucial for IT maturity, Kuzniar et al. [5] explicates the dangers of automation, being a subject under rampaging. Kuzniar share the understanding of the software specifications for a mode of operation under development can be under rapid flux, and with the demand for explicit implementation freedom, vendors does not always follow the specifics provided.

1.2 Problem Statement and Motivation

Manual network configuration requires concentration which humans often lack, especially in repetitive processes, in some cases due to lack of concentration [6]. Typical cases of network configurations are (i) device ports and interface configuration, (ii) link speeds, and (iii) network protocols. In large networks, the typical cases above are augmented with

(11)

Figure 1.1: Example topology with five routers, running three different protocols.

load balancing and peering. For a small network as depicted in Figure1.1, consisting of five devices, a total of 17 interfaces and six links has to be configured. Furthermore, each device has to be configured, with the signposted protocols, sometimes on different interfaces to avoid conflict, emphasizing the configuration repetitiveness even in small network areas.

When working with lab-environments, mechanisms for resetting the lab to basic functioning configurations, for instance a lab environment mirroring the backbone network of an Internet Service Provider (ISP), are in some cases not in place [7]. For each laboratory practical configurations are added into the environment and eventually there is so much configuration that very few people know what is useful and essential, and what to remove.

The lab-environment of an ISP, used to develop and test new services for the deployed network, is considered to be an extremely important platform. Current approaches to network configuration use tools, such as Secure Shell (SSH) to manually connect to each network element’s Command Line Interface (CLI) in order to pass the set of commands required to configure the device. Verification is then performed by entering additional commands and inspecting the output. This manual process requires deep understanding of network protocols and the way these protocols interact and affect each other.

This manual configuration process is performed by the ISP’s dedicated personnel, hence accompanied by concomitant costs relative to several factors such as size of the network, number of people who perform the configuration, their salaries, and other sources of costs, such as value-added taxes , health, and security insurance¹. A simplified financial example of the yearly expenses for manual configuration of the topology depicted in Figure 1.1, operated by one employee at a Swedish private-sector company.

In this example we assume that one employee is in charge of (re)-configuring the example topology shown in Figure 1.1 and for this task he/she spends 2 hours per week.

This time corresponds to 1/20 of this weekly working hours (assuming that he/she works 40 hours/week), hence roughly 1/80 of his monthly working hours. Assuming that the employee’s monthly salary is 33600 SEK [8], this translates to a cost of 420 SEK/month.²

1The last three sources of cost are specific to the laws of the country where the employers are working.

2Indirect costs related to the value-added tax, health, and security insurance are covered by this example.

(12)

In the case that larger networks have to me maintained and (re)configured, a greater amount of hours has to be put into the process, leading to roughly linear increasing cost when increasing the number of devices, and in the case of an augmented topology complexity the cost risk of following an exponential increase.

1.3 Goal

The problems stated in Section 1.2 act as a motivation for this thesis. The goal of this thesis is to: (i) develop a tool that automates network configuration, (ii) compare the automated process to a fully manual network configuration with respect to factors, such as time and costs, and (iii) discuss potential benefits or problems in switching to an automated network configuration. Our automated network configuration tool should: (i) be easy to implement, (ii) generate accurate network configuration, and (iii) expect little from the user regarding knowledge of network configuration. The purpose of the automated tool is to function as a reset process for the lab-environment of the Company stakeholder, providing the Company with a tool to easily return to a basic setup. The purpose of this thesis is to clearly state the potential benefits and problems in automating network configurations of telecommunication stakeholders.

1.3.1 Benefits, Ethics and Sustainability

When humans performed repetitive tasks, as previously mentioned in Section1.2, the psychological response of reluctance can appear. In 2010, Colleen Merrifield published a thesis trying to characterize the signature of boredom [9]. Merrifield found that the psychological signature of boredom, relative to sadness, possesses the same physical characteristics of rising heart rate and increased cortisol levels, also known as the stress hormone [10]. High and prolonged levels of cortisol are known impair cognitive functions such as concentration, lower the immunity system and even decrease bone density and muscle tissue. One could therefore claim that an automated network configuration process could come with not only psychological health benefits in reducing boredom, but psychophysiological as well.

As mentioned in section1.1, automation can be beneficial in its undertaking of accurate execution of configuration. This accuracy, reducing errors and deviations, eventuates that the time and costs, allocated by the corresponding manual process, can be used to perform other important tasks such as helping companies achieve more goals, grow, increase revenues, or be put into activities contributing to the employees quality of life, such as sports. By excluding the possibility for the employees to perform a manual configuration process resulting in errors, the potential concomitant emotions of inadequacy and failure are reduced, thus increasing the mental well-being of the individuals and therefore the employees overall peace of mind.

Many companies today are still relying on manual network configuration processes performed by humans, at least, so is the case for one of the most prominent ISPs in Sweden [7]. To accomplish these processes, companies consume a portion of their operational expenditures by hiring and training personnel, which can often set back the company, not

(13)

only financially, but also, as previously discussed, affect the health of the employees.

When opening a communication channel over an insecure network, enabling remote access to a device, the security is of concern. How to prevent malicious and ill-meaning figures from accessing the device via the same channel is a prominent and current question formulation. For example, a password protected device can fall victim of a brute force attack, where the attacker tries multiple authenticating data until gaining access to the system. By using tools with well-defined security measurements, such as SSH discussed in Section2.1, the security can be trans-located to being of secondary concern for the developer. In an isolated, not externally connected, network of for example a lab-environment, the security problem is already of low concern since there is no external communication channel for the the intruder to intercept. The intruder would have to physically apprehend the devices to gain access.

Working with tools recognized for their security, it is of importance to use the specified standards implemented. For example, if secure tools for remote access is not implemented on their standardized ports, the security features provided by the fully functional protocols is reduced, thus decreasing the complexity level of a feasible attack, thereby increasing the risk of a potential security failure. Note that keeping software updated to the latest release is a requirement to meet the alleged standards.

Current protocols and devices that are well integrated into modern network standards does not risk expiration, until a new modern tool is invented that is. Using well established hardware devices and software when building an automated tool therefore increase the assurance of sustainability over time.

1.4 Methodology

The research methodology that was used for this thesis started with an interview, with the backbone technical team leader at an ISP stakeholder, in order to establish the requirements for an automated configuration tool. A literature search was conducted to gain better understanding of the area and learn about existing network (re)configuration tools.

This project then followed a three-step approach described in Chapter3 and 4 where the tool was built in iterations in an agile manner. The final thesis paper was written in the same agile manner.

1.5 Challenges

Building network configuration software with detailed exception and error handling is not enough, as typically the number of possible states that appear in real networks is huge, hence it is hard to predict and handle all these states in advance. One requires more sound proof regarding the correctness of network automation processes, hence the systematic verification of such processes appears as the ultimate challenge. This serves as motivation for a circumstantial verification process for the automated tool, specified on configuration specifics since a fully exception and error handler is outside of the scoop of this thesis.

(14)

Tools that automatically configure networks might face scalability issues. These issues derive from the following facts: First, the number of devices that need to be configured might affect the time required to configure the network. Second, the type of each device affects the design and implementation of the automation process. For example, if the automated tool has to support a wide range of devices such as Cisco, Juniper, HP, Alcatel- Lucent, and Aruba, it affects the implementation effort that needs to be spent and the generality of the tool. Third, the type of configuration that needs to be installed also affects the completion time of the reconfiguration; for example a link-state (LS) routing protocol, such as Open Shortest Path First (OSPF), has faster convergence than a distance-vector (DV) protocol, such as Routing Information Protocol (RIP) [11].

The focus of the automated tool will be delimited to the needs and requirements given by the stakeholder Com Hem, in regards of deployed hardware and current software standards, practiced by the company.

1.6 Thesis Outline

Chapter2gives an introduction to the area this thesis consists of. Chapter 3explains the steps taken, the approach in addressing the problems, and tools used, in the development of the automated tool. Chapter4 contains the detailed implementation of the approach.

Chapter 5 focuses on the evaluation of the automated tool in a discussion emitting the conclusions drawn in Chapter 6, which also contains possible entries for future research, and a reflection about the thesis totality. As for the entire thesis, the company stakeholder, Com Hem and the Company will be used interchangeably.

(15)

Chapter 2

Theoretical Background and Related Work

This chapter provides the background required to understand the concepts of this thesis, especially network configuration. Sections 2.1, 2.2, 2.3 and 2.4 briefly explain software and protocols that play an important role in network configuration. Section2.5 discusses related work.

2.1 Remote Access

Remote access is useful when trying to facilitate communications to devices that are geographically distant, as well as when having to gain access to several devices at once, not having to connect via cable for each device.

A CLI is a way of interacting with computer systems, such as Operating Systems (OS) and applications. The CLI enables system communication and management in a text based manner. Common protocols used for CLI based network management are for example Telnet and SSH which both allow remote access to network devices.

The Telnet protocol is mostly used in an interactive manner, providing text based communication, between a user and a network element, but the protocol can also be used for communication between two automated processes, each process being e.g., a client and server, respectively. This protocol is not considered a secure protocol since, when logging into a remote host using Telnet, the authentication information is sent in plain text. Telnet runs on top of the connection oriented Transmission Control Protocol (TCP) when communicating with remote devices which provides more reliable communication than a connectionless protocol, such as User Datagram Protocol.

To overcome this limitation, the SSH protocol is one of the most recognized protocols for secure remote network services over an insecure network, providing encryption, cryp- tographic host authentication, and integrity protection [12]. A machine is accessible by a remotely located process via a secure channel provided by the SSH protocol. Like Telnet, SSH runs on top of TCP, but with security measurements providing a secure connection for device management over an insecure network, for example the Internet.

(16)

2.2 Cisco IOS

CiscoIOS^® is a multitasking operating system for Cisco-based network elements that provides services, such as routing[13], switching, internetworking, and other telecommunications functions. According to Cisco[13]: IOS is currently operating on millions of active Cisco devices, making it the most universally leveraged network infrastructure software, providing a wide set of commands to configure Cisco devices.

Cisco IOS provides different configuration modes for different user privileges. For example global configuration mode enables input of commands with the ability to change the system configurations, while the setup mode enables the configuration of more specific features, such as interfaces and protocols [14], in an interactive manner (request-reply). Cisco network elements can also be configured by loading a text file with configurations directly into device, instead of having to input the entire configuration in the form of commands.

Listing2.1 displays common Cisco IOS commands for a Cisco-based network element, performed on router A, found in the topology depicted in Figure2.1. The purpose of the commands in Listing2.1 is to firstly: configure the OSPF protocol on router A, in order for the router to be able to exchange Link State Advertisements (LSAs), the basic OSPF communication mean, with router B, and secondly: verify the configuration. The OSPF configuration postulates the interfaces depicted in Figure 2.1 being properly configured and enabled on both devices.

The configuration of OSPF starts in default User EXEC mode and navigates, firstly to Privileged EXEC mode with the command enable (line one), secondly into Global configuration mode with the command configure terminal (line two), and thirdly into the router configuration mode with router ospf 1 (line three), specifying the IP routing protocol to be configured to OSPF, and sets the OSPF process ID to 1 (the process ID facilitates the routers process handling, is of local significance only). While in router configuration mode the command network 192.168.1.0 0.0.0.255 area 0 specifies the network to distribute the LSAs, the related subnet mask, and the OSPF area (line four). The OSPF areas, which are sub-domains of the OSPF network, limits the distribution of LSAs, thus reducing the size of each device’s link state database. The backbone area combines remaining areas into a single domain and must be configured for OSPF to converge.

The last command on line eight in Listing 2.1 is described in Table 2.1, accompanied by the neighbor parameter, which briefs the output to only display the OSPF neighbor information, which consist OSPF ID (default loopback interface IPv4 address of the device),

Figure 2.1: Simple two-device topology.

(17)

Verification Command Description

show running-config Display the active configuration on devices local memory, including saved configuration changes.

show ip interface brief Display the usability status of interfaces configured for vari- ous IP addresses

show ip protocols Displays the parameters and the current state of the active routing protocol process, such as RIP

show ip ospf Displays general information about OSPF routing processes.

Table 2.1: Status of interfaces and protocols on Cisco IOS and IOS XR network elements.

interface data, and additional information that regards the hierarchy of the OSPF protocol.

This command is used in Privileged EXEC mode, thus navigating from router configuration mode, via global configuration mode, with the command exit. The OSPF configuration has to be performed on all devices in the network for OSPF to exchange LSAs properly. The LSAs in term updates the routing tables of the devices, allowing device apprehension of the topology.

1 Router > enable

2 Router# c o n f i g u r e terminal 3 Router ( c o n f i g )# r o u t e r o s p f 1

4 Router ( co n f i g −r o u t e r )# network 1 9 2 . 1 6 8 . 1 . 0 0 . 0 . 0 . 2 5 5 area 0 5 Router ( co n f i g −r o u t e r )# e x i t

6 Router ( c o n f i g )# e x i t 7

8 Router#show ip o s p f neighbor

9 Neighbor ID Pri State Dead Time Address I n t e r f a c e

10 2 . 2 . 2 . 2 1 FULL/DR 00 : 0 0 : 3 6 1 9 2 . 1 6 8 . 1 . 2 Ethernet0

Listing 2.1: Enable, configure, and verify OSPF on Cisco IOS network element.

2.3 Cisco IOS XR

Cisco IOS XR is part of the Cisco IOS family that is made for building modular, and distributed core routers. Such routers are generally positioned in the core or edge of a backbone service provider network where more stable solutions of process mirroring are required. The modularity reflects into the memory management where IOS XR has integrated memory protection between processes, when processes run in separate memory space. A crashed BGP process will therefore not affect a parallel OSPF process, thus creating a facilitated environment for the multiple process administration that core network of an ISP requires. Additional features supported by IOS XR are for example hardware redundancy, packet based software distribution model, and optional features, allowing multicast routing and Multiprotocol Label Switching (MPLS) to be configured while router still in service, without interrupting the systems scalability.

One of the most essential differences between the IOS and IOS XR software is the execution of input configuration. In IOS, the changes you make to the configuration of the network element are applied immediately, while changes to a device running IOS XR

(18)

have to be committed (using the commit command) before being applied. Syntax can be added to the command to add features, as can be seen in Table 2.2, where common features are listed and described. The commits are stored locally in the device, along with an autogenerated commit ID, and can be displayed when running the command show configuration commit list. The commits in the database serve as rollback points, enabling previous configurations to be activated again with the rollback command.

Syntax Description

best-effort Target configurations and running configurations are merged, and if valid, changes are committed (best effort).

replace Target configurations replaces running configuration.

label labelname Assigns a meaningful label (instead of the autogenerated commit ID).

confirmed time Target configuration replaces running configuration on a trial basis, and rollback if not permanently committed. If not specified, default time is 30 seconds.

clear configuration commits discspace KB

Deletes configurations stored in the commit database, start- ing with the oldest available commit ID. The discspace to free is expressed in kilobytes (KB).

Table 2.2: Feature preferences for the ’commit’ command for Cisco IOS XR network elements.

Even though new commands are not strictly integrated into the IOS XR compared to IOS, the syntax of the commands and Configuration Text Files (CTF), i.e. text files containing the configuration in a language the software can interpret, may differs, as shown in Table2.3, where a simple interior BGP peering is configured between an IOS and IOS XR device. The configuration format is text file, which data can be displayed in the device CLI with the command show running-config, in which the different configuration modes are depicted as indents.

As depicted in Table2.3, the CISO IOS XR configuration start of with the !! characters, enabling the user to leave comments in the CTF, in this case the comment reads the software name and version IOS XR Configuration 5.3.3. The characters enables the text that follow to be ignored by the interpreter. Further, the comment does often include the date the last modification on the CTF was performed, and by whom it was changed. As can be seen on both hand sides of the Table, the configuration files end with the keyword

Cisco IOS (Router ID: 192.168.1.5) Cisco IOS XR (Router ID: 192.168.1.8) router bgp 3402

no synchronization bgp log-neighbor-changes

neighbor 192.168.1.8 remote-as 3402 neighbor 192.168.1.8 update-source Loopback0

no auto-summary end

!! IOS XR Configuration 5.3.3 router bgp 3402

neighbor 192.168.1.5 remote-as 3402

update-source Loopback0

!

! end

Table 2.3: Basic BGP configuration for Cisco IOS routers and Cisco IOS XR routers.

(19)

end. The first line on the left hand side, router bgp 3402, being identical for both devices, described in the OSPF configuration in Section2.2, specifies the configured IP protocol, but differs in that the BGP command is followed by the number of the Autonomous System (AS) for which the device belongs.

The first significant difference between the two configurations is that in IOS XR, features must generally be explicitly enabled, while in IOS the features has to be disabled.

In this configuration example being the no-synchronization and bgp log-neighbor-changes.

If synchronization is not turned off, in case of multiple ASs, will advertise routing information without awaiting the proper convergence provided by BGP. The logging for BGP neighbors is active by default in Cisco IOS XR and is therefor not present on the right hand side of Table2.3.

The similar code snippets in Table2.3that regard the remote-as and update-source is in the Cisco IOS XR configuration configured inside the neighbors submode which facilitates the users reading of the code. The auto-summary command summarize the routes in the local BGP table to their classful boundaries. This feature is non-existing in the Cisco IOS XR version, and therefore not needed to be present in the configuration at all. Finally, the exclamation marks in the Cisco IOS XR configuration in Table2.3 represents the exiting of the submodes.

2.4 Python Network Programming

Python [15] is a general-purpose programming language. Python offers high-level syntax that allows programmers to express concepts in fewer lines of code than possible with other programming languages such as Java. One way to run a Python script is from file. The script is written in arbitrary text editor and saved with a ".py" extension, indication to the OS that the file is a Python program. The syntax of the command and the actual running of the file differs depending on the OS, but most OSs allow for the scripts to be run from the command prompt. The script can also be directly given as input into the interactive Python interpreter, recognized by the three ’greater than’ signs (>>>) that constitute the default prompt.

With regard to network programming, Python offers the Netmiko library, which sim- plifies SSH management of network devices [16], returns a user-friendly output to the developer, enabling the developer to focus on the structure of the system rather than low- level SSH details. Netmiko supports multiple platforms such as a wide range of Cisco platforms, HP and Juniper.

From the Netmiko library, a set of factory functions can be imported, one being the ConnectHandler which selects the correct Netmiko class based upon the device type specified, being a library generic to devices from multiple vendors. Listing2.2shows an example on how to use the Netmiko library to open a SSH connection on the Cisco IOS device and verify the connectivity by sending a simple command to prompt the status of active interfaces on the device, thus verifying connectivity with returned output. As depicted in Listing2.2, the ConnectHandler demands the device_type as an input, which in its generic manner, can be set to a wide range of device types from different vendors, such as Cisco

(20)

IOS, Cisco IOS XR, Juniper, HP and Huawei. The ConnectHandler opens and maintains the SSH communication with the device CLI as depicted in Figure2.2, enabling navigation from script, via the ConnectHandler function, to the device CLI.

Figure 2.2: Netmiko SSH connection to a Cisco IOS device.

1 >>> net_connect = ConnectHandler ( device_type=’ c i s c o _ i o s ’ , ip=’ 1 0 . 1 0 . 1 0 . 2 2 7 ’ , username=’ p y c l a s s ’ , password=’ psw ’ )

2 >>> net_connect . send_command( "show ip i n t b r i e f " )

Listing 2.2: SSH connection, with Netmiko, via Python interpreter.

Moreover, the pyIOSXR library is specialized to facilitate the communication with Cisco IOS XR devices, through the use of an Extensible Markup Language (XML) agent [17]. XML, being a platform and application independent meta language, helps the shar- ing of structured data between the Cisco IOS XR elements and the Python language. The XML agent allows for different approaches of sending commands to devices, enabling commands, as shown in Listing2.3, where the script first opens an SSH connection to device, stores the connection in the device variable (line one), and then enters XML mode for the execution of the additional commands. The commands in Listing2.3first make sure that the communication channel is open (line two), loads a target configuration from a specified path (line three), compares this to the running configurations of the device (line four), and finally commits the new configuration into device (line five), keeping the unique parts, replacing only the configurations that are not alike.

1 d e v i c e = IOSXR( hostname=’ . 1 0 . 1 0 . 1 0 . 2 2 7 ’ , username=’ ∗∗∗ ’ , password= ’ ∗∗∗ ’ , port = ’ 22 ’ , timeout =120)

2 d e v i c e . open ( )

3 d e v i c e . load_candidate_config ( f i l e n a m e= ’ / path / . . ’ ) 4 d e v i c e . compare_replace_config ( )

5 d e v i c e . commit_replace_config ( l a b e l=’ a r b i t r a r y l a b e l ’ )

Listing 2.3: SSH connection, with pyIOSXR, in textfile.py.

As depicted in Figure 2.3, the communication is still channeled via the active SSH connection, to the device CLI, but the XML agent facilitates the system development. Note that the CLI manages the communication with the actual device, as described in section 2.1. For example, the device.commit_replace_config(label=’labelname’) corresponds to the fusion of the commit replace and commit label labelname Cisco command mentioned in Table2.2. In Listing2.3, the IOSXR function corresponds to the ConnectHandler function used in the Netmiko library (see Listing 2.2). The Netmiko and pyIOSXR connection

(21)

Figure 2.3: pyIOSXR SSH connection to a Cisco IOS XR device.

handlers both lock the device under the entire connection, assuring that no configuration changes by another user is merged into the commit.

JavaScript Object notation (JSON) is a compact, text based, language independent, format used for data exchange, which can be written by hand or for example generated from a Java object. JSON enables for structured data sets, similar to XML, and is considered to be easier to use than its mentioned alternative. This consideration is based on that JSON use shorter syntax with a simplified usage of tags, which makes it quicker to comprehend and develop. The most prominent difference of the two hierarchical, human readable formats, regards the parsing, where XML has to be parsed with an XML parser while JSON can be parsed by a standard JavaScript function. In regards of Python, the Json library can be imported, enabling parsing between files in JSON format and a Python dictionary or list. The Reader of this thesis is advised to note the different use of capital letters in JSON as a format and Json as a Python library.

2.5 Related Work on Automating Network Configurations

In 2016 the owner of Twin Briges Technology, Kirk Byers, presented a Python programming example for BGP on Arista EOS, Cisco IOS and IOS XR [18]. Byers used Netmiko, to establish SSH connections to the devices and verify that the connections were setup properly. The devices were stored in Python dictionaries that all corresponded to a network device. Byers then looped through the devices, to check if BGP was configured and then removed the BGP configuration. Byers then loaded BGP commands from a text file, one for each device, and committed these to the devises. He empathized that the commit process for different devises may differ, and so does the syntax of the text files. The building process continued with BGP verification where Byers prompted selected output from the routing command: show ip bgp summary, enabling an inspection of the protocol state. The script that applied all the above commands consisted of 93 lines of code and required about 46 seconds to execute Cisco IOS XR devices.

Byers solution, serves as a great example of what can be created when fusing high- level languages with low-level hardware configuration. Through this automated process, reducing tedious work in form of repetitive protocol configuration. If developed and made generic, not vendor specific, the programming example could serve as a fully automated

(22)

Figure 2.4: Client server communication to obtain network configuration via DHCP.

BGP configuration system used in production, by for example telecommunication stakeholders.

2.5.1 Dynamic Host Configuration Protocol

Dynamic Host Configuration Protocol (DHCP) is a network protocol used to provide network configurations to network elements by allocating IP addressed in an automated fashion. Consequently, IP addresses need not be manually configured on network elements such as work stations, printers, game consoles, and personal computers.

The automatically distributed IP addresses are chosen from a pool of addresses, set by the server administrator and assigned to the clients via the DHCP session depicted in Figure2.4. The client announces its presence in the network by broadcasting a Discovery message. The server sends back an Offer of an IP address and the client responds by sending a Request to keep the address. The address is taken in use by the client first when receiving the Acknowledge message sent by the server, guaranteeing that no other device in the network has claimed the address. This process is performed at least once for each device connecting to the network.

The purpose of the protocol is to convey all configuration information needed, for a computer or network device, to access the network. This automated host configuration provides reliable IP address configuration, thus minimizing the occurrence of configuration errors such as typographical, and address conflict occurring by multiple assignment of an IP address to more than one device in the network.

However, the information provided by DHCP is IP addresses, subnet mask, default gateway, and primary as well as secondary Domain Name System (DNS) server. The only information that is going to be given dynamically is the IP address; the remaining data still has to be manually configured on the DHCP servers. If the manual configuration contains faulty data, distributed to all the network elements, the devices will not be able to communicate. For example if wrong IP address is given to a host by the DHCP server, the host will not able to resolve DNS names because of using incorrect DNS servers, thus the host will not be able to reach the correct IP addresses.

Due to a centralized and automated TCP and IP configuration, and the efficient handling of address changes for mobile devices, such as portable personal computers, DHCP

(23)

reduces network administration. This dynamic protocol is yet another example of automated network configuration that reduces manual management and errors by undertaking the tedious work of statically assigning IP addresses.

2.5.2 Network Configuration Protocol

In the early 21st century, it became clear that the Simple Network Management Protocol (SNMP) [19], was used for monitoring networks instead of, as originally intended, to manage them [20]. The Internet Architecture Board and Members of the Internet Engineering Task Force (IETF) had a meeting to discuss the issue in fact, and found that the management was still performed via CLI scripting, which was, as mentioned in Section2.1, limited in structured error management, transaction handling and prone to changeable syntax of commands. The Network Configuration Protocol (NETCONF) [21], was developed to address the shortcomings of existing approaches, providing different mechanisms to simplify the installation, manipulation and deletion of configuration on network devices, compared to SNMP.

NETCONF uses Remote Procedure Calls (RPC) to allow a network administrator to exchange network configuration information with the devices. The RPC based communication model works in a request-reply manner using XML, providing the NETCONF protocol with transport protocol independent framing, using the <rpc> and <rpc-reply>

elements for communication between the client and server. Listing2.4shows a simple example of a NETCONF request to a invoke the <get> method, used to receive device state information and running configuration from device. The <rpc> element element encloses the method and allocates a mandatory message ID for the session to keep track of the messages sent. The server will respond with a <rpc-reply> using the same message ID.

1 <rpc message−id="101"

2 xmlns=" u r n : i e t f : p a r a m s : x m l : n s : n e t c o n f : b a s e : 1 . 0 ">

3 <get />

4 </ rpc>

Listing 2.4: Simple NETCONF YANG example

The transactions are a set of, non order specific, commands that facilitate network wide configurations of multiple network elements. The transactions are facilitated by the modeling language Yet Another Next Generation (YANG), specially developed for this purpose. Standard NETCONF-YANG syntax is shown in Listing2.4, but since YANG is a protocol independent language, it can be converted into any encoding format e.g. XML or JSON.

Listing 2.5 depicts an OSPF configuration alike the one described in Section2.2, but here represented in XML, already inside the OSPF branch. Here, the interfaces are places in the OSPF area, similar to the Cisco IOS XR approach discussed in Section2.3, rather than the OSPF being configured on the interfaces, as for the Cisco IOS structure.

1 <o s p f xmlns=" h t t p : // example . org / netconf / o s p f ">

2 <area>

3 <name>0 . 0 . 0 . 0</name>

(24)

4 <i n t e r f a c e>

5 <name>eth0</name>

6 <p r i o r i t y>30</ p r i o r i t y>

7 <metric>100</ metric>

8 <dead−i n t e r v a l>120</dead−i n t e r v a l>

9 </ i n t e r f a c e>

10 </ area>

11 </ o s p f>

Listing 2.5: Simple NETCONF YANG example: configure OSPF

The transactions enable error handling of incomplete configurations that the Cisco specific, non transaction approach does not offer. In a Cisco-specific approach, such as the OSPF configuration described in Section2.2, an incomplete configuration, for example the one of only entering the first three rows of the configuration in Listing 2.1, will not be defined as an error, but rather the device will wait for additional commands to complete the configuration. In a transaction based communication, the insufficient configuration will be defined as an error and send feedback to the user that eh protocol is not properly configured. Thus a transaction based platform will enable the network manager to focus on the management of the services in the network, rather than the management of device configurations. Provided that a Cisco device supports the NETCONF protocol, a Cisco-specific approach and NETCONF approach can be used together, as long as the configurations are not contradicting.

2.5.3 Software-Defined Networking

The configurations presented in Section 2.2 and 2.3 has both been been vendor-specific, communicating with some sort of agent or controller of a single device that in turn pushed configurations to the lower level of the device, enabling protocols to converge and routing paths to form. The network configurations has been performed in a device-based fashion, where each device has to be singly and manually configured, something that, as mentioned in Section1.2, simply wont scale.

Software-Defined Networking (SDN) is a modern approach of manage and manipulate networks, using software, by abstracting the data and control plane of network elements.

The data plane, or forwarding plane, performs the forwarding according to the control plane logic contained in the Forwarding Information Base (FIB). The control plane is the logic brain of the device, and is in charge of the forwarding decisions, and controlling the forwarding behaviour e.g., the routing protocols, contained in the Routing Information Base (RIB). By separating the two planes, each can be evolved and deployed, independent of each other.

In SDN, the control planes of all network elements are managed by a logically centralized SDN controller, enabling configurations of large area networks to be pushed to the network elements in order to provision for example new network services. In Figure2.5the abstractions of the two planes are depicted as the data plane being the switches and the control plane being the routers. The SDN controller communicates as an application with all the devices, thus creating a management plane where all the devices can be controlled

(25)

Figure 2.5: Layer abstraction in a Software-Defined Network.

by by the centralized SDN controller. As indicated by the arrows, the centralized controller pushes configurations in a top-down approach, enabling the SDN controller to to reach all the planes and control everything from routing behaviour, to low level packet processing.

This centralized management provides the controller with visibility of the entire network, which can be otherwise hard to get in for example, large scale area networks.

There is software standards used to handle the management of the abstracted planes, for example NETCONF, discussed in Section2.5.2, and OpenFlow [22], which allows for special OpenFlow Switches to be controlled or updated centrally, thus shaping a SDN.

McKeown et al. in [23] defines an OpenFlow Switch as consisting of at least three components: "(1) A Flow Table, with an action associated with each flow entry, to tell the switch how to process the flow, (2) A Secure Channel that connects the switch to a remote control process (called the controller), allowing commands and packets to be sent between a controller and the switch using (3) The OpenFlow Protocol, which provides an open and standard way for a controller to communicate with a switch." McKeown emphasizes that by specifying the protocol where entries in the Flow Table can be manipulated and defined exactly, the switch itself does not need to be programmed and configured, this is managed by the protocol, switching focus from device-based programming to centralized control.

Further, McKeown [23] draws the conclusion that: "OpenFlow is a pragmatic compro- mise that allows researchers to run experiments on heterogeneous switches and routers in a uniform way, without the need for vendors to expose the internal workings of their products, or researchers to write vendor-specific control software". This statement not only imply the facilitation of network configurations on a, to the outside world, uniform network, but also the importance of the quest for uniform network configurations despite vendor-specific network elements.

2.6 Software Development Approaches

To implement Enna, several methods were studies and compared according to [24]: (i) the traditional Waterfall development, (ii) the agile Scrum development and (iii) the iterative Waterfall development, which is considered a crossing between the two.

All the methods mentioned in this Section consist of one or more iterations, all consisting of different phases, as depicted in Figure2.6: The planning of the work, building of the system, testing the system, reviewing the system, and finally, deploying the system.

(26)

Figure 2.6: The five phases of a system development approach.

In the traditional approach, i.e. the Waterfall method, the system development follows the phases depicted in Figure 2.6, each phase is approached one by one, completing the phase before a new one is tackled. The Waterfall approach, when completed, is not meant to be revisited. The Waterfall system development therefore lean on the phases being fully elaborated, and the system specifics obtained, already in the planning phase. All phases in the Waterfall approach are assigned a time interval given in the planning phase, leading to time shortage being a potential problem, and the whole system development can fail due to inadequate planning.

In Scrum development, the phases of system development are depicted in Figure 2.6:

plan, build, test, review, and deploy are repeated for each feature of the system. This agile approach is less risky than the Waterfall development due to the fact that each system feature is implemented and tested in isolation and thereby the chances of producing functional features are increased. After all features are completed, an integration phase is invoked to produce a fully functional system, with all features in place.

Using the Scrum approach, lack of features does not imply a non functional system, but rather a working system with fewer capabilities. Each feature is properly tested and workable before deployment, and the Scrum method can therefore be considered an efficient approach to system development.

Finally, the Iterative Waterfall Approach could be considered a crossing between the Waterfall and the Scrum methods, where the five phases are iterated an additional time, composing a traditional yet agile approach to system development. In this approach, the Waterfall method depicted in Figure2.6is extended with a feedback loop to allow for the next iteration of phases to be established with additional information.

(27)

Chapter 3

Automatic Network

(Re)Configuration with Enna

This Chapter provides understanding of the work performed regarding the tool for automatic network configuration and verification. Section 3.1 contains a step-by-step description of the way in solving the problems, mentioned in Section 1.2, for automatic network configuration and verification. Section2.6contains the system-related content of the approach, i.e. the thesis methodology. Chapter4 contains the detailed system implementation based on the steps described in Section 3.1. Finally, Section 4.5 contains the tools used in the implementation of the proposed system.

In the rest of this thesis, the terms network elements, devices and routers are used interchangeably, since all the devices that comprise the testbed are routers. All devices used in the rest of this thesis are virtual machines running Cisco IOS XR, described in Section2.3, provided by Com Hem.

3.1 Approach

As mentioned in Section 1.2, some companies does not have automated mechanisms for re-configuring the lab-environment to basic functioning conditions. To tackle this problem the automated tool, Enna, has been developed.

To gain a deeper understanding of the subject of automation, and its accompanied components, a quantitative study of the topics covered in Chapter2 was performed. The topics where chosen after detailed discussions with the thesis projects stakeholders. The different approaches and tools used to achieve automation were studied and defined with the delimitations mentioned in Section 1.5, which as mentioned in that Section, were collected via verbal as well as written communication in the form of meetings with the thesis stakeholders.

To facilitate the development, maintenance and troubleshooting of Enna, the system was modeled to consist of three modules. As depicted in Figure 3.1, the modules are as follows: (1) the Read module, which takes a network state and its devices’s CTFs as inputs, (2), the Apply module, which applies the new configuration to the devices, thus forming a preliminary network state, and (3) the Verification module, which performs tests

(28)

Figure 3.1: System Overview 1) Read input

2) Apply new configurations on initial network state, thus forming a preliminary network state

3) Preliminary network state is verified

3a) If correct, use preliminary network state as output.

3b) If validation fails: return to modify input.

on the devices in preliminary network state, verifying whether this preliminary state could be considered a functional network state, i.e. fulfilling the requirements of the network operator as described in Section1.1.

For the modules to be coherent and functional as a whole, they where developed in parallel, with implementation focus to fulfill the system goals described in Section1.3. The goal of easy implementation applies to all modules, the generation of accurate network configuration applies especially to the Apply module, and the goal of the system not demanding extended knowledge from the user regarding network configuration applies in particular to the Verification module and its output.

3.1.1 Enna for a Single Device

In the process of developing Enna, the first step was to approach one single device via a remote shell as discussed in Section2.1. The simple command of configuring an interface and display all active interfaces, as described in Section2.3, was manually sent to the device to verify connectivity, as well as inspecting the modifications due to the input commands.

To ease the administration of the multiple commands pushed to the device, a script, i.e. the first draft of Enna’s Read module, used for storage and iteration of the commands, i.e. the configuration, was created and run locally on the personal computer used for the system development. Once the remote access and modification correctness were established, the running configuration of the device was manually saved into a text file, and stored on a

(29)

local server to provide easy access. This text file containing the running configuration of the single device topology is the first CTF, i.e. the new configuration, used as input to the Read module, as depicted in Figure3.1.

Once the correctness of the CTF is established, the process of integrating it back into the device was commenced, i.e. to initiate the Apply module and form the preliminary network state depicted in Figure3.1. To assert the correctness of the configuration process, the CTF was repeatedly loaded into the router as the device was configured with different network states: (i) device manually configured with the global interface only, i.e. a relatively empty device, configured only with the public interface enabling remote access, (ii), identical interface configuration as the first device, but here configured with the CTF, and (iii), interface configuration differing from the configuration provided by the CTF, configured with a superfluous interface. This was done to ensure that (i) the CTF was properly loaded into the router, overwriting manual configuration (ii) the configuration provided by the CTF and the running configuration of the device would not clash and (iii) the configuration provided by the CTF was replacing the running configuration and not simply merging.

The verification of this preliminary network state was performed manually to facilitate command modification. Note that the devices configured with identical network state as provided by the CTF fulfills the requirements of a steady state before (re)configuration, and the configuration with superfluous configuration does not.

After loading the CTF into the device with different network states, and establishing the correctness of the configuration using the verification commands, the manual verification of the one device topology was integrated into the script, forming the first version of the Verification module depicted in Figure 3.1. In this step, the verification module was composed of verification that (i) the CTF was properly committed, and (ii) generated output of the configuration (and therefore the configuration itself) was consistent.

To provide Enna with resilience against errors and exceptions due to memory shortage of the device commit database, a cleaning process was integrated into the Read module, ensuring that former device configuration would not persist after the new configuration is injected. The cleaning process also facilitates the implementation of the Apply module for it provides the module with devices with empty commit databases, only containing running configuration.

3.1.2 Enna for Multiple Devices

The second step in approaching the development of Enna was to apply the knowledge of the one device case, implemented in previous step, and modifying the script into a more generic approach handling multiple devices.

Therefore, the Read module was modified to handle multiple device names, as well as the reading of multiple CTF locations, allowing the different CTF of the topology to be used as inputs. A menu was implemented to provide a user interface displaying the network elements, as well as enabling selection of what devices to send to the Apply module. The user interface allows the selections of one or more devices to be configured. The user interface was further modified to contain an option for terminating Enna. Lastly, for Enna

(30)

to conquer user input errors, Enna was modified with exception and error handling to manage such errors.

After the Read module, the selected devices (i.e., only those sent to the Read module) pass through a cleaning process that resets their current configuration, before the new configuration is applied.

The Apply module was modified to handle multiple devices from the Read module, based on user selection in the menu discussed above. The reconfiguration of the devices was extended by modifying the Apply module to push configurations to multiple devices, allowing any topology to form iteratively.

The Verification module was modified to handle multiple devices as well. The verification process was extended to a more generic interface verification, making use of the device information provided by the Read module. Further, the output of the verification process was refined, in accordance with the goal of not demanding extensive knowledge of network configurations from the user, as discussed above.

In this second step of the approach, error and exception handling was integrated into Enna, wrapping the three modules in separate exception handlers, thus providing user feedback. The exception handling was designed to clearly display the error, and the module where the error occurred,

3.1.3 Enna for Com Hem

The third step was the refinement of the Enna modules to the Company preferences.

First, the number of devices that comprise the Company’s lab environment (see Figure3.2 with germane Table3.1) was defined, and the device configurations where set accordingly.

The machines can be accessed remotely via their global interface Gi 0/0/0/0 or reached via their inline management MgmtEth0 from a jumphost, i.e. a special purpose secure administrative host. As for this thesis, Enna is to be deployed onto the jumphost, managing its connections from within the security zone the jumphost creates. The steady state was manually configured to provide documentation for later testing, the CTFs were manually created, and the CTFs storage preferences were decided together with Com Hem.

The user interface was refined in accordance with the number of devices, allowing for a customized user interface tailored to Com Hem. The menu, already implemented to hold any number of devices, was further implemented to display a user option for sending the devices to the Verification module, thus verifying the preliminary topology created by the Apply module, as depicted in Figure 3.1, and providing the user with feedback from the verification process.

The Read module was further refined by separating device information and Enna source code. This approach of not containing device information in source code enables user groups without access to specific device information to work with system administration. This separation also facilitates the user overview of device information, as well as allows for secure storage of the configuration file.

In this third step of the approach, the Apply module was modified to verify that the global interface was functioning, ensuring that all devices in the preliminary network state

(31)

Figure 3.2: Lab network topology.

will be reachable by the Verification module. This process of only sending devices with a functioning global interface to the Verification module also ensures that the verification process can take place without errors, due to a misconfiguration of global IP, will occur.

In previous steps (i.e., Section 3.1.1 and Section 3.1.2), the Verification module has solely consisted of interface verification. In this third step of the approach however, the Verification module is modified to verify the configured protocols as well. The verification is modified to inspect the concerned protocols (e.g., routing protocols) in the provided topology, but in a manner that is not protocol specific. This ensures that the forwarding of the protocols are properly established and therefore that the topology is properly converged.

The only manual process in Enna is the initial configurations of the preliminary state that has to be manually configured. The configurations are manually loaded into a text file, i.e. the CTFs, which are then used to configure the device managed by Enna. The creation of CTFs is a crucial step for Enna since the CTFs ensure the correctness of the topology configured by Enna, and this manual approach is considered a subject of fragility for an automated tool, as discussed in Section 1.2. The process of creating the CTFs is however crucial in this early stage of Enna, since their existence is critical for configuration, and the creation of a tailored topology can only be created either by manually writing the CTFs or create them from the running configuration of an existing manually configured topology.

3.1.4 Enna ’s Aspects

Since the CTFs contains the network topology, the CTFs need to be stored securely, pre- venting malicious attacks against the network simplified by the information provided in the CTFs. By storing the CTFs locally and geographically close to the devices, the CTFs can rely on the security already provided by the local network. This local stratification requires storage space to be distributed in the network, the capacity of which depends

(32)

upon the number of device, as well as configuration complexity. In simple networks, for example the ones approached in this thesis, the storage space required is just a couple of KBs, but in larger and more complex networks the space required to store the CTFs would increase. Since the CTFs are rather small to begin with, this weakness of Enna’s need for allocating storage space is not considered to be prominent.

Note that since the SSH connections run on the global interfaces of the devices, a misconfiguration of the global IP would make it unfeasible for Enna to connect to the device. This would force the system administrator to connect to the device using other protocols, such as Telnet as discussed in Section2.1, and manually (re)configure the global IP address. This approach is not preferred due to the Telnet protocols insufficient security mechanism, discussed in Section2.1, but required to establish remote access.

The network configuration provided by Enna is vendor-specific Cisco IOS XR in order to meet the goal of easy system implementation described in Section1.3, and to suit to the Company demands, as discussed in Section1.5. The dependence on Cisco is a configuration to ease the implementation, and not a requirement for Enna to properly function. A more generic implementation would make Enna applicable to additional vendors, similar to Byers’s solution discussed in Section 2.5, which apart from Cisco IOS XR, also address devices running Arista EOS and Cisco IOS. Enna’s vendor-specificity lies in the usage of its modules described in Section 2.4. As discussed in Section 1.5, one of the modules facilitates the development of Enna, targeting only devices running Cisco IOS XR. The module is also preferred by the Company due to existing knowledge that facilitates future maintenance of Enna. The vendor-specificity is not considered a weakness, but rather a strength under these predominated circumstances.

Apart from providing an automated approach to vendor-specific network configuration, Enna’s strength lies in the verification of the configured network. As mentioned in Section 1.5, the ultimate challenge when building an automated tool, is the verification rather than the exception and error handling. As discussed in this Section, the Verification module is a crucial building block of Enna, verifying the preliminary network state created by the Apply module, ensuring that systematic verification is in place.

(33)

HostnameInterfacesAbbreviationAddressesProtocols lab-xrv-1 Loopback0 ManagementEthernet0 GigabitEthernet0/0/0/0 GigabitEthernet0/0/0/1 GigabitEthernet0/0/0/2 Lo0 MgmtEth0 Gi0/0/0/0 Gi0/0/0/1 Gi0/0/0/2 1.1.1.1 172.16.0.66/24 194.71.108.66 10.0.0.0 10.0.0.2

BGP,IS-IS(passive) MPLS,IS-IS MPLS,IS-IS lab-xrv-2

Loopback0 ManagementEthernet0 GigabitEthernet0/0/0/0 GigabitEthernet0/0/0/1 GigabitEthernet0/0/0/2 GigabitEthernet0/0/0/3 Lo0 MgmtEth0 Gi0/0/0/0 Gi0/0/0/1 Gi0/0/0/2 Gi0/0/0/3 2.2.2.2 172.16.0.67/24 194.71.108.67 10.0.0.1 10.0.0.10 10.0.0.4

IS-IS(passive) MPLS,IS-IS MPLS,IS-IS MPLS,IS-IS lab-xrv-3

Loopback0 ManagementEthernet0 GigabitEthernet0/0/0/0 GigabitEthernet0/0/0/1 GigabitEthernet0/0/0/2 GigabitEthernet0/0/0/3 Lo0 MgmtEth0 Gi0/0/0/0 Gi0/0/0/1 Gi0/0/0/2 Gi0/0/0/3 3.3.3.3 172.16.0.68/24 194.71.108.68 10.0.0.6 10.0.0.16 10.0.0.3

IS-IS(passive) MPLS,IS-IS MPLS,IS-IS MPLS,IS-IS lab-xrv-4

Loopback0 ManagementEthernet GigabitEthernet0/0/0/0 GigabitEthernet0/0/0/1 GigabitEthernet0/0/0/2 Lo0 MgmtEth0 Gi0/0/0/0 Gi0/0/0/1 Gi0/0/0/2 4.4.4.4 172.16.0.69/24 194.71.108.69 10.0.0.5 10.0.0.8

BGP,IS-IS(passive) MPLS,IS-IS MPLS,IS-IS lab-xrv-5

Loopback0 ManagementEthernet0 GigabitEthernet0/0/0/0 GigabitEthernet0/0/0/1 GigabitEthernet0/0/0/2 Lo0 MgmtEth0 Gi0/0/0/0 Gi0/0/0/1 Gi0/0/0/2 5.5.5.5 172.16.0.70/24 194.71.108.70 10.0.0.9 10.0.0.7 BGP,IS-IS(passive) MPLS,IS-IS MPLS,IS-IS Table3.1:Labnetworktopologyinterfacespecification.

Automating Network System Configurations for Vendor-Specific Network Elements