Integration och realtidsprestanda av ROS 2.0 i ett inbyggtoch distributerat system

(1)

Integration och realtidsprestanda av ROS 2.0 i ett inbyggt

och distributerat system

Integration and real-time performance of ROS 2.0 in an

embedded and distributed system

Författare: Anton Söderlund

Vårterminen 2021

Examensarbete för civilingenjörsexamen, 30 hp Huvudområde: Datateknik

Civilingenjörsprogrammet i datateknik

Institutionen för naturvetenskap och teknik, Örebro universitet. Handledare: Henrik Andreasson, Universitetslektor, Örebro universitet

(2)

Sammanfattning

Gruvindustrin är idag en av de mest farliga jobb som nns, och den snabbt utvecklande tekniken inom självstyrande fordon har bevisat sig tillföra båda en säkrare och produktivare miljö för hållbar utveckling. Med autonoma system följer robotikmjukvara, och en väl använd metod är att utveckla middlewares för att standardisera och fortgöra utvecklingen av robotapplikationer. En av de mest använda i forskning idag är ROS, och har nyligen släppt versionen ROS 2.0 vilket satsar på att vara mer tillmötesgående för industriapplikationer. Den här uppsatsen föreslår en lösning på en integration mellan ROS 2.0 och ett existerande styrsystem genom att utveckla ett gränssnitt mellan de olika systemen. Det integrerade systemet testas på relevant hårdvara och utvärderas baserat på de realtidskrav som redan nns i det ursprungliga systemet. Re-sultaten visar ett linjärt samband mellan den mängd meddelandesöverföringsmekanismer som används i systemet och latens, CPU-användning och minnesanvändning. Systemet uppfyllde de realtidskrav som är sätta efter hårdvaran med ett mindre antal signaler.

Abstract

Mining is one of the most hazardous industries today, and with the advancements of autonomous vehicles this has proven to be a safe and productive approach to ensure a sustainable future. With autonomous vehicles comes robotic software, and a common approach is to develop so called middleware to speed up and standardize robotic applications development. One of the most popular middlewares in research is ROS, has recently released ROS 2.0, an improved framework from its predecessor that is tailored towards a professional environment. This thesis presents a proposed solution to integrating a native control system and ROS 2.0 by developing an interface that facilitates the communication between the two systems. The system is tested and evaluated on relevant hardware and evaluated in terms of the real-time requirements already present in the vehicles. The results showed a linear increase in latency, CPU usage and memory usage in respect to the amount of message-passing mechanisms that were added to the system, but fullled the real-time requirements at a lower amount of signals.

(3)

Acknowledgements

A sincere thanks to my supervisors Håkan Lundvall at Epiroc and Henrik Andreasson at Örebro University for their help and guidance during this project. I also thank department manager Ola B. Petterson and lead architect Kristofer Hansson for their valuable feedback and welcoming introduction I received at Epiroc. Lastly, I would like to thank the entirety of the Core team at Automation-Platform Development for making me feel included in the group and their positive environment.

(4)

List of Figures

2.1 A Comparison between ROS 2.0, ROS 1.0 extensions and other middleware by

Maruyama et.al. [17] . . . 10

2.2 This image shows two examples of priority inversion and priority inheritance. In priority inversion the lowest task locks a resource required by the highest priority task, blocking it and resulting in it nishing last. With inheritance, it is elevated such as the highest priority can start as soon as possible. . . 12

2.3 CAN Bus Architecture with n nodes . . . 13

2.4 High Level RCS Architecture, three example tasks . . . 14

2.5 I/O Finite State Machine . . . 14

3.1 Generalized Proposed system . . . 16

3.2 H6072 Display . . . 19

3.3 I/O T2 Module . . . 20

3.4 I/O T2 State Diagram . . . 20

4.1 Testing setup for PS=n . . . 22

4.2 CPU Usage . . . 22

4.3 Error plot of min, max, mean and standard deviation with PS=1 . . . 23

4.7 CPU Usage . . . 25

4.8 Memory Usage . . . 26

4.9 Testing hardware setup and software stack . . . 26

4.10 Testing Architecture . . . 27

4.11 Bar plot of 2000 worst case latencies . . . 28

(6)

List of Tables

4.1 4 Byte message size . . . 23

(7)

Glossary

Middleware - An abstraction layer that resides between the operating system and software applications

API - Application Programming Interface ROS - Robot Operating System

QoS - Quality of Service

CAN - Controller Area Network ECU - Electronic Control Unit PWM - Pulse Width Modulator RCS - Rig Control System

RCSOS - Rig Control System Operating System DDS - Data Distribution Service

RTPS - Real-Time Publish Subscribe RDK - RCS Development Kit SDK - Software Development Kit

(8)

1. Introduction

Robotic applications in industry has been used since the 1960s, and with the recent advances in autonomous vehicle technology their importance has only grown. Mining is both an impor-tant and global industry, and as automation progresses the desire of autonomous or teleoperated control increases. Generally regarded as one of the most hazardous jobs globally, autonomous vehicles have shown to be both an increase in safety and productivity.

Dierent types of robots often dier in both hardware and functionality, making managing a software stack from driver-level to application level a time-consuming task. A common approach of robotic development is to create and build on top of a robot middleware, an abstraction layer that resides between the operating system and software application development. Dierent middlewares are designed for dierent purposes, but their goal is often the same, faster and stan-dardized development of robot software applications. One of the most popular middleware in research is ROS[1], and with its new release of ROS 2.0 it aims to gain more traction in industry. This thesis aims to present an integration of ROS 2.0 into an existing embedded platform and evaluate how the new communication protocols performs in a real-time critical system.

This thesis is conducted in cooperation with Epiroc Rock Drills AB, with its main goal of inte-grating ROS 2.0 into their native platform Rig Control System (RCS) and evaluate its viability and performance. Epiroc is mainly a manufacturer of mining machines such as drill rigs, loaders and trucks. Due to the inherent safety requirements of the robotic industry, determinism regard-ing the sent and received messages is highly important.

A successful implementation of ROS 2.0 architecture into a native control system platform would be interesting to measure the performance and ease of integration as it may help move robotic industries more towards a common open-source platform.

1.1 Research Questions

This thesis aims to answer the following questions: 1. How can ROS 2.0 be integrated into RCS?

2. What is the minimal software component for sensor data reading and actuation using ROS 2.0?

3. Does ROS 2.0 fulll the real-time requirements of the system?

1.2 Requirement Gathering

Because of the exploratory nature of this thesis, the requirements of the project was developed incrementally across the timeline. It was not clear if the ROS 2.0 framework would be compatible

(9)

with hardware and the dierent operating systems. Much time was spent on the attempt to t the ROS 2.0 Foxy Fitzroy distribution into older operating systems that uses x86 32-bit archi-tecture, and while it may still be possible the decision to avoid it was made due to the amount of time it would require to sort, reacquire and sometimes recompile dependencies. In hindsight, this decision probably should have been made earlier to allow more time for evaluation testing. The requirements regarding a distributed environment and separated from user space arise throughout the project from discussions with supervisors. To be in any regard viable to be deployed in a product, ROS 2.0 network control must be functioning to allow dierent machines to communicate via the DDS implementation, and user space separation is highly desirable to supply a development process of applications without having to modify the interface. The real-time requirements however were set by the existing hardware specications and what current applications are constrained to, and the evaluation were conducted in respect to those require-ments.

These requirements were developed during the project after gaining an understanding of the existing system and deciding on an appropriate scope.

ROS 2.0 shall be deployed on Epiroc embedded computers. An interface between ROS 2.0 and RCS shall be established. The interface shall be functional in an distributed environment. The interface shall be separated from ROS 2.0 application user land.

Evaluation of the real-time performance of ROS 2.0 in both local and distributed environ-ments in relation to the current real-time constraints.

(10)

2. Background

This section will provide an background to the existing system architecture in Epiroc vehicles, robotics middleware in general and the historical use of ROS.

2.1 ROS and ROS 2.0

The Robot Operating System (ROS) project was initially created in 2007 and given contrary to its name, it is not an actual operating system, but a collection of software frameworks and set of tools designed to provide services for a heterogeneous computer cluster, and often classied as a middleware[2][3]. ROS functionalities includes hardware abstraction, low-level device control, inter-processes message-passing and package management[4, p.6]. Distribution and modularity is at the core of the design philosophy where ROS processes are built as a graph architecture, where the processes are referred to as nodes. While ROS has become the de facto standard in robotics research, it has yet to establish itself in the same size in industry for two main reasons, lack of support in real-time systems and embedded systems[5].

Message passing can be created using three dierent communication patterns: publisher-subscriber, service-client or action-client[6]. Publisher-subscribers intended use is for continuous data ow such as sensor data, in which subscribers act through callback function implementations once the publisher has decided the message is fully sent. Service-client are intended as remote proce-dure calls initiated by the client. This implementation is blocking and recommendation for good practice is short and ecient queries. Action-client is similar to services, but preemptive and provides feedback in intervals back to the client and is more suitable for longer tasks.

ROS uses a custom built transport layer using TCP/IP sockets called TCPROS[7]. Message passing between nodes is dependent on a centralized network conguration, which is realized by a ROS master which has to exist as a running process before any nodes can be executed. The master is responsible for providing naming and registration services to every component in the ROS graph structure, which includes nodes, topics, services, actions and the parameter server. This means that components have no necessity to known anything about each other as long as they have communication with the master[8]. This centralized architecture and TCPROS communication has shown to not be robust in distributed systems and real-time critical sys-tems[9][4]. These design decisions were made for a smaller set of use cases than ROS is used for today, and originally as stated by author Brian Gerkey for the Willow Garage PR2 robot. It was then designed for being used for a single robot with high computational resources available, and expecting no real-time critical requirements or poor connectivity environments[10]. Solutions to these problems have been proposed such as Chen et.al. implementing a real-time architecture called RT-ROS[11] and Tiderko et.al. providing a ROS multimaster extension for multi-robot systems[12].

To address these limitations, ROS 2.0 is a redesigned framework in which embedded systems, real-time systems and distributed systems are thought of from the beginning while still keeping

(11)

backwards compatibility of already developed ROS packages. The most important change is the implementation of Object Management Group's DDS standard which is both decentralized, and designed to be compatible with real-time critical systems[13]. DDS in ROS 2.0 is made available through an abstract middleware layer (RMW)[14] in which multiple DDS implementations can be chosen. Currently, ROS 2.0 Foxy Fitzroy released in June, 2020 has full support for eProsima's FastRTPS, Eclipse CycloneDDS and RTI Connext[15]. The message passing patterns remains the same as ROS 1.0 through publish-subscribe, actions and services, although improvements has been made such as message denition standardization, separation of namespaces and message compatibility between C++ and Python. Additionally, ROS 2.0 API provides functionality for setting QoS proles compatible with the DDS standard[16].

A study comparing the functionalities of ROS variations compared to ROS 2.0 by Maruyama et.al. demonstrates the motivation to create an encompassing framework that supports all the necessary functionalities, instead of continuously building on top of a system which was not originally designed for this purpose, see Figure 2.1.

Figure 2.1: A Comparison between ROS 2.0, ROS 1.0 extensions and other middleware by Maruyama et.al. [17]

Another notably addition to ROS 2.0 is the support of intra-process communication between nodes[18]. This means that multiple publishers and subscribers can be executed from the same binary resulting in nodes passing messages through shared in-process memory. Intra-process communication is not to be confused with inter-process communication, the process in which messages are sent between separate processes through the RMW middleware layer and DDS. Intra-process communication is limited to nodes running on the same machine, but can remove the steps of translation from ROS messages to DDS samples and the serialization steps that is inherent to many of the RMW implementations. Avoiding these extra steps may end in lower latency message passing which can be very important for real-time critical systems.

2.1.1 Why ROS and alternatives?

Choosing an appropriate middleware for a robotic use case is not a trivial question, and often the answer is that it depends. This section will highlight some of the advantages of picking ROS as a middleware, and some use cases where other frameworks may be more suitable.

ROS has one of the largest communities in the robotics middleware sphere, and it has had con-tinuous growth since its release as demonstrated by their yearly metrics reports. The 2020 report comparing the user base to 2019 saw a 33% increase in users at the discourse board in which members discuss the future development of ROS, a 8% increase in users visiting the wiki and a 19% increase in users at the QA forum [1]. The growth and user base is reected in the vast

(12)

amount of community built packages, arguably one of the greater strengths of using ROS today, and since ROS 2.0 provides cross usage of package via ros1-bridge[19], nothing changes in this aspect when moving to ROS 2.0.

ROS is open-source, and heavily promotes code reuse by design, which is valuable in any larger project. It is also complete, in a sense that it supports functionalities for many steps in devel-opment from low-level device control to simulation via the Gazebo physics engine, visualization through tools like RVIZ and more. This however requires any project using ROS to extensively integrate it into the platform to reap the benets, but for any project that are only looking for specic functionalities, ROS may result in unnecessary overhead, such as the necessary build systems ROS requires to keep package dependencies in order.

For example, if a project is in need of an message-passing mechanism and nothing else, they may instead consider gRPC[20] that fullls this goal through remote procedure calls, or ZeroMQ[21] that supports multiple dierent messaging patterns. If the interest instead lies on visualization, one might consider OpenRAVE[22] instead.

2.2 RCS

RCS is the platform in which all systems and applications exist upon in Epirocs products.

2.2.1 RCSOS

RCSOS is a custom built real time embedded ubuntu-based operating system developed by Epiroc targeted specically for being deployed in their products. Multiple releases exists, but they all use the RT_PREEMPT Linux kernel for x86 or ARM architecture. This allows almost the entire kernel to be preemptive by replacing spin locks with mutexes that supports priority inheritance[23]. This allows user dened priorities to be set on tasks, in which lower tasks are preempted (i.e. yields) to higher priority tasks. Preemption however introduces the priority inversion problem, in which lower priority tasks yields to a higher priority task, while still owning a lock to a shared resource. This problem is solved by the priority inheritance protocol, which elevates the priority of tasks that has locked some resource, and is restored when the resource is freed. As seen in gure 2.2, priority inversion may result in the highest priority task being completed last, and in some cases may result in a so called dead-lock, in which a circular wait between tasks occur and neither task is able to nish. Priority inversion is not inherent to RCSOS, but a problem that should always be taken into account when considering preemption mechanisms.

(13)

Figure 2.2: This image shows two examples of priority inversion and priority inheritance. In priority inversion the lowest task locks a resource required by the highest priority task, blocking it and resulting in it nishing last. With inheritance, it is elevated such as the highest priority can start as soon as possible.

2.2.2 CAN

Epiroc's products relevant to this project uses the Controller Area Network (CAN) protocol as the main communication channels for I/O devices. It allows microcontrollers and ECUs to com-municate in a message-based protocol decentralized from a host computer. It is widely used in industries were robustness and determinism is of high importance, such as the automotive indus-try for which it was originally designed[24]. Since then CAN has developed multiple standards tailored towards dierent industries including aviation, agriculture, marine vehicles and more[25]. Devices connected to the CAN bus are referred to as nodes, and they are connected through a two wire bus (CAN HIGH and CAN LOW). CAN is a multi-master serial bus, meaning every node is capable of initiating data transfer to other nodes, making every node a master node. These nodes communicate in messages dened by the CAN specication[26] called frames that are broadcasted to all nodes in the network and it is up to all nodes to decide if they should keep or discard the data, depending on if the data is important to them. All frames are sent with a priority in form of an identier eld in the frame, and nodes will react by yielding to higher priorities and trying to resend their messages later. Each node requires three components to utilize the protocol.

1. Some sort of CPU, most often a microprocessor has to be part of each node that decides what incoming messages mean and what messages it should transfer to other nodes. 2. A CAN controller, most often built into the microprocessor which is responsible for the

actual send and receive mechanisms.

3. A transceiver that converts the incoming and outgoing data streams between CAN bus level and CAN controller levels.

(14)

Figure 2.3: CAN Bus Architecture with n nodes

To put CAN into a common use case in this context, sensors and actuators are often connected to an I/O module via an M12 connector[27] and the I/O module communicates to embedded computers and other modules via CAN. In this setup, sensors and actuators would not be referred to as CAN nodes since they only emit a voltage or current to the I/O module.

2.2.3 RCS IO

RCS IO is the interface that allows RCS applications to communicate with I/O modules across the CAN bus. As a consequence of the CAN driver developed for RCSOS, RCS IO only allows one process to use the CAN interface simultaneously, which causes tasks that the application want to achieve has to be structured in a monolithic architecture, as seen in Fig 2.4. These may or may not be multi-threaded depending on necessity, but constrained to a single process application. The real-time requirements of the system consists of a synchronization pulse sent across the CAN network each 80 milliseconds in which the system needs to read the input, conduct necessary computations and provide outputs to the I/O modules before the next sync pulse which is described by the nite state machine in Fig 2.5. While the applications are generally ran as a single process, multi-threading allows the system to meet the required deadlines. This is done by giving responsibility over the CAN interface for reading, writing and notifying other threads when the deadlines are due to a specic thread and the rest of the threads are dedicated to computing the necessary I/O input data. The idle period exists partly to make sure the 80ms deadline is deterministic, but also to allow lower priority processes computation time.

(15)

Figure 2.4: High Level RCS Architecture, three example tasks

Figure 2.5: I/O Finite State Machine

2.3 Related work

Erös et.al proposed a communication architecture for a six degrees of freedom industrial robot with multiple machines running dierent distributions of ROS and ROS 2.0. Due to ROS 2.0

(16)

cur-rently being in development, it does not have the luxury of as large of a package selection as ROS distributions but compatibility is provided by ROS 2.0. The study utilizes the package selection of ROS on nodes local to each machine, and ROS 2.0 as communication bridges utilizing DDS[28]. Park et.al. conducted a performance evaluation of the real-time characteristics of ROS com-pared to ROS 2.0. The metrics consisted of measuring scheduling latency, task periodicity, task response times and message loss. The study found a signicant improvements in real-time per-formance in favor of ROS 2.0 in both idling and stressed scenarios[29].

Kronauer et.al. measured the impact payload size, publisher frequency, number of nodes and DDS implementation had on the latency of the system. The study found that CycloneDDS performed the best in all scenarios, and was a large source in the dierences between latencies. The second largest impact regarding latency is that the delay between message notication and message retrieval is a large contributor.

Gutíerrez et.al. performed a comparison between using an real-time kernel vs a non-real-time kernel and found a signicant improvement in favor of the RT_PREEMPT kernel. The results show the largest dierences in which the system was under heavy stress, which points to the importance of preemption so that time-critical processes are executed as fast as possible. To evaluate ROS 2.0 communication performance, two actively maintained frameworks were found. The most rigorous one that supports distributed testing environments are ApexAIs performance-tool, but does not support newer distributions of ROS 2.0 and were found unsuitable for this project. The second tool is iRobot's performance tool[30], which supports newer versions but lacks support for measuring diagnostics across dierent machines.

(17)

3. Reference Implementation

This chapter presents an overview of the systems architecture of the proposed solution to inte-grate ROS 2.0 in the RCS platform, and how it is implemented. Figure 3.1 describes how the system architecture was designed to t a general amount of sensors, actuators and distributed task nodes.

Sensors send their signals to the I/O modules which are then sent across the CAN bus to the proposed listener, which is the developed interface of the system. The listener facilitates the communication and message conversation between CAN and ROS 2.0, utilizing the RCS IO library to setup and maintain the connection. Once sensors messages are converted they are published to the ROS 2.0 node space, where any node can pick up desired messages. The load nodes performs the necessary computations, and passes the output back through the listener to the actuators that are waiting for control signals.

Figure 3.1: Generalized Proposed system The implementation can be broken down into three phases:

1. ROS 2.0 and RCSOS compatibility 2. RCS listener development

3. System Evaluation

Integration and development was done in parallel due to RCS IO providing a hardware simula-tor called IO SIM. This lets smaller implementations of the listener to be tested against both hardware and the simulator, verifying that both systems have the same behavior.

(18)

3.1 Software and Middleware

Since RCSOS is an embedded operating system, it lacks many functionalities of general-purpose Ubuntu distributions, most notably the package manager apt which ROS 2.0 uses to gather pre-installation dependencies. Installation of the ROS 2.0 framework from source on Ubuntu 20.04 can be divided into three steps. Firstly, install pre-installation dependencies using the package manager, including an additional tool of managing dependencies called rosdep. Secondly, use rosdep to acquire a larger set of dependencies which also uses apt. Thirdly, compile the ROS 2.0 framework. Making these dependencies available on target hardware poses a problem since it is highly reliant on the package manager. The considered solutions to this is to:

1. Alter the RCSOS release to include these dependencies and build ROS 2.0 from source. 2. Cross compile towards RCSOS while making missing dependencies available to the compiler

externally.

3. Compile towards Ubuntu 20.04 and use an RCSOS release with the same architecture. Option 1 is undesirable because it mixes the build and runtime environments on the embedded system, which by request of Epiroc should be separate if possible. Option 2 was attempted but proved dicult because of the extensive amount of required build dependencies. Option 3 was achieved by choosing RCSOS 3.2, which results in an 64-bit x86 architecture on both the H6072 disp and Dell workstation then making the runtime libraries required for ROS 2.0 available to the disp, which is a lot fewer.

3.1.1 Build Tools

The components of RCS used in this project are built in C++, and the build environment is managed with RDK which provides functionalities in form of CMake macros and Python scripts. ROS 2.0 however provides colcon, a universal build tool designed to build both C++ and Python packages, and most importantly manage the dependencies between the ROS 2.0 package tree structure. To avoid collision between colcon and RDK, the RCS repositories were migrated to pure CMake and built inside the ROS 2.0 workspace as a single package. This allows all libraries from RCS to become available inside the installation directory inside the ROS 2.0 workspace.

3.1.2 RCS Listener

Executed as a ROS 2.0 node, the listener is the RCS-ROS2 interface. It is responsible for estab-lishing and maintaining the connection to the CAN interface. When connection is validated, it publishes the input it receives, and subscribes to the outputs it is congured to have in accor-dance to the 80ms sync loop criteria described in gure 2.5.

The listener is built as a single ROS 2.0 node and is dependent on the following parameters for initialization:

1. Module cong - species communication protocol and which runtime library to use. This is provided with a yaml le either by manual input or a default conguration.

2. Hardware conguration - species how many modules it should expect in the system, which communication channels that are open on the modules and which messages it should expect. 3. CAN channel - an I/O module commonly has two I/O channels, CAN0 and CAN1.

(19)

This process sets up the interface between to the CAN network via RCS IO.

The listener has two states, loading or operational. During the loading phase the node communi-cates with the hardware modules through the stages from boot to operational. The operational state is reached when loading has been completed, and the system now adheres to the 80ms synchronization criteria.

The main function that drives the node forward is implemented as a ROS 2.0 callback function at an interval of 1 millisecond. This function is responsible for deciding which action to take depending on which context and state the system is located in. Timing is controlled by a custom built pulse clock class which determines which action should be taken.

The real time criteria are described in gure 2.5 and their control are implemented as functions: Tick - Drives communication forward to the CAN network, executes timed operations such

as read or write to the CAN bus. This action is taken when no other action is queued. Load - Drive loading state machine forward such as send congurations and fetch module

objects.

Sync - Send a prompt to synchronize the CAN network and start the reading phase. Read - Reading stage is completed. All received inputs are published through ROS 2.0. Write - The computation stage is now done. All outputs should have returned from load

nodes via ROS 2.0 subscriptions, and are written to the I/O bus. Algorithm 1 Listener Action Controller (1ms interval)

1: if state == load then 2: tick()

3: load() 4: else

5: if state == sync then

6: sync()

7: else if state == read then 8: read()

9: else if state == write then 10: write()

11: else 12: tick()

3.1.3 RCS Load

This node can be executed on any machine in which ROS 2.0's discovery mechanism can nd it, and it is designed to simulate a task that has to be completed during the computation phase. It subscribes to inputs published by the listener, and publishes outputs which the listener picks up and writes to the I/O module.

(20)

3.2 Hardware

This section describes the hardware components that were used in testing and development of the system.

3.2.1 Dell Workstation

Development is conducted on a Dell Workstation with a 2.6GHz processor and 64GB RAM. ROS 2.0 oers full support of Ubuntu 20.04 Focal, which is ran in a virtual environment using VMWare Player. Epiroc provided an image which includes SDKs to all RCSOS releases including toolchains for cross compilation. Additionally, the image includes a custom designed development kit RDK which provides remote access to projects via GIT and build/launch congurations. This machine also serves as host of ROS 2.0 nodes that are masquerading as computational load. These can be run on the embedded computer too, but this choice was made to further examine if distributing the ROS 2.0 user space to another machine imposes any limitations.

3.2.2 H6072 Display

The embedded computers responsible of executing RCS applications and communicating with I/O modules are called Disps, and during this project the H6072 model was used, see Fig 3.2. Its purpose is to host the RCS-ROS2 interface as a ROS 2.0 node which facilitates the message pipeline from sensors connected to the I/O module to the ROS 2.0 user space hosted on the Dell workstation.

Figure 3.2: H6072 Display

3.2.3 I/O T2 Module

The T2 I/O module serves as a mediator between the disp and I/O devices. It supports two CAN networks, current controlled PWMs, digital outputs, analog and digital inputs and encoder interfaces. In this system one CAN network is utilized and a potentiometer is used as input for testing purposes.

(21)

Figure 3.3: I/O T2 Module

The module has four dierent states. In the boot state it is on standby, waiting for initialization. During the preoperational state it awaits congurations messages on which channels to use. The operational state is reached when conguration is done, and an application is actively running. The stopped mode is reached on some hardware failure. To return the module to boot, a restart of the module is required.

(22)

4. Results

This section will present the results of the performance testing of the system. Using iRobot's performance tool, the metrics gathered for performance evaluation are latency, CPU usage and memory usage. Since the 80ms synchronization pulses are not subject to change, the goal is to keep a constant 12.5 Hz publishing rate and is used as a benchmark. Message payload size and the amount of publisher and subscribers are used as varying parameters to see how they impact the system. To create a more coherent view of ROS 2.0 roles in the embedded system, two dif-ferent operating modes were selected for evaluation, local testing and distributed testing. Local testing refers to ROS 2.0 performance when isolated on the embedded hardware, and not actively communicating with a I/O module or the interface. This is motivated by the question of how ROS 2.0 and CycloneDDS performs with as few outside dependencies as possible. Distributed testing is done through measuring latencies between publishers and subscribes when the listener is executed on an embedded computer connected to the I/O module, and the load nodes is exe-cuted on the PC workstation. No suitable tools for this task was found, and measurement were done by multithreaded logging in the listener, and using ROS 2.0's command line diagnostics tool.

4.1 Local Performance Testing

This section presents the isolated tests of real-time characteristics of ROS 2.0 on the H6072 display. The decision of not using intra-process communication was made as it is not possible in a distributed environment and often preferred in local systems to maintain modularity and preserve memory separation. The experiments were run for 120 seconds with alternating count of publishers and subscribers, and varying payload sizes. The minimum message that would be sent as a CAN input or output is an 4 byte integer and therefore serves as the lower bound and as an upper bound 500 bytes were used as this would represent 125 I/O signals in a single message. To put this in perspective, a larger machine may have up to 200 signals. The metric used for these tests are referenced as PS, which represents the amount of sensors to actuator signals. Each experiment with PS = n, has one master node with n subscribers that represent output signals and n publishers that represent input signals. The load is designed as n nodes with one publisher and one subscriber each, so P S = n is n + 1 nodes, 2n total publishers, 2n total subscribers. The amount of P S was incrementally increased until packet loss started occurring.

(23)

Figure 4.1: Testing setup for PS=n

4.1.1 Latency

The results in terms of latency shows a linear increase in respect to the amount of publishers and subscribers as seen in gure 4.2. Tables 4.1 - 4.4 represents the average round trip times from listener -> load -> listener of all topics in the system. Messages are considered late if the round trip time is greater than 5 ms and lost if they fail to meet the real-time requirements of 27 ms as described in gure 2.5.

Late packages started appearing when PS = 10 for all message sizes, showing that payload size had no signicant impact on latency between 4 B to 500 B. Lost packages was occurring at PS = 20, with the highest percentage at table 4.2 at 0.000922 % packages lost.

(24)

PS mean[us] Late[#] Lost[#] 1 470 0/3000 0/3000 5 668 0/15007 0/15007 10 842 10/30015 1/30015 20 1248 49/58529 5/58529

Table 4.1: 4 Byte message size

Table 4.2: 32 Byte message size PS mean[us] Late[#] Lost[#]

1 545 0/3000 0/3000 5 698 0/15007 0/15007 10 836 10/30015 0/30015 20 1248 46/58529 16/58529

Table 4.3: 100 Byte message size

Table 4.4: 500 Byte message size Figures 4.3 - 4.6 presents the error plots of the latencies of individual topics for each experiment. The low standard deviation is consistent with with previous results showing that the absolute majority of packages are sent and received in similar timings but the mean latency is linearly increased. The lost packages that appear at P S = 10 are shown to be completely from topic 20 as seen in gure 4.5 where it is the only topic that exceeds 27 ms.

(25)

Figure 4.4: Error plot of min, max, mean and standard deviation with PS=5

(26)

Figure 4.6: Error plot of min, max, mean and standard deviation with PS=20

4.1.2 CPU and memory usage

Similarly to latency, CPU and memory usage has a linear increase with the amount of publishers and subscribers in the system while showing no signicant dierence between payload sizes. At P S = 20the H6072 reached 40% CPU usage, although it is also hosting all the load nodes which would not be present in an distributed system.

(27)

Figure 4.8: Memory Usage

4.2 Distributed Testing

In distributed testing, a more realistic setup is done utilizing the hardware. A potentiometer is attached to the I/O T2 module as a sensor that emits a variable voltage. The input from the sensor is transported across the CAN bus to the H6072, which is running only the listener interface. The input is published through DDS, and using UDP multicasting across the local network and is received by the load node running in VMWare on the PC workstation. The load continues the process back to the I/O module, but there is no actuator connected to the system and the output signal is terminated.

(28)

Figure 4.10: Testing Architecture

4.2.1 ROS 2.0 Hz Tool

ROS 2.0's command line tool ros2 topic hz measures the average publishing of any given topic. The interface should ideally publish its topic every 80 ms, and the load should return the corre-sponding output topic within the 27 ms time frame. Both nodes receive a publishing frequency of 80 ms, which is the desired goal of both nodes The load node executed on the workstation presents slightly higher standard deviation, although higher on computational resources it is not running real-time threads or a real-time OS.

Topic mean[ms] σ[ms] min[ms] max[ms] Interface -> Load 80.0 0.5 79.0 81.0 Load -> Interface 81.0 1.0 78.0 83.0

Table 4.5: ROS2 Hz tool latencies

4.2.2 Worst case latencies

The internal logging of the listener records a set of 2000 samples that contains the worst latencies acquired during this test, to see how many packages are late or lost. The majority of worst case latencies occur in the interval of 2 - 4 ms, which is in an acceptable time frame and not considered late. Among the late messages there was 7/2000 occurances, none which are considered lost.

(29)

Figure 4.11: Bar plot of 2000 worst case latencies

(30)

5. Discussion

5.1 Requirements Compliance

Deployment of ROS 2.0 on the embedded H6072 display was achieved, but was not compiled for the system and may pose future problems if for example the standard libraries on the virtual machine is not fully compatible with the standard libraries on the embedded devices, but it was decided adequate as a proof of concept.

The interface was established and functioning for both simulation and hardware.

The interface was functional and tested as part of a distributed system in which the ROS 2.0 load nodes was executed on another machine.

The evaluation was conducted which is presented in chapter 4 and discussed in section section 5.2

5.2 Result Analysis

The experiments performed on varying payload sizes and publisher - subscriber count provided an insight to the limitations of ROS 2.0 and this particular design architecture. As seen in 5.1 - 5.4 and gures 5.1 - 5.3, payload size did not have a signicant impact on the system in ei-ther latency, CPU usage or memory usage. This however is subject to change when considering larger message size dierences, as shown by the authors of the framework[30].In this specic use case, 500B is considered an realistic upper bound. A linear relationship however emerged from the mean of all three metrics depending on the amount of publishers and subscribers present in the system. As expected, adding more nodes and message passing mechanisms would add more overhead and computation time to the system which is conrmed by these results. Lost messages started to occurring at a interface node with 10 publishers and 10 subscribers, and the examination of individual topics provides some insight to this.

The latencies of individual topics show a both low standard deviation in latency for subscribers and and publishers, see g 4.4 - 4.7. It does however show a greater average latency for sub-scribers than publishers. An interesting observation of the design that the interface has to serve as a central hub for communication can be observed in gure 4.5. Topics 1-10 represents the topics subscribed to by the load nodes, and as separate processes all are measured to similar and low latencies. Topics 11-20 however represents the topics subscribed to by the interface node received from each load node, and due to the single threaded executor the callback functions of the subscriptions gets stacked in a queue, creating this ladder pattern. Considering in some vehi-cles may carry up to 150-200 I/O signals, in the current implementation these would have to be split into groups less than 10 to not incur any message losses in the system. A possible solution to this may be to implement multiple multithreaded callback queues, or or create intra-process nodes for each subscriber and use ROS 2.0's own multithreaded executor.

(31)

The distributed testing performed well and showed no lost packages and both the listener and load node performed adequately to the real-time requirements. This testing was however limited to a lower payload size and load on the system, but showed the functional proof-of-concept of the system.

5.3 Social and Economical Implications

As a topic regarding the automation of vehicles, both social and economical view points can be noted here. Driverless vehicles is predicted to be the largest eliminator of jobs in the foresee-able future, although for this particular case I consider the arguments stronger due to the vast increase in safety aspects when considering the underground mining industry and its hazards. Another interesting consideration is the movement of industries to open source platforms, which may result in higher cooperation between industries, including researchers, in development. Com-panies and corporations are perhaps less likely to release their own developed ROS applications, but with a vested mutual interest in the frameworks success the projects will receive more de-velopers.

5.4 Recommendations for further work

The interface in this thesis acts as a master node that has full responsibility of distributing the information received from the CAN bus. In the current monolithic RCS system, RCS applica-tions are generally split between multiple CAN channels to spread the load and ensure a more robust fault insurance, and this can be done for the listener as well.

Multiple topics that did not t the scope of this thesis can be considered for further analysis regarding this system. This evaluation has only considered CycloneDDS as the middleware im-plementation, and a larger study comparing all of them can yield a more conclusive result. As conrmed by previous studies mentioned in section 2.3, an evaluation with a highly stressed environment should be conducted to become aware of any degradation of the system if that were to happen.

The tools available of ROS 2.0 performance testing was limited, and including support for newer ROS 2.0 distributions can be considered for ApexAIs performance-test to further analyze dis-tributed diagnostics. To add to this, one may also consider increasing the numbers of disdis-tributed computers and measure its eect on the system.

Simulation testing was a topic that was initially planned, but was left out towards the end of the project. An existing loader model was ported to be compatible with ROS 2.0, but no controller is yet not be implemented. Some controller that subscribes to I/O inputs and forwards it to actuators in a simulation engine like Gazebo could be evaluated in terms of value as this is a very valuable tool in development.

5.5 Reections on own learning

This project was a great opportunity to learn about software engineering in an agile approach where starting requirements are unknown or uncertain and adaptions has to be made during

(32)

development. Compatibility issues and system integration is something that is present in many software projects, and had to be taken into account at multiple times during this thesis regarding hardware, operating systems and software.

It also provided deeper understanding of how autonomous vehicles are designed and the safe and deterministic communication protocols that are vital to ensure a quality product.

(33)

Bibliography

[1] ROS 2020 Metrics Report. May 27, 2021. url: https://discourse.ros.org/t/2020-ros-metrics-report/16394.

[2] Robot Operating System (ros). url: https://docs.ros.org.

[3] Morgan Quigley et al. ROS: an open-source Robot Operating System. In: ICRA workshop on open source software. Vol. 3. 3.2. Kobe, Japan. 2009, p. 5.

[4] Cooperative Robots and Sensor Networks 2015. eng. 2015th ed. Vol. 604. Studies in Com-putational Intelligence. Cham: Springer International Publishing. isbn: 9783319182988. [5] Ayssam Elkady and Tarek Sobh. Robotics Middleware: A Comprehensive Literature

Sur-vey and Attribute-Based Bibliography. In: Journal of Robotics 2012 (May 2012). doi: 10.1155/2012/959013.

[6] ROSPatterns. url: http://wiki.ros.org/ROS/Patterns/. [7] ROS/TCPROS wiki. url: http://wiki.ros.org/ROS/TCPROS. [8] ROS/TCPROS wiki. url: https://wiki.ros.org/Master.

[9] Byoungwook Choi et al. Integration of ROS and real-time tasks using message pipe mech-anism on Xenomai for a telepresence robot. In: Electronics Letters 55 (Nov. 2018). doi: 10.1049/el.2018.5560.

[10] Why ROS2? url: https://design.ros2.org/articles/why_ros2.html.

[11] Hongxing Wei et al. RT-ROS: A real-time ROS architecture on multi-core processors. In: Future Generation Computer Systems 56 (2016), pp. 171178. issn: 0167-739X. doi: https: //doi.org/10.1016/j.future.2015.05.008. url: https://www.sciencedirect.com/ science/article/pii/S0167739X15001831.

[12] Alexander Tiderko, Frank Hoeller, and Timo Röhling. The ROS Multimaster Extension for Simplied Deployment of Multi-Robot Systems. In: vol. 625. Feb. 2016, pp. 629650. isbn: 978-3-319-26052-5. doi: 10.1007/978-3-319-26054-9_24.

[13] The Real-time Publish-Subscribe Protocol DDS Interoperability Wire Protocol (DDSI-RTPS) Specication. Apr. 3, 2019.

[14] ROS 2 middleware interface. url: https://design.ros2.org/articles/ros_middleware_ interface.htmll.

[15] ROS2 Foxy Fitzroy, About dierent ROS 2 DDS/RTPS vendors. url: https://docs.ros. org/en/foxy/Concepts/About-Different-Middleware-Vendors.html.

[16] ROS2 QoS. url: http://design.ros2.org/articles/qos.html.

[17] Yuya Maruyama, Shinpei Kato, and Takuya Azumi. Exploring the Performance of ROS2. In: Proceedings of the 13th International Conference on Embedded Software. EMSOFT '16. Pittsburgh, Pennsylvania: Association for Computing Machinery, 2016. isbn: 9781450344852. doi: 10.1145/2968478.2968502. url: https://doi.org/10.1145/2968478.2968502.

[18] ROS2 Intra-process communication. url: https://design.ros2.org/articles/intraprocess_ communications.html.

(34)

[19] ros1 bridge. url: https://github.com/ros2/ros1{\_}bridge.

[20] gRPC (Google Remote Procedure Communication. url: https://grpc.io/. [21] zeroMQ. url: https://zeromq.org/.

[22] OpenRAVE. url: https://github.com/rdiankov/openrave.

[23] The Real-Time Linux Wiki. Apr. 20, 2021. url: https://rt.wiki.kernel.org/index. php/CONFIG_PREEMPT_RT_Patch.

[24] CAN History. url: https://www.can-cia.org/can-knowledge/can/can-history/. [25] CAN in Automation, international standardization. url: https://www.can- cia.org/

groups/international-standardization/.

[26] Steve Corrigan HPL. Introduction to the controller area network (CAN). In: Application Report SLOA101 (2002), pp. 117.

[27] M12 connector specication, IEC 610762101. url: https : / / webstore . iec . ch / p -preview/info_iec61076-2-101%7Bed1.0%7Den.pdf.

[28] Endre Er®s et al. A ROS2 based communication architecture for control in collaborative and intelligent automation systems. In: Procedia Manufacturing 38 (2019). 29th Interna-tional Conference on Flexible Automation and Intelligent Manufacturing ( FAIM 2019), June 24-28, 2019, Limerick, Ireland, Beyond Industry 4.0: Industrial Advances, Engineer-ing Education and Intelligent ManufacturEngineer-ing, pp. 349357. issn: 2351-9789. doi: https: //doi.org/10.1016/j.promfg.2020.01.045. url: https://www.sciencedirect.com/ science/article/pii/S2351978920300469.

[29] Jaeho Park, Raimarius Delgado, and Byoungwook Choi. Real-Time Characteristics of ROS 2.0 in Multiagent Robot Systems: An Empirical Study. In: IEEE Access PP (Aug. 2020), pp. 11. doi: 10.1109/ACCESS.2020.3018122.

[30] iRobotBenchmark testing. url: https://github.com/irobot-ros/ros2-performance/ tree/master/performances/experiments/crystal/pub_sub_latency.

Integration och realtidsprestanda av ROS 2.0 i ett inbyggtoch distributerat system