• No results found

Non-intrusive Logging and Monitoring System of a Parameterized Hardware-in-the-loop Real-Time Simulator

N/A
N/A
Protected

Academic year: 2021

Share "Non-intrusive Logging and Monitoring System of a Parameterized Hardware-in-the-loop Real-Time Simulator"

Copied!
82
0
0

Loading.... (view fulltext now)

Full text

(1)

IN

DEGREE PROJECT ELECTRICAL ENGINEERING, SECOND CYCLE, 30 CREDITS

,

STOCKHOLM SWEDEN 2019

Non-intrusive Logging and

Monitoring System of a

Parameterized

Hardware-in-the-loop Real-Time Simulator

MUHAMAD ANDUNG MUNTAHA

KTH ROYAL INSTITUTE OF TECHNOLOGY

(2)
(3)

Non-intrusive Logging and

Monitoring System of a

Parameterized

Hardware-in-the-loop

Real-Time Simulator

MUHAMAD ANDUNG MUNTAHA

Master of Science in Embedded Systems Date: June 13, 2019

Supervisor: Rodolfo Jordão Examiner: Ingo Sander

School of Electrical Engineering and Computer Science Host company: Scania CV AB

Swedish title: Icke-påträngande loggnings och

övervakningssystem för en parametrerad hårdvara-in-the-loop realtidsimulator

(4)
(5)

iii

Abstract

Electronic Control Unit (ECU) is a crucial component in today’s vehicle. In a complete vehicle, there are many ECUs installed. Each of these controls a single function of the vehicle. During the development cycle of an ECU, its functionality needs to be validated against the requirement specification. The Hardware-in-the-loop (HIL) method is commonly used to do this by testing the ECU in a virtual representation of its controlled system. One crucial part of the HIL testing method is an intermediary component that acts as a bridge between the simulation computer and the ECU under test. This component runs a parameterized real-time system that translates messages from the simulation computer to the ECU under test and vice versa. It has a strict real-time requirement for each of its tasks to complete.

A logging and monitoring system is needed to ensure that the intermediary component is functioning correctly. This functionality is implemented in the form of low priority additional tasks that run concurrently with the high priority message translation tasks. The implementation of these tasks, alongside with a distributed system to support the logging and monitoring functionality, is presented in this thesis work.

Several execution time measurements are carried out to get the information on how the parameters of a task affect its execution time. Then, the linear regression analysis is used to model the execution time estimation of the parameterized tasks. Finally, the time demand analysis is utilized to provide a guarantee that the system is schedulable.

Keywords: Mixed-Criticality System; Real-Time Linux; Hardware-in-the-loop; Integration Testing; Electronic Control Unit; Time Demand Analysis; Regression Analysis

(6)

iv

Sammanfattning

Elektronisk styrenhet (ECU) är en viktig del i dagens fordon. I ett komplett fordon finns det många ECU installerade. Var och en av dessa kontrolle-rar en enda funktion hos fordonet. Under en utvecklingscykel för en ecu måste dess funktionalitet valideras mot kravspecifikationen. HIL-metoden (Hardware-in-the-loop) används vanligtvis för att göra detta genom att testa ECU i en virtuell representation av sitt styrda system. En viktig del av HIL-testmetoden är en mellanliggande komponent som fungerar som en bro mellan simuleringsdatorn och den ecu som testas. Denna komponent driver ett parametrerat realtidssystem som översätter meddelanden från simuleringsdatorn till ECU som testas och vice versa. Det har en strikt realtidskrav för att alla uppgifter ska kunna slutföras.

Ett loggnings och övervakningssystem behövs för att den mellanliggande komponenten ska fungera korrekt. Denna funktionalitet är implementerad i form av extraordinära uppgifter med låg prioritet som körs samtidigt med de högsta prioritetsuppgifterna för översättningstjänster. Genomförandet av dessa uppgifter, tillsammans med ett distribuerat system för att stödja loggnings och övervakningsfunktionaliteten, presenteras i detta avhandlingar-arbete.

Flera utförandetidsmätningar utförs för att få information om hur para-metrarna för en uppgift påverkar dess körtid. Därefter används den linjä-ra regressionsanalysen för att modellelinjä-ra exekveringstidestimeringen av de parametrerade uppgifterna. Slutligen används tidsanalysanalysen för att garantera att systemet är schemaläggbart.

Nyckeord: Mixed-Criticality System; Realtid Linux; Hårdvara-in-the-loop; Integrations Testning; Elektronisk Styrenhet; Tidskrävnings Analys; Regres-sions Analys

(7)

v

Acknowledgements

Firstly, I would like to thank my supervisor at Scania, Maximilian Perzinger, for his valuable guidance during the development of this project. I would also like to extend my thanks to Paula Härelind, who gave me the opportunity to do a master thesis at Scania, as well as all System Automation Environment engineers who were very helpful in answering my questions.

Secondly, I would like to express my appreciation to Rodolfo Jord¯ao as my second supervisor at KTH for his time providing valuable and constructive suggestions in completing this project.

Thirdly, I would like to express my gratitude to the government of Indonesia through Indonesia Endowment Fund for Education (LPDP), for providing full two years of financial support of my study.

Finally, I wish to thank my beloved wife, Ernita Dewi, my dearest daughter, Khaira Andini, my brothers, my sister, and my parents, for their support and encouragement throughout my study.

Stockholm, June 13, 2019

(8)

Contents

Contents vi List of Figures ix List of Tables xi Listings xii Acronyms xiii 1 Introduction 1 1.1 Project Description . . . 2 1.2 Motivation . . . 3 1.3 Problem Statement . . . 3 1.4 Purpose . . . 3 1.5 Goals . . . 3

1.6 Risks, Consequences, and Ethics . . . 4

1.7 Methods . . . 4

1.8 Requirements . . . 5

1.9 Limitations . . . 5

1.10 Organization . . . 6

2 Background 7 2.1 Electronic Control Unit . . . 7

2.2 Hardware-in-the-loop Simulation . . . 7

2.3 CompactRIO Controller . . . 9

2.4 User Datagram Protocol . . . 10

2.5 Real-Time Systems . . . 11

2.5.1 Real-Time Model . . . 12

(9)

CONTENTS vii 2.5.2 Schedulability Test . . . 12 2.6 Real-Time Linux . . . 13 2.6.1 Scheduling Policy . . . 14 2.6.2 Priority Inheritance . . . 15 2.7 Regression Analysis . . . 16

2.7.1 Simple Linear Regression . . . 16

2.7.2 Multiple Linear Regression . . . 16

2.8 Web Technology . . . 17

2.8.1 ELK Stack . . . 17

2.8.2 ASP.NET Core . . . 18

2.8.3 SignalR . . . 18

2.8.4 Single Page Application . . . 19

2.8.5 Container . . . 19

2.9 Related Works . . . 20

3 Implementation 22 3.1 Logging & Monitoring Server . . . 23

3.1.1 Backend . . . 23

3.1.2 Frontend . . . 24

3.1.3 ELK Stack . . . 25

3.1.4 Deployment . . . 26

3.2 Logging & Monitoring RTLinux Tasks . . . 26

3.2.1 Architecture . . . 26

3.2.2 Logging Task . . . 27

3.2.3 System Load Measurement Task . . . 27

3.2.4 Monitoring Task . . . 28

3.3 I/O Server RTLinux Tasks . . . 28

3.3.1 Architecture . . . 29

3.3.2 UDP Receiver Tasks . . . 30

3.3.3 UDP Sender Tasks . . . 31

4 Methods 32 4.1 Execution Time Measurement . . . 33

4.2 Regression Analysis . . . 35

4.3 Schedulability Analysis . . . 35

5 Results & Analysis 38 5.1 Execution Time Estimation . . . 38

5.1.1 Logging Task . . . 38

5.1.2 I/O Server Tasks . . . 40

(10)

viii CONTENTS

5.2 Parameters Variation . . . 43

5.2.1 Varying Boolean and Float Variables . . . 43

5.2.2 Varying Log Messages . . . 43

5.3 Discussion . . . 44

6 Conclusions & Future Work 48 6.1 Conclusions . . . 48

6.2 Limitation . . . 49

6.3 Future Work . . . 49

Bibliography 50

Appendices 54

A Regression Analysis Code 55

B Configurations 58

(11)

List of Figures

1.1 ECU variation depending on vehicle types and features [1] . . 1

1.2 HIL simulator building blocks . . . 2

2.1 Abstract view of a HIL local environment simulator for con-troller design, adopted from [8] . . . 8

2.2 The CompactRIO controllers, adopted from [6] . . . 9

2.3 CompactRIO heterogeneous architecture, adopted from [6] . . 10

2.4 The UDP packet . . . 11

2.5 Real-time task workload model . . . 12

2.6 RT-Linux implementation architecture . . . 14

2.7 An example of ELK stack topology . . . 18

2.8 Containers and virtual machines comparison, adopted from [29] 20 3.1 High-level overview of the complete system architecture . . . 22

3.2 Data flow diagram of the complete system . . . 23

3.3 Backend process flow . . . 24

3.4 Polling packet format . . . 24

3.5 Logging and monitoring agent tasks implementation diagram 27 3.6 Logging task process flow . . . 27

3.7 System load measurement task process flow . . . 28

3.8 Monitoring task process flow . . . 28

3.9 I/O server tasks implementation diagram . . . 29

3.10 UDP receiver task process flow . . . 30

3.11 UDP sender task process flow . . . 31

4.1 Methods overview . . . 32

4.2 General execution time measurement mechanism . . . 33

4.3 Logging task execution time measurement mechanism . . . . 34

(12)

x LIST OF FIGURES

4.4 I/O Server task execution time measurement automation . . 35

4.5 Schedulability analysis process flow . . . 36

5.1 Logging task execution time regression analysis result . . . . 39

5.2 I/O server task execution time regression analysis result . . . 41

5.3 Varying boolean & float variables result . . . 46

5.4 Varying log messages result . . . 47

C.1 Real-time monitoring dashboard . . . 63

C.2 Centralized logging dashboard . . . 63

(13)

List of Tables

2.1 CompactRIO 9030 technical specification, adopted from [9] . 9

3.1 Logging mechanism execution time comparison . . . 29

4.1 A task set used as a test case . . . 37

5.1 Parameterized logging task execution time data sample . . . 39

5.2 Parameterized UDP receiver task execution time data sample 41

5.3 Parameterized UDP sender task execution time data sample . 42

5.4 System’s tasks model . . . 42

(14)

Listings

3.1 Logstash configuration . . . 25

3.2 Monitoring data . . . 26

3.3 Logging data . . . 26

A.1 Logging task regression . . . 55

A.2 UDP sender/receiver task regression . . . 56

B.1 tasks-model.json snippet . . . 58

B.2 agent-config.json snippet . . . 59

B.3 ioserver-config.json snippet . . . 60

(15)

Acronyms

ABS Anti-lock Braking Systems. 8

API Application Programming Interface. 10, 18, 19, 24

CAN Controller Area Network. 1, 2, 7, 8 CI Continues Integration. 49

CSS Cascading Style Sheets. 19 CSV Comma Separated Values. 34

ECU Electronic Control Unit. iii, 1, 2, 7, 8, 9, 10, 28 EDF Earliest Deadline First. 15

ELK Elasticsearch, Logstash, Kibana. 17, 18

FPGA Field Programmable Gate Array. 9, 10, 30, 31 GPOS General Purpose Operating System. 13, 14

HIL Hardware-in-the-loop. iii, 1, 2, 8, 10, 11 HTML Hyper Text Markup Language. 19 HTTP Hyper Text Transport Protocol. 24, 26

I/O Input/Output. ix, 2, 7, 9, 10, 11, 26, 28, 29, 30, 31, 33, 34 IP Internet Protocol. 10

(16)

xiv Acronyms

IPC Interprocess Communication. 28

JSON JavaScript Object Notation. 17, 18, 23, 25, 42

KPI Key Performance Index. 3 MVC Model-View-Controller. 24

NI National Instruments. 5, 14

OS Operating System. 5, 18, 19, 20

POSIX Portable Operating System Interface. 26, 28, 30

REST Representational State Transfer. 18, 24 RTLinux Real-Time Linux. 5, 6, 9, 13, 14, 15, 20, 22 RTOS Real-Time Operating System. 13, 14, 20

SPA Single Page Application. 19, 24 SSH Secure Shell. 34

TCP Transport Control Protocol. 18 TCS Traction Control Systems. 8

UDP User Datagram Protocol. 9, 10, 11, 18, 30, 31, 33, 34 UI User Interface. 19

(17)

Chapter 1

Introduction

This thesis project was carried out at Scania research and development, particularly in the System Automation Environment department, EPSE. EPSE is responsible for providing various Hardware-in-the-loop (HIL) test environment for vehicle integration test and system test. Another department called EPSI is the actual tester which uses the infrastructure provided by EPSE to run the actual tests.

Electronic Control Unit (ECU) is a critical component in today’s vehicle controlling many parts of it. In one complete vehicle, there are up to 93 ECUs connected in a Controller Area Network (CAN) [1][2]. The number of ECU and its configuration varies depending on the vehicle type and configuration, as illustrated in figure 1.1. Scania develops all types of these ECUs needed for their product line.

ECU ECU ECU ECU

ECU ECU ECU ECU ECU ECU ECU ECU ECU ECU ECU ECU ECU ECU ECU ECU ECU ECU ECU ECU ECU Trailer ECU ECU ECU ECU ECU ECU

ECU ECU ECU ECU ECU ECU ECU Trailer PAGE 14

require vehicle properties that match the demands. For example, a trans-port truck needs to come with opti-mal confi gurations for its special type of cargo (volume freight, reefer freight, liquids, bulk cargo, etc.), confi gurations that satisfy every little detail.

High-Performance Vehicle Technology

Safe driving, smooth-running vehicle handling, and low fuel consump-tion – these goals appear in the specifications document for every Scania vehicle. Technically speak-ing, the electronic components and software play the main role, driving innovation towards these goals. Powerful brakes, electronic stability systems, camera surveillance and advanced driver assistance systems (ADAS) are typical examples of Scania’s commitment to improving traffi c safety, providing economic and environmental performance, and achieving maximum comfort. Modern Scania vehicles therefore carry a complex electrics/electronics (E/E) system in which a variety of ECUs work together.

Effi cient Modular System

The vehicles’ variety of tasks creates an even higher variety of challenges for vehicle developers. The only way to manage these development tasks and implement them economically is to use a sophisticated modular system. Depending on the require-ment, a specialized and often wholly unique vehicle is put together from a pool of main components (includ-ing the cab, engine, transmission and chassis).

Variant System for ECUs

While the mechanical components are confi gured differently to match the nature of the task, the E/E sys-tem, and particularly its ECUs, is generic in design. From trucks, to buses, to snowplows, all vehicles have the same E/E system, but fea-turing a modular approach regard-ing the ECUs actually needed. The vehicle-specifi c functions and func-tional characteristics of the ECUs are implemented by parameters derived from a vehicle confi gura-tion fi le (Scania Onboard Product Specifi cation, SOPS). This fi le is like a vehicle’s DNA, describing the con-fi guration and features to the last detail. The ECU variants created according to this variant method must be validated in all the in-stalled combinations.

T

rust is written with a capital ‘T’ in the transport and com-mercial vehicle industry because every day many people depend on reliability. High reliability and maxi-mum uptime are not the result of chance, though. They are the result of years of experience and work to-wards the goal of matching perfor-mance exactly to the requirements of day-to-day driving. These require-ments are quite different, with an application area ranging from smooth asphalt roads to muddy construction sites. Sometimes passengers are carried, sometimes goods. Some-times non-road mobile machinery needs to be moved safely.

Perfectly Matching Vehicles

Scania’s goal is to provide a perfectly matching vehicle for each task. The requirements from the fi elds of transportation, construction site traffi c (bulk cargo vehicles), municipal vehicles (garbage trucks, street sweepers, snowplows), passenger transport (buses) and special vehicles (fi re engines) are extremely different. Within these different fi elds, there are further segments, which in turn

Depending on the vehicle and features, optional ECUs are installed in addition to the basic ECUs.

CUSTOMERS

The E/E system of the vehicle fl eet consists of 93 ECUs. It is validated with a simulator system from dSPACE.

dSPACE Magazine 1/2015 · © dSPACE GmbH, Paderborn, Germany · info@dspace.com · www.dspace.com

Figure 1.1: ECU variation depending on vehicle types and features [1]

(18)

2 CHAPTER 1. INTRODUCTION

25/05/2019 draw.io

chrome-extension://pebppomjfocnoigkeepgbmcifnnlndla/index.html 1/1

Simulation Computer IntermediaryComponent ECU Under Test

ECU 1 ECU N CAN Bus I/Os I/Os Ethernet

Figure 1.2: HIL simulator building blocks

In the integration and system testing phase of the V-Model [3] product development life-cycle that Scania follows, the functionality of the ECU being developed needs to be validated against its requirement specification in a simulated environment. The well-established HIL method for ECU tests states that each ECU should be tested in a virtual representation of its controlled system for example engine, transmission, suspension, and comfort and driver assistance system [1].

To achieve this, a computer system to generate simulation data is needed. The simulator interacts with the ECU under test by writing and reading ECU’s Input/Output (I/O) pins. This mechanism is achieved by placing an intermediary component as a translator between the simulator and the ECU under test. The ECU under test is also connected to a CAN bus simulating a real vehicle setup. The basic building block of the HIL simulator can be seen in figure 1.2.

The intermediary component has main tasks with a real-time requirement to process the message in one millisecond. On top of that, this test equipment needs to be monitored to ensure that the system meets its requirement, and to know its resource usage status. Moreover, any error or warning message during the testing period needs to be logged centrally to minimize the troubleshooting time if an error occurs.

1.1

Project Description

This project introduces a logging and monitoring mechanism of the interme-diary component with minimal affection to the main tasks. The necessary software components both for the intermediary component and the

(19)

surround-CHAPTER 1. INTRODUCTION 3

ing distributed system are developed. Execution time measurements of the complete task are performed. Finally, a schedulability analysis result is provided as a guarantee that the system can meet the timing constraints under several parameters.

1.2

Motivation

The implementation of the logging and monitoring functionality will help the tester in seeing the system status both during the manual or automatic testing session. This information will increase the confidence level of the test result if the intermediary component meets its requirement and no error from the test equipment produced during the test. The monitoring data is also valuable to keep track of the uptime information of the test equipment as one of the Key Performance Index (KPI) of the EPSE department to maintain it as high as possible. In the end, the schedulability analysis result will help the EPSE department in optimizing the real-time system running on the intermediary component.

1.3

Problem Statement

The test configuration is dynamic based on the test environment that needs to be simulated. This condition affects the number of corresponding tasks and their execution time running on the intermediary component. Moreover, the additional low priority logging and monitoring tasks that are introduced also affect the task model of the intermediary component. The question is how to efficiently implement the centralized logging and monitoring mechanism for the parameterized real-time system, with guaranteed scheduling?

1.4

Purpose

The purpose of this project is to introduce logging and monitoring func-tionality for the intermediary component and to provide a guarantee that the system will meet its timing requirement while a various combination of parameters is used.

1.5

Goals

(20)

4 CHAPTER 1. INTRODUCTION

• Provide a non-intrusive implementation of logging and monitoring functionalities.

• Formulate an approach in validating the mixed-criticality system’s requirement.

1.6

Risks, Consequences, and Ethics

Scania has been producing heavy vehicles including buses and trucks which represent safety-critical systems. The correctness of the system in Scania product is critical as a minor mistake in the end product could cause material or environmental damage as well as harm persons inside and outside of the vehicle. One crucial factor in minimizing these risks is by doing a thorough testing procedure. Hence, having an accurate testing environment which simulates the real vehicle and environment as close as possible is extremely critical.

Before the project started, a non-disclosure agreement was signed. It stated that the author should not reveal any confidential information regarding the company’s research and development to the public. The risk in exposing this information in the report is addressed by ensuring that Scania approves the report before it gets published to the public.

1.7

Methods

Firstly, the surrounding infrastructure for logging and monitoring is prepared. The application is designed and proposed to the project’s owner. Upon approval, it is developed and deployed as a supportive environment to achieve the complete logging and monitoring system.

Secondly, the tasks needed on the intermediary component to deliver the logging and monitoring functionalities are developed. Several mechanisms are evaluated, and one final design is chosen.

Thirdly, these tasks are integrated with the main tasks that had been developed by EPSE. From this fully implemented system, execution time data are gathered as the necessary information for task modeling and analysis at a later stage. At this stage, the possible combination of parameters is tested to see its relation with the tasks’ execution time.

Finally, an empirical method [4] is used to provide a guarantee if the system will meet its real-time requirement based on the schedulability analysis of

(21)

CHAPTER 1. INTRODUCTION 5

the complete system.

1.8

Requirements

The implementation of the system has several requirements based on the currently running laboratory setup that the EPSE department has.

• The response time of the main tasks should be no longer than one millisecond.

• The number of main tasks and the corresponding message size ex-changed between the simulation computer and the intermediary com-ponent varies depending on the simulation configuration.

• The logging task should be able to push the log data to a central logging system.

• The monitoring task should be able to reply to a polling request which arrives every parameterized interval from a monitoring server. • The monitoring data should be available in real-time and archive mode. • The implementation of the tasks are in C++ and running on a

Real-Time Linux (RTLinux) [5] Operating System (OS).

• The intermediary component is a National Instruments (NI) Com-pactRIO 9030 [6].

1.9

Limitations

Due to the complexity in probing the kernel behavior on how it serves the real-time tasks, some practical factors are neglected. Hence, in doing the analysis, several assumptions have been made as listed below.

• Every task can be preempted at any time • Released tasks never suspend themselves

• Scheduling jitter and context switch time is negligible • Priorities assigned never change

(22)

6 CHAPTER 1. INTRODUCTION

1.10

Organization

This report is divided into five following chapters. Chapter 2 describes technology and terms used in this research as well as related research in the area of RTLinux. Chapter 3 details the design and implementation of the logging and monitoring system. Chapter 4 describes the methods used to measure the execution time of the system and a brief description of how the result will be derived. Chapter 5 discusses and analyzes the result of the execution time measurement using regression analysis and time demand analysis. Finally, chapter 6 summarizes the outcome of this research and suggests possible directions to be explored in future research.

(23)

Chapter 2

Background

In this chapter, relevant topics and areas are described. Mainly, it covers the technology and terms used in this thesis. At the end of the section, related works are mentioned to highlight how others solve a similar problem that this project addresses.

2.1

Electronic Control Unit

An ECU is a small brain controlling a single function of the vehicle. For example, there is one dedicated ECU to control the braking system of a truck. An ECU generally consists of an electronic board with several processors and I/O connectors. This complete component is packaged in an aluminum block [7] enclosure to protect the electronic part from external factors, which also acts as a heat sink for the electronic component. An ECU is connected to a CAN bus to communicate with the entire system of the vehicle. While the I/O connectors connect the ECU with sensors or actuators that the ECU needs to interact with.

2.2

Hardware-in-the-loop Simulation

One single ECU has its running software crafted specifically to deliver specific functionality. During the integration test, the ECU being developed is placed in a vehicle simulator. It is connected to a CAN network, and its I/O pins are connected to the simulator. With this setup, the ECU is conditioned

(24)

8 CHAPTER 2. BACKGROUND

23/05/2019 draw.io

chrome-extension://pebppomjfocnoigkeepgbmcifnnlndla/index.html 1/1

Transparency  

Control Environment ModelGlobal  

Interface Real Local Environment Controller Real System System outputs,    inputs and HWIL   sensor data   Demands Outputs Simulation Computer

Figure 2.1: Abstract view of a HIL local environment simulator for controller design, adopted from [8]

as it is running in a real vehicle. The behavior of the ECU software, which represents a particular control system, will be validated against its design specification during the simulation.

The testing scheme mentioned above is the basic idea of HIL simulation that includes a part of the real hardware in the simulation loop. It is a well-established technique used in the design and evaluation of control systems. In automotive industry, it is used for design of Anti-lock Braking Systems (ABS), Traction Control Systems (TCS), suspension systems and others

[8].

Figure 2.1 shows the abstract view of a HIL simulator with local environment. This figure relates to the HIL setup shown in figure 1.2 that this research is based on. The controller block shown in figure 2.1 is the ECU under test block in figure 1.2. While the real system block in figure 2.1 is represented by surrounding ECUs that are connected together in a CAN bus in figure 1.2. Finally, the intermediary component in figure 1.2 represents the interface in figure 2.1 which acts as a bridge between the simulation computer and the test equipment in providing real local environment.

(25)

CHAPTER 2. BACKGROUND 9

2.3

CompactRIO Controller

To communicate with the ECU under test, the simulation computer men-tioned in 2.2 needs an intermediary component. In this research, a Com-pactRIO 9030 is used as can be seen in figure 2.2. The complete specification of this controller is shown in table 2.1.

Table 2.1: CompactRIO 9030 technical specification, adopted from [9]

CPU Intel Atom E3825

Number of cores 2

CPU frequency 1.33 GHz

On-die L2 cache 1 MB (shared)

Operating system NI Linux Real-Time (64-bit)

Memory 1 GB DDR3L

Ethernet Ports 2x 1000 Mbps

FPGA Type Xilinx Kintex-7 7K70T

A CompactRIO controller is an industrial controller equipped with a single board computer, a user programmable Field Programmable Gate Array (FPGA), and several I/O extension cards. The computer is equipped with two ethernet ports and runs a RTLinux operating system that will be discussed in section 2.6. The RT-Linux application is able to interact with the I/O pins by communicating with the FPGA as illustrated in figure 2.3.

Page 2 | ni.com | CompactRIO Controllers

CompactRIO Controllers

cRIO-9063, cRIO-9064, cRIO-9065, cRIO-9066, cRIO-9067, cRIO-9068, cRIO-9030, cRIO-9031, cRIO-9032, cRIO-9033, cRIO-9034, cRIO-9035, cRIO-9036, cRIO-9037, cRIO-9038, and cRIO-9039

• Modular, high-performance embedded controllers for industrial deployment • NI Linux Real-Time OS for reliability, security, and determinism • Industrial-grade processing component options from ARM, Intel, and Xilinx • Time Sensitive Networking (TSN) enabled Ethernet ports

• Up to -40 °C to 70 °C temperature range, up to 50 g shock and 5 g vibration • 200+ I/O modules to support a variety of sensors and signal types

• LabVIEW graphical development platform eliminates the need for hardware description language (HDL) expertise to use reconfigurable FPGA hardware

Built for Embedded Control and Monitoring

CompactRIO Controllers are modular, high-performance embedded controllers for rapid development and deployment into industrial control and monitoring applications.

They feature extreme ruggedness, industry-standard certifications, modular I/O, and integrated motion, vision, industrial communication, and HMI capabilities. C Series modules deliver high-accuracy I/O with measurement-specific signal conditioning so you can connect to any sensor or device on any bus. Using the productive, integrated NI software toolchain, you can customize and reconfigure CompactRIO Controllers through software to rapidly deliver complex embedded control and monitoring systems.

Figure 2.2: The CompactRIO controllers, adopted from [6]

The core of this research is in ensuring that the tasks running on this controller meet their real-time requirements. The simulation computer produces User Datagram Protocol (UDP) messages every one millisecond.

(26)

10 CHAPTER 2. BACKGROUND

Page 4 | ni.com | CompactRIO Controllers

Heterogeneous Architecture

CompactRIO Controllers feature a heterogeneous architecture with two processing targets: (1) a real-time processor for communication and signal processing and (2) an FPGA for implementing high-speed control, custom timing and triggering, and signal processing directly in hardware.

Figure 1. Use the CompactRIO heterogeneous architecture to meet your processing needs.

Processor

The ARM or Intel processor runs NI Linux Real-Time, a real-time OS based on a standard kernel and designed specifically for reliable and deterministic operation in long-term deployments. It includes standard Linux features such as native support for Security-Enhanced Linux for security and reliability. FPGA

The user programmable FPGA allows you to offload time-critical processes for advanced control, signal processing, filtering, advanced timing, and other logic. NI uses the latest FPGA technology from Xilinx, with chips that contain up to 326,060 logic cells, and 840 DSP slices.

Figure 2. An FPGA contains a matrix of reconfigurable gate array logic circuitry.

Traditional development for FPGA-based systems requires the use of low-level software tools, HDLs, and vendor-specific FPGA implementation toolchains and constraint languages.

The LabVIEW FPGA Module provides a graphical programming approach that simplifies the task of interfacing to I/O and communicating data. It also eliminates the need to think through timing constraints, I/O configuration, and place and route settings.

Figure 2.3: CompactRIO heterogeneous architecture, adopted from [6]

These messages need to be interpreted by this controller, and it then set the ECU’s I/O pins accordingly through the FPGA interface. This process should be completed in one millisecond so that the subsequent messages can be processed on time. Similarly, the controller needs to read I/O pins and send the reading data back to the simulator every one millisecond. Additionally, the logging and monitoring tasks are implemented alongside the main tasks mentioned earlier.

2.4

User Datagram Protocol

UDP is a transport layer protocol used in conjunction with the Internet Protocol (IP) [10]. The protocol is usually mentioned as connectionless, where no connection establishment mechanism occurs before the actual transaction. The delivery of packets sent with this protocol is not guaranteed as there is no transmission control mechanism. However, this simple protocol is very suitable for a high-speed real-time communication scenario as used in HIL simulator mentioned in subsection 2.2 and 2.3 where packet dropping is preferred rather than delaying the whole communication by resending undelivered packets.

A UDP packet, as illustrated in figure 2.4, consists of 8 bytes header consists of source and destination port, message length, and checksum. It is encapsu-lated in an IP packet with 20 bytes of the header for addressing purpose. Thus, maximum 65,507 bytes left for the data field. In this research, the handling of this packet is done by the Linux kernel. The kernel provides an Application Programming Interface (API) to send and receive UDP packets. It means any packet that is successfully retrieved from the kernel is guaranteed its integrity due to the checksum checking. Otherwise, the packet will be dropped. For the message exchanging mechanism between the HIL simulator and the CompactRIO controller, a custom protocol was

(27)

CHAPTER 2. BACKGROUND 11

16­bit source port 16­bit destination port

16­bit UDP length 16­bit UDP checksum

Data

Figure 2.4: The UDP packet

developed to craft a special message that will be encapsulated inside the data field in this UDP packet.

2.5

Real-Time Systems

A system is classified as real-time if it has a requirement to complete the tasks on a timely basis. It means the correctness of the system does not only depend on the logical result of the computation but also on the time at which the results are produced [11, p. 1]. For example, in the HIL simulator use case, the intermediary component is correctly running if it can process the message and drive the I/O no more than one millisecond after receiving the UDP message. Even though the execution time of the whole process is less than one millisecond, if the task completes more than one millisecond after the arrival of the UDP message, then the system does not satisfy the timing constraints.

Buttazzo [11, p. 9] mentioned that real-time tasks are usually distinguished in three classes, hard, firm, and soft:

• Hard: If producing the results after its deadline may cause catastrophic consequences on the environment under control.

• Firm: If producing the results after its deadline is useless but does not cause any damage to the system or its environment.

• Soft: If producing the results after its deadline still have some utility although the overall performance of the system degrades.

Based on the classification mentioned above, the intermediary component system mentioned in section 2.3, is classified as a mixed-criticality system because it has two categories of tasks. Firm real-time tasks to handle the simulation messages and soft real-time tasks to handle the logging and

(28)

12 CHAPTER 2. BACKGROUND monitoring functionalities.

2.5.1

Real-Time Model

04/05/2019 draw.io chrome-extension://pebppomjfocnoigkeepgbmcifnnlndla/index.html 1/1 time Release Time   ri   Response Time Ri Relative Deadline Di Absolute Deadline   di   Completion Time

Figure 2.5: Real-time task workload model

A real-time system may have one or several tasks with a different priority. When a single CPU needs to execute multiple concurrent tasks, a scheduling policy is needed to assign the CPU to the various tasks. Thus, a task that may be executed on a particular CPU can be either in an execution mode if it is being executed by the CPU or in a waiting mode if the CPU is busy executing another task. In a common scheduling policy mechanism, if during execution, there is another task with higher priority, the task under execution can be preempted by the scheduler, and the CPU is assigned to the higher priority task. The preempted task can be continued later after the higher priority task completes its execution.

A feasible period of a task is the period when a particular task should be executed in order to satisfy its timing requirement, as illustrated in figure 2.5. The response time of a task (Ri) is the length of time from the task’s release time until its completion, including preemption time as indicated by the blue and yellow blocks. The execution time of a task (Ci) is the total independent time required by the CPU to complete a particular task alone and has all resources it needs as indicated by the blue blocks. A schedule is called a feasible schedule if it can produce a schedule in which all the tasks it holds complete before its deadline.

2.5.2

Schedulability Test

In order to provide a guarantee that a system will meet its timing require-ments, a schedulability test is required. There are two possible tests that

(29)

CHAPTER 2. BACKGROUND 13

could be used. They are the utilization based test and the time demand analysis.

The utilization based theorem states that if the system’s utilization is less than the least upper bound of the processor utilization factor, the system is schedulable. However, it does not know whether the system is schedulable if the system’s utilization is higher than the least upper bound of the processor utilization factor, but still less than 1.0. This test is only applicable to verify a task set scheduled rate-monotonically [11, p. 82-84].

On the other hand, the time demand or workload analysis is agnostic. It is used to compute the total demand for processor time by a task released at its critical instant, which is the time when a particular task is ready at the same time with the readiness of all other higher priority tasks. The time demand function of a task τi denoted with wi(t) is defined in equation 2.1. Bi is the longest blocking time due to shared resources usage that a particular task will experience at the critical instant. Whereas, T is the task’s deadline or period if the deadline is equal to the period. It is assumed that tasks are indexed by priority such that Pi−1> Pi [11, p. 109-115].

wi(t) = Ci+ Bi+ i−1 X k=1  t Tk  .Ck f or 0 < t < Ti (2.1)

A system consists of a set of periodic, independent, and preemptible tasks, is schedulable on one processor by a particular scheduling algorithm if for all the tasks i there exist t such that wi(t) is less than or equal to t for t is more than 0, and t is less than or equal to Ti as described by the following condition.

∀i : ∃t : wi(t) ≤ t, 0 < t ≤ Ti (2.2)

2.6

Real-Time Linux

Barbanov [12] began the development of RTLinux in 1997 as an effort in turning a feature rich Linux kernel to have hard real-time capabilities. Instead to have a dedicated Real-Time Operating System (RTOS) to handle real-time tasks and separate General Purpose Operating System (GPOS) to handle non-real-time tasks, the development of RTLinux was aiming to combine both based on readily available Linux kernel.

(30)

14 CHAPTER 2. BACKGROUND Hardware Emulation Layer Linux Kernel Real­Time Kernel Real­Time tasks Generic tasks

Figure 2.6: RT-Linux implementation architecture

The fundamental nature of the Linux kernel is to be used in a GPOS. It adopts time-sharing scheduling to serve user tasks fairly in order to keep the whole operating system run smoothly. The kernel tasks are also non-preemptible from userspace. These prevent it from being used as a RTOS.

The problem is addressed by putting an emulation layer between the Linux kernel and the interrupt controller hardware [12] as illustrated in figure 2.6. This emulation layer redirects all hardware interrupt to the designated kernel. In this scheme, the emulation layer has a critical role in prioritizing real-time tasks over the generic tasks even over the Linux kernel. The Linux kernel is running as a general task with idle priority [13]. This mechanism prevents the generic tasks and the Linux kernel to preempt the real-time tasks.

One of the several ways to turn a general Linux kernel into a fully preemptible one is by implementing PREEMPT_RT patch. The main aim of this patch is to minimize the amount of kernel code that is non-preemptible [14]. The technical details on how to implement this patch can be seen in [15]. One can also build a custom Linux distribution utilizing this patch using Buildroot [16] or Yocto [17]. NI has also worked to use the PREEMPT_RT patch to create NI Linux Real-Time using Yocto, which is the RTOS that is used in this research and will be mentioned simply by RTLinux.

The other way is a co-kernel solution called Xenomai [18]. It is a small real-time kernel which separates the scheduling of time-critical threads with the Linux’s main kernel logic.

2.6.1

Scheduling Policy

(31)

CHAPTER 2. BACKGROUND 15

• SCHED_FIFO: Valid priority between 1 (lowest) and 99 (highest). Under this policy, a task is executed until it finishes or a higher priority task preempts it. The system used for this research was configured using this policy.

• SCHED_RR: This policy is derived from SCHED_FIFO policy. The difference is a task runs for the duration of a defined time slice. Then another task with the same priority can be run, or any other higher priority task can preempt it.

• SCHED_DEADLINE: This policy implements Global Earliest Dead-line First algorithm. Any task runs under this policy can preempt any task scheduled with SCHED_FIFO or SCHED_RR. The preemption mechanism of this policy is based on the nearest deadline of each task, which is evaluated in run-time.

2.6.2

Priority Inheritance

When a task is blocked by one with a lower priority due to a mutually exclusive resource, the third task with a priority in between those which does not need to access the resource can run and finish before the task with the highest priority resumes. This problem is called priority inversion and RTLinux resolves this by a priority inheritance technique [20].

Under this implementation, referring to the scenario above, the lower priority task holding the resource inherits the higher priority task’s priority such that the resource can be released sooner. Then the higher priority task can continue to run immediately even though a task in between them is ready. In summary, the system maintains the rule that the highest priority task always runs under all circumstances.

The blocking time of the priority inheritance theorem states that a task can be blocked only once on each different semaphore by another low priority task for at most the duration of one critical section [11, p. 213]. The blocking time consists of direct blocking, which is caused by shared resources that the task is using, and indirect blocking, which is caused by shared resources that the higher priority tasks use. The maximum value of several identified blocking times is considered as the maximum blocking time of a particular task.

(32)

16 CHAPTER 2. BACKGROUND

2.7

Regression Analysis

Regression analysis is a statistical method to investigate whether there is a relationship between two or more variables. In this research, this method was used to estimate the execution time of a task based on the supplied parameters. There are three types of regression: simple linear regression, multiple linear regression, and nonlinear regression [21, p. 2-3]. The last is out of the scope of this research.

2.7.1

Simple Linear Regression

Simple linear regression is used to investigate the linear relationship between one dependent variable y and one independent variable x. For example, it can predict the CPU temperature y based on the supplied CPU usage information x. The predicted linear function of this model is often written as the following form.

b

y = b0+ b1x (2.3)

Wherey is the estimate of a dependent variable y, which is the value thatb

we want to predict. b is the slope of the regression line, and b0 is the b

y-intercept. A least square estimation method is often used to find these

values by choosing the regression line that is the most "closest" line to all data points (xi, yi). The following formula can be used to find the value of b1 and b0 which represent the best-fitted line to model the relationship [21, p. 10-12]. b1= Pn i=1(yi− ¯y)(xi− ¯x) Pn i=1(xi− ¯x)2 (2.4) b0= ¯y − b1x¯ (2.5)

2.7.2

Multiple Linear Regression

Multiple linear regression is a linear regression model with one dependent variable and more than one independent variables. The prediction form of this model is usually presented as follows:

b

(33)

CHAPTER 2. BACKGROUND 17

whereby is the estimate of a dependent variable y, b0, b1, b2, . . . , bpare regres-sion coefficients, and x1, x2, . . . , xp are independent variables. This model can also be expressed in the matrix format

b y = Xb (2.7) where X =     x11 x12 . . . x1k x21 x22 . . . x2k . . . xn1 xn2 . . . xnk     b =       b0 b1 b2 . . . bk−1       (2.8)

The value of b can be obtained using the least squares principle as follows [21, p. 58-62]:

b = (X0X)−1X0by (2.9) where b0 = (b0, b1, . . . , bk−1)0, a k-dimensional vector of the estimations of the regression coefficients.

2.8

Web Technology

In order to deliver the complete requirement of this research, the surrounding environment for logging and monitoring purposes needs to be built. This environment is a small, distributed system that is built using the following technology.

2.8.1

ELK Stack

ELK stands for Elasticsearch, Logstash, Kibana, which are open-source projects managed by Elastic [22]. Elasticsearch is a distributed search engine based on JavaScript Object Notation (JSON). Due to its scalability, flexibility, and easiness to use, this software has been gaining popularity among users and corporations. Logstash is used for server-side data processing pipeline. It ingests data from a multitude of sources and performs rule-based pipeline on it. Kibana visualizes Elasticsearch data in intuitive ways. The combination of this three software forms an ELK stack.

(34)

18 CHAPTER 2. BACKGROUND Application 3 Application 1 Logstash Application 2 Pipeline 2 Elasticsearch Kibana Pipeline 1

Figure 2.7: An example of ELK stack topology

An example of how the ELK stack works can be seen in figure 2.7. The Logstash server runs two pipelines. The first pipeline accepts UDP packets containing JSON data in port 20001, transforms one of its fields using some predefined rule, then sends out the modified JSON data to the Elasticsearch. The second pipeline accepts TCP packets containing syslog message via port 20002. It then parses the syslog message into a JSON data, adds a timestamp field, and finally sends the newly constructed JSON data to the Elasticsearch. The Kibana server queries the data from the Elasticsearch database through Representational State Transfer (REST) APIs that the Elasticsearch provides then visualizes it in a nice dashboard view.

2.8.2

ASP.NET Core

ASP.NET Core is the open-source and cross-platform version of the ASP.NET popular web-development framework for building web apps [23]. The original ASP.NET can only be run in Microsoft Web Server running on a specific Microsoft Windows OS, which is popular among enterprises. In an attempt in getting open-source and Linux users aboard, Microsoft released the first version of ASP.NET core in 2016. The .NET ecosystem is prevalent due to its maturity, and a vast amount of libraries which make the software development using this platform keeps productive.

2.8.3

SignalR

SignalR is a real-time WebSocket [24] library for ASP.NET application [25]. A general website loads a page from the server, and the opened page needs to be refreshed in order to see an update from the server side. A SignalR-enabled website creates a WebSocket connection with the server upon page loading. At this state, the server side program is able to push content to all connected clients. The pushed content will be received by the client

(35)

CHAPTER 2. BACKGROUND 19

immediately without the need to refresh the whole page. The client then updates the User Interface (UI) accordingly using JavaScript [26] code that has been previously retrieved during page loading.

2.8.4

Single Page Application

A common website type is a dynamic website. Upon a client request, this type of website will render the requested page dynamically. It will then return the requested page as a bundle of Hyper Text Markup Language (HTML) [27] which defines the meaning and structure of web content, Cascading Style Sheets (CSS) [28] which describes the presentation of HTML, and JavaScript code to interact with the first two components programmatically. Another page request will be treated in the same manner by the server.

A Single Page Application (SPA) takes a different approach by loading all the necessary component when the website initially loads. User interactivity is handled locally by the JavaScript code. If a query is needed, the client will use the API that the server exposes. It decouples the logical implementation of frontend and backend then connect them using a sort of API. With this scheme, the page load time decreases significantly, and the website is more responsive in handling user interactivity. This SPA type is now the current trend in building an application as the backend component can be written once to serve a different kind of frontends, for example, a SPA website and a mobile application.

2.8.5

Container

A container is a standardized unit of software [29]. It is the current approach in bundling an application into a standard package complete with its de-pendencies and run-time environment. The container can be instantiated just like an executable and can be scaled up easily. As an example, in the ELK stack use case mentioned in section 2.8.1, instead of preparing three different run-time environments in the form of virtual machines, the ELK stack can be containerized into three different containers without worrying each guest OS setup.

The difference between containers and virtual machines architecture is illus-trated in figure 2.8. With a container, applications share an OS, which also possibly includes binaries and libraries. As a result, this type of deployment will be significantly smaller in size compared to the virtual machine way [30].

(36)

20 CHAPTER 2. BACKGROUND

(a) Containers (b) Virtual Machines

Figure 2.8: Containers and virtual machines comparison, adopted from [29]

2.9

Related Works

Several studies have been done in evaluating the performance of PRE-EMPT_RT patch in turning a general Linux OS into a RTOS. Fayyad, Perneel, and Timmerman [31] analyzed the performance gap between Linux PREEMPT_RT and commercial RTOSs. They compared the performance of RTLinux with QNX and Windows Compact 7 in terms of thread switch latency, interrupt latency, sustained interrupt frequency, and semaphore and mutex acquire-release timing in contention case. The results showed that the performance difference between the tested RTLinux and the best performing commercial RTOS is still more than a factor of two. They believed that RTLinux system is starting to be a competitor against commercial RTOSs. It is interesting to see how is the current PREEMPT_RT performance as the study was done six years ago.

Regnier, Lima, and Barreto [32] evaluated the interrupt handling timeliness in RTLinux. They showed that Linux PREEMPT_RT provides temporal guarantees with a precision below 20 µs. This shows that the 1 ms task period requirement mentioned in section 1.8 is feasible to be achieved with the RTLinux RTOS.

An interesting technique to reduce network interrupt latency in RTLinux has been demonstrated by Lee and Kim [33]. They showed the algorithm and implementation of how to identify a special packet that requires

(37)

real-CHAPTER 2. BACKGROUND 21

time treatment and process it inside the top-half interrupt priority without delaying.

Lisper [34] described a technique to fully automate parametric Worst-Case Execution Time (WCET) analysis that finally yield a final WCET calculation. The method uses known methods for parametric calculation. This is a fascinating technique to acquire WCET information as an important part in validating system’s timing constraints in this research. However, due to its complexity, another method called dynamic timing analysis [35] is chosen to do WCET estimation in this research.

Baruah, Burns, and Davis [36] used the same time demand analysis technique as used here. However, instead of evaluating a fixed-priority system as the context of this project, they used it to study two dynamic priority scheduling algorithms performance. Static mixed criticality (SMC) that assumes limited run-time support, and adaptive mixed criticality (AMC) that requires additional run-time support.

(38)

Chapter 3

Implementation

25/05/2019 draw.io chrome-extension://pebppomjfocnoigkeepgbmcifnnlndla/index.html 1/1 Logging & Monitoring Server   Ubuntu Server                                                 CompactRIO Controller   RTLinux                                                     I/O Server                       Real­Time Monitoring                   Logging Infrastructure             UDP Receiver Task 1

Logstash Elasticsearch Kibana

Backend     Kestrel   ASP.NET Core   SignalR   Frontend     NGINX   Vue.js   SignalR   UDP Receiver Task n UDP Sender   Task 1 UDP Sender   Task n Logging Task Monitoring Task System Load Measurement Task Logging & Monitoring Agent

Figure 3.1: High-level overview of the complete system architecture

Figure 3.1 shows how the complete system was implemented. The system can be broken down into three main categories. The first is the logging and monitoring agent which is RTLinux tasks that section 3.2 covers. The third is the main tasks which have a strict timing requirement, the I/O server tasks, which will be described in section 3.3. The second is the logging and monitoring infrastructure that will be discussed in section 3.1. The data flow diagram illustrated in figure 3.2 will be discussed in the component level in the following sections.

(39)

CHAPTER 3. IMPLEMENTATION 23 13/06/2019 draw.io chrome-extension://pebppomjfocnoigkeepgbmcifnnlndla/index.html 1/1 Polling Request Monitoring Data Monitoring Data   Backend Frontend   Logging    & Monitoring   Data   Logstash   Log Data I/O Values I/O Values I/O Server   Monitoring Data Log Data Logging &    Monitoring Agent   Kibana   Logging    & Monitoring   Data Elasticsearch Simulation Data Simulation Computer I/O Values ECU Under Test

Figure 3.2: Data flow diagram of the complete system

3.1

Logging & Monitoring Server

The purpose of this component is as the centralized logging and monitoring infrastructure for all the intermediary components. The physical server that hosts the application that will be described in the following subsections, was located in a network that can be reached by the intermediary components. The UDP was used as the primary communication protocol between the server and the intermediary components.

3.1.1

Backend

The backend application is responsible for getting the intermediary compo-nent’s status. It stores the database of devices that needs to be monitored. Several threads are created based on the number of devices. Each thread is doing a polling job every a specific interval, by sending a UDP packet to the intermediary component’s address and port that has been specified in the devices database. The complete flow of the polling request is illustrated in figure 3.3, while the content of the packet can be seen in figure 3.4. The polling id is generated randomly for a particular poll request to keep track of a specific reply. A functioning client will reply with the same header data followed by the monitoring data in JSON. The data will be forwarded to the Logstash endpoint as described in section 3.1.3 as well as to any connected

(40)

24 CHAPTER 3. IMPLEMENTATION SignalR client. 13/06/2019 draw.io chrome-extension://pebppomjfocnoigkeepgbmcifnnlndla/index.html 1/1 Send Polling   Request Receive reply   before    timeout Craft offline data

Craft online data  data to LogstashSend monitoring  Push monitoring data toSignalR clients T

F

Figure 3.3: Backend process flow

Header  3 bytes  Polling ID  4 bytes  Delimiter  1 byte  OK? 7182 #

Figure 3.4: Polling packet format

Other than that, the backend also exposes several Hyper Text Transport Protocol (HTTP) APIs to serve the frontend which basically manage the registered devices and act as an endpoint of the SignalR server from which the real-time monitoring data will be coming from.

The backend application was implemented using ASP.NET core Model-View-Controller technology as the primary web framework. It also exposes a

WebSocket server using SignalR to serve the real-time monitoring

function-ality. It runs on top of a HTTP server named Kestrel as the built-in web server in ASP.NET core technology.

3.1.2

Frontend

The frontend component was built using a Single Page Application (SPA) method. Vue.js JavaScript framework was utilized as the main component in building the SPA. It communicates with the backend through HTTP REST API that the backend exposes to retrieve the registered devices. To get the real-time monitoring data, the SignalR client JavaScript library was used to communicate with the WebSocket server that the backend exposes. The monitoring data is displayed in a real-time dashboard that updates promptly as the data being pushed by the backend. NGINX was used as the HTTP server to serve the static content of the SPA.

(41)

CHAPTER 3. IMPLEMENTATION 25

3.1.3

ELK Stack

1 input { 2 udp { 3 id => " main_udp_listener " 4 codec => " j s o n " 5 port => 7000 6 } 7 } 8 9 f i l t e r { 10 date {

11 match => [ " LocalTimestamp " , "UNIX " ] 12 t a r g e t => " Timestamp " 13 remove_field => [ " LocalTimestamp " ] 14 } 15 } 16 17 output { 18 e l a s t i c s e a r c h { 19 h o s t s => [ " e l a s t i c s e a r c h : 9 2 0 0 " ] 20 index => "%{Type}−%{+YYYY.MM. dd }" 21 } 22 }

Listing 3.1: Logstash configuration

The implementation of the ELK stack was quite similar to what has been mentioned in section 2.8.1. As seen in figure 3.1 and listing 3.1, the Logstash server opens one main UDP listener on port 7000. The pipeline accepts a JSON document as described in listing 3.2 and 3.3. Then, if a UNIX time field is found, it will be converted to a format that Elasticsearch understands. It then set the index field of the data based on the Type field content concatenated with the current date. Finally, it pushes the modified JSON data to the Elasticsearch on port 9200.

(42)

26 CHAPTER 3. IMPLEMENTATION 1 { 2 Type : " mondata " , 3 I s O n l i n e : true , 4 CPUUsage : 21 , 5 CPUTemp: 36 , 6 MemUsage : 41 , 7 NetUsage : 0 , 8 OverrunCount : 0 9 }

Listing 3.2: Monitoring data

1 { 2 Type : " l ogdata " , 3 DeviceName : " cRIO " , 4 LocalTimestamp : 15532835 , 5 S e v e r i t y : "ERROR" , 6 F i l e : " main . c " , 7 Line : 78 ,

8 Message : " Overrun \ dots " 9 }

Listing 3.3: Logging data

3.1.4

Deployment

Docker container was utilized for packaging the backend and the frontend application, as well as the ELK stack mentioned earlier. All of these appli-cations run in their run-time environment and communicate with each other using the internal docker network. The containers run inside an Ubuntu server which hosts the docker application. Several ports are exposed using the server’s network port such as the main NGINX HTTP port (80) expos-ing the frontend application and the Kibana port (5601) exposexpos-ing its web interface. Screenshots showing the logging and monitoring dashboard can be seen in appendix C.

3.2

Logging & Monitoring RTLinux Tasks

The logging and monitoring agent is an independent executable that runs on the intermediary component, which has a low priority. It consists of three periodic tasks that will be described in the following subsections.

3.2.1

Architecture

The logging and monitoring agent’s architecture is illustrated in figure 3.5 in the purple block. It has three periodic tasks for sending log messages to the Logstash server, replying the monitoring request, and measuring the system load. They communicates with the I/O server executable through a POSIX message queue and POSIX shared memories. While for the communication between tasks, a mutex-protected shared variable is used.

(43)

CHAPTER 3. IMPLEMENTATION 27 25/05/2019 draw.io chrome-extension://pebppomjfocnoigkeepgbmcifnnlndla/index.html 1/1   Logging Task Monitoring Task System Load Measurement Task POSIX   Message Queue   POSIX   Shared Memory Logging & Monitoring Agent Monitoring Data     UDP Receiver Task n UDP Sender   Task 1 UDP Sender   Task n POSIX   Message Queue   POSIX   Shared Memory Error   Counter   Log   Message   UDP Receiver Task 1 I/O Server Variant Collection

Figure 3.5: Logging and monitoring agent tasks implementation diagram

13/06/2019 draw.io

chrome-extension://pebppomjfocnoigkeepgbmcifnnlndla/index.html 1/1

Receive  

 Polling Request Monitoring DataRead   Polling ReplyCreate   Monitoring ServerSend Reply to

First Measurement 1000 msSleep   

Second Measurement Monitoring DataWrite    Get Log Data from    Message Queue T F Empty Sleep   Period ­ Exec. Time   Send to Logstash Sleep   Period ­ Exec. Time   Sleep   Period ­ Exec. Time  

Figure 3.6: Logging task process flow

3.2.2

Logging Task

The logging task is responsible for pushing log data which reside in the message queue to the Logstash server as illustrated in figure 3.6. When it executes, it starts by trying to get a message from the message queue. If it is not empty, the message will be forwarded to the Logstash. It repeats this process until all messages have been successfully pushed to the Logstash. Hence, the execution time of this task depends on how many messages are available in the message queue.

3.2.3

System Load Measurement Task

This task is responsible for preparing the monitoring data as listed in listing 3.2. How this task was implemented can be seen in figure 3.7. A first value provided by the kernel, needs to be captured and be compared to the second value that will be recorded a few moments later to calculate, for example, the CPU usage time and network utilization. It also reads the error counter, and the message missed counter to be appended in the monitoring data.

(44)

28 CHAPTER 3. IMPLEMENTATION

Finally, the formatted data is written in the monitoring data shared variable that will be read later by the monitoring task. The red block indicates that it accesses a mutually exclusive shared resource.

13/06/2019 draw.io chrome-extension://pebppomjfocnoigkeepgbmcifnnlndla/index.html 1/1 Receive    Polling Request Read    Monitoring Data Create    Polling Reply Send Reply to Monitoring Server First Measurement 1000 msSleep   

Second Measurement Monitoring DataWrite    Get Log Data from    Message Queue T F Empty Sleep   Period ­ Exec. Time   Send to Logstash Sleep   Period ­ Exec. Time   Sleep   Period ­ Exec. Time  

Figure 3.7: System load measurement task process flow

3.2.4

Monitoring Task

The monitoring task is responsible for listening to a specific socket connection to receive a monitoring request. Upon receiving the polling request, the updated monitoring data will be read, and a polling reply will be crafted. It then sends the reply via the same socket connection which was initially created by the monitoring server.

13/06/2019 draw.io

chrome-extension://pebppomjfocnoigkeepgbmcifnnlndla/index.html 1/1

Receive  

 Polling Request Monitoring DataRead   Polling ReplyCreate   Monitoring ServerSend Reply to

First Measurement 1000 msSleep   

Second Measurement Monitoring DataWrite    Get Log Data from    Message Queue T F Empty Sleep   Period ­ Exec. Time   Send to Logstash Sleep   Period ­ Exec. Time   Sleep   Period ­ Exec. Time  

Figure 3.8: Monitoring task process flow

3.3

I/O Server RTLinux Tasks

The I/O server tasks are periodic high priority tasks which bridge the communication between the simulation computer and the ECU under test. These tasks have a one millisecond timing requirement to complete its execution. To communicate with the agent tasks which are in a different executable, Portable Operating System Interface (POSIX) message queue and POSIX shared memory Interprocess Communication (IPC) mechanism offered by the Linux kernel were utilized.

(45)

CHAPTER 3. IMPLEMENTATION 29

3.3.1

Architecture

Table 3.1: Logging mechanism execution time comparison

Mechanism Execution Time (µs) % of Deadline

Syslog 174.7 17.47

UDP socket 74.8 7.48

Message queue 29.7 2.97

The decision to offload the logging functionality to another low priority task was based on several communication mechanism measurements. The test was done to send 320 bytes of data 1000 times then the upper-bound execution time was obtained as shown in table 3.1. The syslog message is stored locally in the system’s non-volatile memory, and it took the highest execution time that consumes 17.47 % of the task’s requirement. If the logging message is sent directly from the I/O server tasks, it consumes 7.48 % of its period. The best decision is to offload the logging mechanism by sending the log messages to a message queue which consumes 2.97 % of its requirement and let another low-priority periodic task handle it later.

04/05/2019 draw.io chrome-extension://pebppomjfocnoigkeepgbmcifnnlndla/index.html 1/1   Logging Task Monitoring Task System Load Measurement Task POSIX   Message Queue   POSIX   Shared Memory Logging & Monitoring Agent Monitoring Data     UDP Receiver Task n UDP Sender  

Task 1 UDP SenderTask n  

POSIX   Message Queue   POSIX   Shared Memory Error   Counter   Log   Message   UDP Receiver Task 1 I/O Server Variant Collection

Figure 3.9: I/O server tasks implementation diagram

The final implementation of the I/O server tasks is illustrated in figure 3.9. There can be n number of receiver tasks and n number of sender tasks based

(46)

30 CHAPTER 3. IMPLEMENTATION

on the testing configuration. The receiver tasks receive messages from the simulation computer, decode them, and set the FPGA I/O accordingly. On the other hand, the sender tasks read the I/O value from the FPGA, encode reply messages, and push them back to the simulation computer.

In between these tasks, there is a shared variable called variant collection that is used as a synchronization control. If the task is overrun or any missing message from the simulation computer is detected, it will increment counters value stored in POSIX shared memory. This value will be read regularly by the agent tasks that will be reported to the monitoring server. The access to the variant collection shared variable and the shared memory is protected by a binary mutex. Hence, the access time to these resources is considered as blocking time to other high priority tasks.

3.3.2

UDP Receiver Tasks

Figure 3.10 shows the main loop of the UDP receiver tasks. The red blocks indicate that this process is accessing a mutex-protected shared resource. The loop starts by retrieving the message from the socket buffer. Inside the payload, there exists a message id that is incremented periodically. This data is used to detect if there is a missing message during transmission. If a missing message is detected, the message missed counter that is located in the shared memory is incremented. Then the message will be decoded, and the result will be stored in the variant collection shared variable. If there is an error, it will be logged by sending the log message to the message queue. If overrun is detected, the overrun counter will be incremented.

13/06/2019 draw.io chrome-extension://pebppomjfocnoigkeepgbmcifnnlndla/index.html 1/1 Receive    UDP Message Decode R/W to    variant collection T F Error   Occurs F T Prev.   Message   Missed   Send a log to    Message Queue Increment Message Missed Counter

Read FPGA I/O variant collectionR/W to  

Set FPGA I/O Encode Send UDP message T F Error   Occurs Send a log to    Message Queue T F Overrun   Detected   Increment    Overrun Counter T F Overrun   Detected   Increment    Overrun Counter Sleep   Period ­ Exec. Time   Sleep   Period ­ Exec. Time  

References

Related documents

It is worth noting that the strange term in the local problem appears with a coefficient ρx/ε with spatial oscillations with the same frequency as the heat conductivity coefficient

The algorithms; random forests, support vector regression and artificial neural networks, are tested in research about residential real estate, but the area of commercial real

Free elevators are parked in different sectors to provide fast service for pas- sengers.An elevator car that is assigned to a sector where no calls are made parks itself at the

The key difference is that the main board requires a reserve power source to properly shut down the circuit and a function board is always powered by battery source.. When the

635, 2014 Studies from the Swedish Institute for Disability Research

The r ythm should be slow on all the reg istr y and it should be felt as a transition for what comes

We have seen several similarities between the Joker and Iago: they have a high sense of superiority; they question morality and goodness; they do not approve of self- sacrifice as

The other respondent that said that he did not send videos due to them containing different brands, later gave us an example where he sent a Pepsi commercial video with