Monitored Neural Networks for Autonomous Articulated Machines

V¨

aster˚

as, Sweden

Thesis for the Degree of Master of Science in Engineering

-Dependable Systems 30.0 credits

MONITORED NEURAL NETWORKS

FOR AUTONOMOUS ARTICULATED

MACHINES

Erik Beckman

Erik-beckman@hotmail.com

Linus Harenius

Linusharenius@gmail.com

Examiner: Masoud Daneshtalab

M¨

alardalen University, V¨

aster˚

as, Sweden

Supervisor: Anas Fattouh

M¨

alardalen University, V¨

aster˚

as, Sweden

Company

Supervisor: Viktor Gustavsson

Volvo Construction Equipment, Eskilstuna,

Sweden

Abstract

Being able to safely control autonomous heavy machinery is of uttermost importance for the conver-sion of traditional machines to autonomous machines. With the continuous growth of autonomous vehicles around the globe, an increasing effort has been put into certifying autonomous vehicles in terms of reliability and safety. In this thesis, we will investigate the problem with a deviation from the planned path for an autonomous hauler from Volvo Construction Equipment. The autonomous hauler has an error within the kinematic model, the feed-forward curvature-steering controller, due to a slip-effect that comes with the third wheel-axle. The deviation can especially be seen in sharp curves, where the deviation needs to be decreased in order to make the autonomous hauler more dependable and achieve an increased accuracy when following any given path. The aim of the thesis is to develop a fully functional Artificial Neural Network that has a new steering angle as output. The hypothesis for this thesis is to use an ANN to mimic the steering of a human driver, since a real driver compensates for the slipping behavior; both because the operator knows where on the road the machine is and also in the way that a human thinks many steps ahead whilst driving. This proposed ANN will have a monitor function which ensures that the steering angle command operates within its boundaries. Hence this thesis implies that it is indeed possible to ensure that the ANN performs reliably with the help of a monitor function in a simulated environment and can thus be used in dependable systems.

Sammandrag

Att s¨akert kunna kontrollera sj¨alvk¨orande anl¨aggningsfordon ¨ar av yttersta vikt f¨or att kunna omvandla traditionellt styrda fordon till sj¨alvk¨orande fordon. Den kontinuerliga tillv¨axten av forskning inom sj¨alvk¨orande fordon runt om i v¨arlden har medverkat till en ¨okad anstr¨angning f¨or att certifiera dessa fordon med h¨anseende till tillf¨orlitlighet och s¨akerhet. I den h¨ar avhandlingen kommer vi att unders¨oka problemen med avvikelsen fr˚an den planerade rutten f¨or en autonom dumper fr˚an Volvo Construction Equipment. Den autonoma dumpern har ett fel i den kinematiska modellen p˚a grund av slirningen som uppst˚ar av den tredje hjulaxeln. Avvikelsen ¨ar tydligast i skarpa kurvor och m˚aste minskas f¨or att g¨ora den autonoma dumpern mer tillf¨orlitlig och uppn˚a en ¨

okad precision vid k¨orning. Syftet med avhandlingen ¨ar att utveckla ett fullt funktionellt Artificiellt Neuralt N¨atverk som skickar ut en ny styrvinkel till maskinen. Hypotesen f¨or avhandlingen ¨ar att det ska vara m¨ojligt att anv¨anda ett ANN f¨or att h¨arma en m¨ansklig f¨orares styrning, eftersom en m¨anniska kompenserar f¨or slirningen - dels f¨or att en m¨anniska ¨ar medveten om var den ¨ar p˚a en v¨ag och dels f¨or att en m¨anniska t¨anker flera steg i f¨orv¨ag. Det f¨oreslagna neurala n¨atverket ska ha en ¨overvakningsfunktion som ser till att styrvinkelkommandot opererar inom satta begr¨ansningar. Denna avhandling p˚avisar att det ¨ar m¨ojligt att s¨akerst¨alla att ett ANN presterar p˚alitligt med hj¨alp av en ¨overvakningsfunktion i en simulerad milj¨o och kan d¨armed anv¨andas i ett system med tillf¨orlitlighetskrav.

Acknowledgement

This thesis would not have been possible without Volvo CE. We would like to express our gratitude to Volvo CE for giving us the opportunity for the thesis work and for providing us with the materials needed to fulfill the objectives of the thesis. We would not have been able to accomplish this thesis without our great company supervisor Viktor Gustavsson. We want to emphasize our appreciation for his availability and his expertise in articulated vehicles. We would also like to express our gratitude to our university supervisor Anas Fattouh, for his continuous guidance throughout the whole thesis and his knowledge in the field of neural networks.

Acronyms

AAH Autonomous Articulated Hauler. 12, 16, 24, 26, 33, 34, 36, 37 ANN Artificial Neural Network. 1, 2, 8, 9, 16, 22–26, 31–37

CPU Central Processing Unit. 9 DNN Deep Neural Network. 14, 15 DOF Degrees Of Freedom. 4 ECU Electronic Control Unit. 3 ESD Electrostatic Discharge. 34

GNSS Global Navigation Satellite System. 2–4 GPS Global Positioning System. 4, 9, 19, 22, 32–34 GPU Graphics Processing Unit. 9

IMU Inertial Measurement Unit. 2, 3, 9 LiDAR Light Detection and Ranging. 3 MPC Model Predictive Control. 7

NN Neural Network. 9, 13, 14, 17, 22–26, 28, 31–34, 37 PID Proportional-Integral-Derivative. 1, 2, 7, 16, 36 POD People and Obstacle Detection. 3

RADAR Radio Detection and Ranging. 3

RAIM Receiver Autonomous Integrity Monitoring. 34 RAM Random Access Memory. 24

RTK Real Time Kinematic. 2, 3

TMR Triple Modular Redundancy. 14, 33, 34 UAV Unmanned Aerial Vehicle. 13

List of Symbols

Symbol Description Phonetically

α Angle βlong Alpha (lower case)

s Arc length Latin

∆x Change in x-axis Delta (upper case)

∆y Change in y-axis Delta (upper case)

κ Curvature Kappa (lower case)

κpath Curvature of the reference path Kappa (lower case)

κmachine Curvature of the actual machine’s movement Kappa (lower case)

∂ Derivative Cursive d (mathematically)

σ A function expressed as σ(x) Sigma (lower case)

ψ0 Heading-angle on time zero Psi (upper case)

˙

ψ0 Heading-angle derivative Psi (upper case)

ψ1 Heading-angle on time instance one Psi (upper case)

˙

ψ1 Heading-angle derivative on time instance one Psi (upper case)

ϕ Heading-angle error Phi (lower case)

`e Lateral error Cursive l (mathematically)

L0 Length of front Latin

L1 Length of rear Latin

¯

x1s Position x under slip conditions Latin

¯

y1s Position y under slip conditions Latin

Pr(Xr, Yr) Position for rear body at (x,y) Latin

Pf(Xf, Yf Position for front body at (x,y) Latin

r Radius of circle Latin

δ Steering angle Delta (lower case)

˙δ Steering angle velocity Delta (lower case)

γ Steering angle from source Gamma (lower case)

β Slip angle Beta (lower case)

βlat Slip angle in latitude vector Beta (lower case)

βlong Slip angle in longitudinal vector Beta (lower case)

βf Slip angle for Front body Beta (lower case)

βr Slip angle for Back body Beta (lower case)

θf Slip angle for front body from source Theta (lower case)

θr Slip angle for rear body from source Theta (lower case)

t Time Latin

v Velocity Latin

v0 Velocity on time zero Latin

v1 Velocity on time instant Latin

¯

v1s Velocity under slip condition Latin

x0, y0 x and y coordinate starting position Latin

˙

x0, ˙y0 x and y coordinate starting position Latin

ˆ

x1, ˆy1 x and y coordinate predicted position Latin

List of Figures

1 Volvo Hauler [1] . . . 1

2 Principle sketch of the two interception points between the bodies . . . 2

3 Hauler [1] . . . 3

4 Wheel Loader [1] . . . 3

5 Excavator [1] . . . 3

6 Volvo CE’s autonomous electric site concept [1] . . . 4

7 Volvo Hauler [1] . . . 4

8 Principle sketch of an articulated machine with two bodies and two axles . . . 5

9 Principle sketch of a hauler following a curvature . . . 6

10 Principle sketch of the one interception point between the bodies . . . 7

11 Principle sketch of Deep Neural Network Layers . . . 8

12 Model of a front-wheel steered vehicle inspired from [2] . . . 10

13 Model of an articulated vehicle [3] . . . 11

14 Principle of side-slip for a wheel loader inspired by equation [4] . . . 12

15 Dependability in coherence with some neural network metrics [5] . . . 14

17 Flow Diagram of Multi-methodological Research Approach [6] . . . 18

18 Illustration of how Beta long and Beta lat is calculated . . . 19

19 Illustration of Psi and Phi . . . 20

20 Phi illustration . . . 20

21 Illustration of κmachineand κpath . . . 21

22 Test Track . . . 22

23 Oval Track . . . 22

24 Max turn Track . . . 22

25 Simplified NN Architecture 2x4 instead of 10x10 . . . 23

26 MATLAB Regression plot, from left to right 5x5, 7x7, 10x10 and 15x15 architectures 23 27 Neural Network architecture in MATLAB . . . 24

28 Simulink monitor chart . . . 25

29 Controller vs NN vs NN Monitor Volvo test track . . . 26

30 Simulated track using current controller, NN and NN monitor . . . 27

31 Simulated track, zoomed in version, using current controller, NN and NN monitor 28 32 Oval track using current controller, NN and NN Monitor . . . 29

33 Beta over time oval track where the color represents the segment . . . 30

34 NN vs NN Monitor on data which was used for training . . . 31

35 NN vs NN Monitor verification . . . 32

Table of Contents

1. Introduction 1

2. Background 3

2.1 Volvo Construction Equipment . . . 3

2.2 Controlling the Hauler . . . 5

2.3 Artificial Neural Network . . . 8

2.4 Related Work . . . 10 3. Problem Formulation 16 3.1 Hypothesis . . . 16 3.2 Research Goals . . . 16 3.3 Research Questions . . . 16 3.4 Limitations . . . 16 3.5 Research Contribution . . . 17 4. Research Approach 18 4.1 Method . . . 18 4.2 Methodology . . . 18 5. Results 19 5.1 Dynamics and Odometry . . . 19

5.2 Artificial Neural Network . . . 22

5.3 Monitor . . . 25

5.4 Simulated Testing . . . 26

5.5 Real Machine Testing . . . 31

6. Discussion 33 6.1 Results Discussion . . . 33

6.2 Design Improvements Discussion . . . 33

7. Conclusion 36

8. Future Work 37

1.

Introduction

During recent years, the strive for automation has increased across all industries where automation can be beneficial. Realizing the dream of putting autonomous vehicles into society has been challenging engineers for years, due to the strict requirements and standards to certify and verify a safe and secure motion of autonomous vehicles. Examples of these standards are e.g. IEC 61508 and ISO 26262. Dependability is a key aspect when designing autonomous systems and proving that a solution is dependable is an increasing cost in resources for companies worldwide. Therefore, validating and verifying if different methods are sufficient for a specific degree of dependability is an important question in state-of-the-art research.

Autonomous solutions for heavy machinery are well suited to further increase safety, productiv-ity and efficiency in concerned industries. An autonomous platform could thus completely remove accidents involving humans, which is beneficial for both the workers and the companies [7] [8]. The workers that operate the machines today, would instead monitor or assist the machines remotely [9].

The technology to help the development of autonomous vehicles advance further and further for each year, and increased efforts from institutions and companies further accelerate the progress. The main problem with autonomous vehicles is that they require robust, yet effective complex hardware and software, whilst still proving that the system is dependable [10]. A considerable amount of time will therefore be put into modelling and simulating every aspect of a system during development, where flaws of the system can be detected.

For autonomous vehicles, the most fundamental part is the motion planning and control algo-rithms. An effective motion plan and control of a vehicle affects the performance of the vehicle significantly and will allow a reduction in cost, in time spent moving with optimal and optimized paths, and in wear on components such as tires. The benefits of optimizing the motion of an autonomous machine are very clear [11].

Figure 1: Volvo Hauler [1]

This master thesis aims at investigating and developing a solution for accurately controlling an autonomous articulated hauler as shown in Figure 1. The authors of the master thesis are students of the dependable aerospace program at M¨alardalen University in V¨aster˚as, and the thesis is done in collaboration with Volvo CE in Eskilstuna, both of whom are located in Sweden. The expected outcome of the thesis work is a solution that increases the accuracy of the articulated hauler’s motion on a pre-defined path with an Artificial Neural Network. Why an ANN will be used is since it is a well proven method for these type of problems, where a lot of research has been conducted on the topic of ANNs.

Currently, the autonomous hauler deviates from the planned path in sharp curves and U-turns due to an increased slip-effect called β. The slip-effect is mainly caused by the third wheel-axle which is added to the rear body. The third wheel-axle is not accounted for in the model which is in use, since the model was previously developed for a two-body articulated machine with two wheel-axles, hence the deviation from planned paths. A basic illustration of the wheel-axle problem is shown in Figure 2. To solve this problem, advanced calculations of the vehicle’s dynamics could be performed instead of using the current Proportional-Integral-Derivative controllers. However, the common practice for these types of calculations is based on knowing the rotational speed of each wheel and wheel-axle, which could be achieved by using odometers as of today.

Figure 2: Principle sketch of the two interception points between the bodies

Odometers for haulers and wheel loaders have been developed but are extremely costly since the wheel-axle of such vehicles is significantly larger than the wheel-axle of a car. If an odometer-solution was to be used, the amount of computation power needed to calculate the complex FEM-equation in real-time would be even more costly.

The aim of this master thesis is to develop a fully functional ANN that can mimic a human’s driving behavior and replace today’s autonomous solution which uses PID controllers. This should reduce the tracking error since a human driver manually compensates for the built-in dynamic flaw that comes with the articulated body and the third wheel-axle. The output for the ANN should be the new steering angle δ for the machine and as inputs for the ANN, the calculated slip errors βlongand βlat, velocity v, heading-angle error ϕ, and the curvature of the machine κmachine. These

variables depend on some sensors, to receive the heading of the machine one would need an Inertial Measurement Unit sensor. The Global Navigation Satellite System and Real Time Kinematic are used together in order to ensure a more accurate position than traditional GNSS, while other errors or variables will be calculated in real-time using classical optometry and dynamics calculations. A monitor should also be implemented to increase the dependability for the overall system in a way that the monitor verifies that the output of the neural network is operating within a safe-range. The monitor should also ensure that the output value from the ANN is not completely different from previous values or a random spike.

2.

Background

The background chapter of the thesis covers the basics of what an articulated vehicle is and the kinematics of it. Relevant information about neural networks, how they work, and relevant state-of-the-art research done in these areas will also be discussed.

2.1

Volvo Construction Equipment

The heavy machine industry is at a new stage of the invention where remotely operated and autonomous machines are becoming increasingly important for streamlining the industry and to get to places where it previously was impossible to have humans work for longer periods. Volvo Construction Equipment has invested in a considerable amount of research to develop these new state-of-the-art machines, such as autonomous haulers and remotely operated wheel loaders. Volvo Construction Equipment AB is part of the Volvo Group, where the main sales and development focus is on construction equipment such as haulers (Figure 3), wheel loaders (Figure 4), and excavators (Figure 5), as represented in the pictures below [1].

Figure 3: Hauler [1] Figure 4: Wheel Loader [1] Figure 5: Excavator [1]

2.1.1 General Automation Platform

The general automation platform for these machines is designed for manufacture, meaning that much of the equipment on the wheel loaders and haulers is the same to reduce cost and increase the dependability for the specified system. The machines use cameras, LiDARs and RADARs for the People and Obstacle Detection to prevent any unforeseen accident happening at the worksite. GNSS antennas are directly connected to a centralized command-station using an RTK solution which increase the precision of the positioning system. The ambition for the automation platform is to have a single operator in the command-station that supervises the whole fleet of machines and can at any time take control over each machine remotely. IMUs are also mounted on the machines for position estimations together with the GNSS system. If the connection is lost to the command-station, dead reckoning is used for a limited time until the GNSS connection is regained, or if the connection is lost for too long, the machine stops. Today an operator is needed to set up a list of tasks for each machine to execute. A typical task for a wheel loader or a hauler is Transport, Wait for Load, and Unload. For path set-up, an operator is needed to place a set of x and y coordinates to create a path for each type of mission or machine. Each machine is still equipped with a local ECU to examine and execute its base function, such as work hydraulics steering, braking, and actuator control of the engine. In Figure 6 shows the can be seen of Volvo CE’s electric site concept, where the haulers are completely autonomous and battery-driven, the excavators are grid powered using a Ground Power Unit and the wheel loaders are hybrids. This concept was tested back in 2018 and reduced the CO2 commission by 97% compared to a traditional quarry [1].

Figure 6: Volvo CE’s autonomous electric site concept [1]

2.1.2 Articulated Autonomous Hauler

The autonomous hauler, which is the subject of this master thesis, was initially a commercial Volvo A30G hauler. The A30G hauler has had its hardware modified in order to support autonomous driving and the project to make the A30G autonomous started in 2016. The hauler has a net weight of 23,000 kilos and can take a payload of up to 29,000 kilos, which adds up to a total gross weight of 52,300 kilos. It has six wheels, two at the front and four at the rear, and all the wheels are powered, giving the hauler a six-wheeled-drive. The front is the part of the vehicle which has the cabin and the motor, while the rear is the body that carries all the payload. Between the rear and the front, the machine has a joint point which makes it articulated, meaning that it can fold in the middle to take sharper curves and handle rougher terrain. The articulated hauler has 8 Degrees Of Freedom, xyz (front), roll, pitch, yaw (front), steering angle and roll (rear). These DOFs are sufficent to create a two pose multibody system xyz, roll, pitch, yaw (front), xyz, roll, pitch, yaw(rear), which is easier to use for control. This makes the machine very agile but increases the complexity of the dynamics greatly. This will be described later in the report.

Figure 7: Volvo Hauler [1]

To position the hauler accurately, it is equipped with a GNSS receiver on the cabins top which is connected to a ground station that sends out a correcting location signal through triangulation between satellites, the ground station and the hauler itself. Since the ground station has a static reference point the position of the hauler becomes very accurate compared to consumer GNSS units. One has to take into consideration that everything does not execute at the exact moment as an input is received, e.g. if the hauler receives a new GPS position, the alteration of the steering angle wouldn’t happen instantly. The machine has a built-in inertia, hence the delay. Hydraulics, mechanical movements, computation, and signal processing are all elements that contribute to the inertia of the machine. This will further be mentioned as the inner-loop of the hauler.

2.2

Controlling the Hauler

To capture the true movement of a ground vehicle or working machine, one has to consider the forces acting upon it along with the machine’s inertia.

2.2.1 Steering algorithm

Feed-forward motion controllers generally employ some form of a model for their feed-forward purpose. A kinematic model for articulated machines can be derived by employing non-holonomic constraints on the axles by stating that they only move in the direction that the wheels are facing (the no-slip condition), see Figure 8 for reference. The resulting kinematic model has the following appearance for the front body;

˙ x0= v0cos(ψ0) (1) ˙ y0= v0sin(ψ0) (2) ˙ ψ0= v0sin (δ) + L1δ L1+ L0cos (δ) (3)

Figure 8: Principle sketch of an articulated machine with two bodies and two axles Assuming that the machine is already very close to its assigned path, the machine will track the path, if the machine’s heading follows the curvature of the path. Thus, the curvature (κ) of the machine’s movement must be equal to the curvature of the path (κpath). For the front body

this equates: κ0= ∂t ∂s ∂ψ0 ∂t = sin (δ) + L1∂δ_∂s L1+ L0cos (δ ) = κpath (4)

Solving the above for the steady state case when ∂δ_∂s = 0 gives the following expression, where atan2 is the four-quadrant arc-tangent:

δf f= atan2  L1κ − L0κ  L0L1κ2−pL02κ2− L21κ2+ 1  L2 0κ2+ 1 , −L0L1κ 2_−pL2 0κ2− L21κ2+ 1 L2 0κ2+ 1   (5) The above expression can be programmed into a steering controller and used in parallel with a feedback control loop to ensure good path following control [12]. If the vehicle should go backward instead of moving forward, replace the front in the the relationship by the rear for all the equations. Volvo CE has two different types of machines which are articulated: wheel loaders and artic-ulated haulers. The model derived in this section with regards to the non-holonomic constraints

is valid for two-wheel-axle wheel loaders. However, it is less valid for articulated haulers since they have three-wheel-axles, which invalidates the no-slip condition used to derive the model. The validity of the articulated machine model decreases as the curvature of the hauler’s path increases, making it impractical for feed-forward control through sharp curves or advanced movements such as U-turns.

2.2.2 Path planning

The path planning for the articulated haulers can be described in many different ways, one way to look at it is illustrated in Figure 9 below, where the front wheel steering is illustrated relative to the curvature of the path (κpath).

Figure 9: Principle sketch of a hauler following a curvature

As shown in Figure 9, r is the radius and κ is the curvature for the circle. The predefined path (κpath) is neither a straight line nor a perfect circle, therefore the path could be described as a

continuous line of piece-wise clothoids. The curvature is defined by how much the curve deviates from being a straight line, where a straight line is zero. The autonomous hauler steers according to the radius of a circle given in that specific time instant, in accordance with the pre-programmed path.

2.2.3 Slip

The hauler has an articulated body that separates the front wheels from the back-body wheels. Normally the back wheels and the front wheels of a vehicle or an articulated machine intercept in one point so that the pivot point always can be determined, no matter what the angle of the wheels is. As illustrated in Figure 10, a wheel loader always intercept at a given point, which makes it easier to calculate the true dynamics of the machine.

Figure 10: Principle sketch of the one interception point between the bodies

However, for a three-wheel-axle hauler as in this thesis work, another interception point will be created as illustrated in Figure 2. This creates an area where the hauler steers around. It is not the same area for every turn since the length between the interception will vary depending on the exact angle of the front wheel and the angle for the front body relative to the back-wheels and the friction coefficient of the ground. This creates an unpredictable steering effect since the radius of the true turn is unknown for the hauler.

An existing multi-body-simulation model of the articulated hauler, previously developed by Volvo CE, is available. The model simulates the kinematics and the motion of the articulated hauler through MATLAB/Simulink. The model can be used to simulate how the hauler behaves in different scenarios and tracks. The model does not consider the no side-slip condition.

2.2.4 Model Predictive Control

A common way of simulating the behavior of a system or vehicle is through models. Model

Predictive Control is an approach where the goal is to redefine a control problem of e.g. a vehicle into a constrained optimization problem, where the cost function is ought to be minimized. The cost function takes inputs, future inputs, outputs and states up until a horizon which is commonly noted as N. Thereafter, sequential states as a function of previous states are estimated [13]. Using the method of MPC together with a robust model of the system, allows simulation of the behavior on the considered system to be produced. An advantage of MPC compared to more classic methods such as PID controllers is that MPC can anticipate future events, which PID controllers cannot. An example of an implementation of an MPC in an articulated trailer can be found in [14].

A big problem with MPC is that in some cases the predictions will be faulty, due to assumptions of ideal conditions or when there is a large number of dependent variables. In the case of vehicles for example, considerations to the surface has to be made. Finding the logic for predicting an accurate control function depending on outer factors is a complex task, which is extremely complicated in the case of side-slip. Therefore, another prediction method, such as neural networks, is a good option to further optimize a model. No MPC is used in the case of a three-wheel-axle, articulated hauler. The decision was made due to the computational complexity involved with an MPC for the hauler, but also due to that the trajectory tracking already is sufficiently accurate, except for the slipping behavior. It should also be noted that the trajectory tracking is stable and therefore it can be concluded that the slipping is what causes the deviations from the correct path. A very simplified equation of the feedback steering regulator of the hauler can be seen in equation 6, where the new command takes the lateral error, heading-angle error and the curvature into consideration.

2.3

Artificial Neural Network

The characteristics of an ANN which separates it from a general artificial intelligence or computing system is that an ANN learns from examples of an already performed path, image or recording in some sort. It also has to do with how well the pattern recognition is and if the ANN can recognize the relationship between unrelated items (associative memory).

2.3.1 Artificial Neural Network Principle

ANNs is a machine learning principle inspired by biological neural networks discovered by studying animals’ behavior and the human brain, trying to translate the behaviors to a computing system. An example of this could be ants searching for food; starting from the anthill, ants are sent out in all directions to search for food, this is our ”artificial neurons”. When an ant finds food it will send out a signal to nearby individuals to get help to bring the food back to the anthill. Each new ant that gets informed about the food will repeat the signal to others and start to head towards the food instead of searching for more. The artificial neurons inside the ANN work in the same way as ants communicate with each other; they are all sent out to find the best output for the system with different input and coherence. The ANN uses different functions and combinations of inputs, trying to generate the best output. Each neuron corresponds to a given number from the input system and they can inherit different units depending on how many variables the system needs to take into consideration when calculating the output. For example, consider calculating the heading ( ˙ψ0) of an articulated machine. A qualified guess of what the heading could be dependent on would

be v0(velocity), δ (steering angle), L1(length of the front body), and L0(length of the rear); these

are now the artificial neurons classes. These are now mapped together in full random order for each hidden layer. There are three different types of layers in a neural network; an input layer (1) which is connected to a hidden layer/layers (2), which in turn is connected to an output layer (3). In our case, the output layer is ( ˙ψ0). See Figure 11 for the layer principle. A deep learning neural

network consists of multiple hidden layers, allowing for advanced and complex problem solving for multiple inputs and outputs. Normally a classic neural network consists of one single hidden layer [15].

There are two different types of operations going on inside an artificial neural network: learning and recall. When talking about learning it is often about activation functions [16] in the form of a rectifier. There is a total of seven activation functions but mainly it is spoken about three of them, since they are best suited for ANNs. The first one is the classic Sigmoid function:

f (x) = 1

1 + e−x (7)

The second one is the most popular Rectified Linear Unit ReLU:

ReLU (x) = max(0, x) (8)

The third is the state of the art Swish function:

f (x) = x

1 + e−x (9)

These three functions represent different behavior for rectifying a mean value between multiple input data. This is included in the system’s learning phase, from loads of input data generated from a simulation or real-data recording of the motion of the system, the rectifying function tries to, with different combinations, guess a generalized output function for the system. Sometimes this learning process takes days to complete depending on what CPU/GPU is in use, how big the learning data-set is, the number of hidden layers and how many input and output values the Neural Network has. When the learning, or training as it is often called, is complete, a trained file has been created and should be moved to the designated system; in this case the hauler steering system. What actually has been produced in this training file is weights and bias, the weight is a generalized constant multiplied with the input x and the bias is an added offset value resulting in a linear graph y = wx + b. The next phase for the ANN is the recall process. Here the trained output-file is in use and takes values from the machine in real-time to make decisions of what to do. How well the system performs in real-time is limited to how well the neural network was trained and how good the data-set was from the beginning, i.e. the path, image, or recording.

2.3.2 Artificial Neural Network for Motion Control

Dynamic machine modeling tends to result in complex systems of differential equations that are not necessarily feasible for real-time implementation in motion control algorithms. Furthermore, the dynamic models often assume knowledge of states that are not always measurable in all machines. For example, Volvo CE’s wheel loaders and haulers do not have individual odometers on each wheel, since the rugged axle design makes it impractical. Thus, it is not possible to compute the individual wheel slip, which is a fundamental input to dynamic vehicle modeling. However, one can still utilize existing measurements such as GPS location, steering angle, IMU data, and drive-line speed in order to approximate a model of the machine’s movement. In the specific case of feed-forward path following control, it is interesting to investigate a model that maps from the desired curvature and any combination of measured or estimated machine states to the optimal steering angle. For example, body side-slip can be computed with the GPS position, heading of the machine, and the de-facto velocity vector. Where the current and the previous values are compared. This is a relatively accurate indicator of how much the machine’s movement is currently violating the no-slip condition, which can potentially be used together with other available motion states in order to map onto the optimal steering angle.

According to Lewis et al. [17], if a smooth function f (x) : Rn→ Rm_{exists, given the compact}

set S ∈ Rn and a positive number εN, there exists a two-layer neural network such as for all, given

a sufficiently large number of hidden layer neurons:

f (x) = WTσ VTx
+ , ||, || < N (10)

If there is a smooth function that maps from desired curvature and an assortment of other signals onto the optimal steering angle, then it can be approximated by a neural network. Since such a function might be difficult to derive by hand, it is a very interesting area of investigation to let a neural network adapt to it. In place of the actual function, one could train the NN versus recordings of actual driver input in conjunction with the aforementioned sensor measurements.

2.4

Related Work

The following section discusses related work and state-of-the-art research in the field of motion planning for articulated machines and the utilization of neural networks for vehicle motion control. The main topics of this section are motion planning and modeling of articulated machines, followed by available methods, and lastly, neural networks in the context of the thesis.

2.4.1 Motion Planning and Modeling of Articulated Machines

There is plenty of research done on motion planning, modeling and control of vehicles, see for example [18], [19], and [20]. For Articulated Machines with three-wheel-axles, the amount of research done is more limited. This is partly due to a limited configuration for vehicles, but also because the problem with ”side-slip” is more obvious when the articulated machine is driving autonomously since a driver learns to compensate for this phenomenon. Therefore, the whole concept of autonomous machines in practice is somewhat new and the existing research is limited. Front-wheel steered vehicles are the most common steering architecture for vehicles, where the back wheel is following a path which the front wheel determines. A model can be seen in Figure 12.

Figure 12: Model of a front-wheel steered vehicle inspired from [2]

Since front wheel steering is very common and fairly simple to understand and model, a lot of research has been done in the field of study and there are many vehicles that use these kinds of models, where they are sufficiently accurate. Front-wheel steering is a good concept to start with to be able to further investigate articulated steering. The main problem with these models is that both another wheel-axle and an articulated body is added to the rear. It will make the model in Figure 12 inaccurate, and to create an accurate version the model would have to be much more complex. In Figure 13, a model of an articulated machine is shown. The figure illustrates the different parameters that are added because of the articulated body.

Figure 13: Model of an articulated vehicle [3]

Articulated machines have different characteristics than ordinary vehicles, where the kinematics differ in certain ways and where the front and the rear have to be taken into consideration [21], [4], [22], [23], [24].

Apart from a very complex steering algorithm that comes with the articulated body, one has to consider the pneumatic tires as well, since forces are generated when the tires slip along the ground while moving. These forces are generally split into a longitudinal mode and a lateral mode, corresponding to the tires longitudinal slip and their side-slip angle. The longitudinal slip on the tires is caused by the tires spinning slightly faster or slower than the equivalent speed that the machine moves in with regard to the ground. The torque providing this difference in speed is facilitated by the brakes and/or drive-line. In contrast, the side-slip is facilitated by the wheels having a slight angle of attack, e.g. when steering. A full machine model would also account for the body side-slip, the turning torque on the wheels, and the steering system as a whole [25], [26], [27]. High-end models also account for suspension effects and multi-body descriptions. The resulting differential equations are then solved in order to output linear and rotational accelerations, velocities and positions for each body in the model. R. D. Martino presents a good overview of dynamic automobile modeling in one of his books [18]. In practice, the differential equations are solved with numerical methods. State of the art software for modeling includes Adams Car and the SimMechanics library for Simulink.

Since there is no fixed rotary point for the hauler to rotate around, as can be seen in Figure 2, the case of side-slip occurs. This happens when a hauler moves and turns, especially in tighter curves. It is not easy to accurately describe the effects of the side-slip condition [28], [29], [30], [31]. To be able to describe it accurately, the following variables have to be taken into account: the environment (dry, wet, ice, etc.), condition of the tires, characteristics of the track and the vehicle, and the speed of the vehicle [32].

A calculation of the kinematic effect for a wheel loader under slip conditions, where slip is noted as s, for the front part of the vehicle:

¯

x1s= ¯v1scos(ψ1+ βf) (11)

¯

y1s= ¯v1ssin(ψ1+ βf) (12)

For this scenario the motion actually takes the slip angle into account and not only the velocity, the articulated angle, and the length for the front and rear. If it is assumed that the articulated machine makes a steady-state motion for the front and the rear, individual velocities can be

calculated as follows:

v1scos(βr) = v2scos(δ − βr) + ˙ψ2L1sin(δ + βf− βr) (13)

v2ssin(δ + βf− βr) = ˙ψ1L0cos(βr) + ˙ψ2L1cos(δ − βr) (14)

If the previous equations can be solved, the heading for the front can be expressed as:

˙ ψ1s =

v1ssin(δ + βf− βr) + L1˙δ cos(βr)

L0cos(δ − βr) + L1cos(βr)

(15) Let’s compare the above equation to the previous equation 3. Equation 3 describes how the current heading for the machine is calculated but does not take the slip into account as the new equation 15 actually does. See Figure 14 for a visualization of the equations previously described.

Figure 14: Principle of side-slip for a wheel loader inspired by equation [4]

As previously stated, a few research papers have discussed modeling motions of articulated machines where the model is accurate to a certain degree, but these models lack the third wheel-axle. The third wheel-axle induces a problem with rotary points, which is one of the big problems for autonomous control of an articulated hauler, as shown in Figure 2.

Odometers are used for measuring the rotational speed of the wheels/wheel-axles for commercial vehicles and other applications [33], [34], where the production cost of those odometers decreases as more vehicles use that exact type of odometer. Odometers have been proven to be very beneficial for autonomous vehicles for motion estimation, where integrating odometers with other sensors can produce a robust and reliable system [35], [36]. In the case of the articulated hauler, the wheel-axle is very large, and there is not a massive amount of them produced. For testing purposes, odometers for the AAH would be very expensive, and therefore it is not feasible.

2.4.2 Neural Networks

Neural networks and machine learning are heavily used in all kinds of industries. A large scale of research has been previously done on this specific topic. There are many different architectures

of neural networks, where each different type can be more or less qualified for a specific type of application or learning behavior [37]. Using neural networks for controlling vehicles is not a new idea either, and is heavily used in many types of vehicles across the globe as of today. Neural networks can help define a specific behavior in a certain environment with a fixed amount of input variables for a machine. Neural networks can handle any type of problems such as keeping plants alive [38], to more problematic and complex problems, such as the landing of an Unmanned Aerial Vehicle or an intelligent path planner for buses [39], [40].

Motion control with Neural Networks

Using neural networks for complex problems can in many cases reduce the complexity, which in turn makes it easier to grasp a complex sub-problem. A neural network can essentially help the developer to understand and overcome complex problems. In the field of autonomous vehicles, neural networks are mainly used for increasing the performance of a specific attribute of the vehicle, for a subsystem, or for testing theoretical solutions to complex problems. In terms of motion control of vehicles together with neural networks, research is mostly done on commercial cars, where the aim of the neural network is to help systems such as lane assist or to increase the precision of the vehicle’s motion [41], [42]. When using neural networks in the field of motion control for vehicles, a model of the vehicle is of uttermost importance. The better the model, the better the output of the NN will be in general. An accurate model will behave similarly as the underlying vehicle, which will ensure that if the NN is used on a real vehicle, the performance will be similar to the simulated model. Secondly, the controller or the control-loop has to be valid to the corresponding model. If both the model and the controller is verified to be accurate, the neural network will most likely be accurate as well, as long as the set of data which was used for training is valid. This approach is portrayed well by Wang et al. [43]. In their paper, neural networks are used for longitudinal sliding control. The development process, to go from a model phase to simulating the neural network and then evaluating the results, is well described. For similar approaches, see [44], [41] and [42].

Implementing a neural network in an articulated vehicle for stability control is discussed by Adamiec-W´ojcik et al. [45]. Their proposed method introduced a mathematical model in com-bination with a neural network. The goal of the proposed model was to stabilize the articulated vehicle during heavy braking. The purpose of the neural network was to get a real-time response from the braking-system. The procedure to develop the neural network is interesting and can be used as an input for developing neural networks for articulated vehicles.

Neural Networks with side-slip

Neural network research with the aim to overcome problems with side-slip is very limited. An evaluation of whether a neural network can estimate or predict side-slip or slip-angles has been done, but the material available is very limited. In the work done by Du et al. [46], an experiment with neural networks was conducted, where the aim was to see if the NN could predict the side-slip angle half a second in advance. It was found that the simulation model was fairly accurate and that the hypothesis was feasible. The main problem with their experiment is that it was done in a simulation, which means that the functionality of the prediction model has not been verified for a ”real” vehicle.

To help analyze data-sets in combination with neural networks Martino et al. [47] constructed an experiment containing two procedures; firstly, training a single layer neural network, and sec-ondly, a technique called ”Principal Components Analysis”. The second procedure’s goals were to both find correlations between different variables that correlate with side-slip and to reduce redundancy for the data collected, by clustering the correlated variables. By using the Principal Components Analysis to conclude the results of the experiment, the computational load could be reduced with next to no reduction in prediction efficiency, which could be essential if the compu-tational power is very limited in a likewise application.

A real-time solution for estimating slip angles is discussed by Gr¨aber et al. [32]. By using a recurrent neural network architecture [37], they found that the slip angle could be estimated in real-time. The most interesting part about the work done by Gr¨aber et al. is that a real vehicle was used to support their result. The model was also used on different road conditions where it

was illustrated and highlighted that slip-angles definitely are influenced by the condition of the terrain in which the vehicle is active. Furthermore, two variants are introduced in the work, the first one being E1, which basically is the neural network with ReLU as the activation function. The second variant builds upon the first, where the difference is that E2 has a physical model added to it. These two variants are later compared, where it was found that E2 was slightly more accurate. In other words, it was concluded that integration between a neural network and a physical model increased the accuracy of the slip angle estimation. Moreover, it was concluded that the estimation’s accuracy decreases with the friction coefficient between the tires and the ground. Lastly, the work achieved its purpose; the model was able to estimate the slip angle with high precision, especially on a dry surface.

Monitored Neural Networks

Neural networks do not go hand in hand with dependability. A deep neural network’s input/output combinations are not fully verifiable as of today. If several values are put into the NN, the output may vary in some ways since Deep Neural Networks are very complex and verifying everything that goes on inside a neural network is impossible. For safety-critical applications it is the “un-verifiability” attribute of DNNs that hinders the usage of such networks. Being able to measure different dependability attributes is something that could increase the dependability of DNNs, as proposed by Cheng et al. [5]. Figure 15 illustrates the connections between four dependability sub-attributes and some key aspects of different neural network metrics.

Figure 15: Dependability in coherence with some neural network metrics [5]

Another approach to ensure a more verifiable neural network is by a ”subsumption architecture”[48]. With this method, it is feasible to create complex behaviors by subsuming the already trained el-ementary ”perception action-loops”. This will make the learning process simpler. The method proposed is most likely not applicable to every usage of DNN, but it can certainly be useful in some cases. An example of such a case would be an application where the neural network predic-tion process could be divided into three parts, such as a predicpredic-tion of an obstacle’s path at different time instants. In each of those parts, the proposed method by Plebe et al. could be used, which if successful, would reduce the complexity of the training process for the DNN.

Using sanity checks or check-sums for neural networks is proposed by Ozen and Orailoglu [49], which is a common method to ensure that a higher degree of dependability is achieved. Other solutions such as duplication of components or Triple Modular Redundancy could also be a solution to achieve an increased level of dependability. These two methods can be very expensive to employ for system safety-critical applications and the computational overhead could be increased as well, according to Ozen and Orailoglu. The authors declare that their proposed solution is set to decrease the overhead and that it offers acceptable coverage at a reasonable cost. The sanity check model which is proposed utilizes the mathematical invariants of the DNN’s computation. The implemented check-sums or sanity checks are said to be able to detect a single point of error

for the input, bias, and output values. The results were verified with a fault injection method. The method was tested in a simulated environment and concluded as a successful experiment. Fault tolerance is an important property of systems, and sanity checks are interesting and in some cases quite a simple method to increase the dependability of the considered system. In addition to Ozen and Orailoglu’s method, Cheng et al. [50] discussed the challenges with certifying neural networks in terms of dependability. A vehicle switching lane on a highway is given as an example. A method to prove that the vehicle is unable to switch lanes whilst there is another vehicle blockading the movement in the other lane is presented. Further, different aspects of neural network issues in terms of dependability are brought up, for example, the black-box structure of DNN, traceability issues and the validity of data-sets. They conclude that even when dependability is set at the highest priority when developing a neural network, it is still impossible to guarantee a completely safe behavior of the system. Thus, a monitor or some kind of sanity check is very useful for arguing for the dependability degree of a system which contains neural network decision making.

3.

Problem Formulation

An articulated hauler has very complex steering dynamics and it is therefore nearly impossible to implement a model of such a vehicle without any deviation. Today the autonomous hauler uses a PID controller solution for controlling the steering, consisting of two parts; the first part, a PID-feedback controller of lateral deviations and the second part which is the kinematic feed-forward controller of the path’s curvature. It is the PID-feed-feed-forward regulator that would need an improvement. The PID controller solution works well for a wheel-loader with two wheel-axles, but in this case, the PID-feed-forward regulator cannot compensate for the side-slip effect caused by the third wheel-axle that comes with the hauler’s body. This deviation can especially be seen when the hauler takes sharp curves or U-turns while driving autonomously.

3.1

Hypothesis

In order to follow a curvature better for the Autonomous Articulated Hauler, an Artificial Neural Network is investigated if it could decrease the deviation from the predefined path. Why an ANN is used and no other specific machine learning method is partly because of that Volvo requested this solution. It is motivated with that there is a lot of research done in the field of ANNs and it is a well-proven method for these types of problems, according to section 2.4. The network should be trained with data from a professional driver since a human manually compensates for the slip that comes with the third wheel-axle. If the neural network’s output mimics the behavior of a human operator, we believe that the network has learned the kinematics of the machine, which would result in that the machine is able to follow a reference curvature. Inputs for the network, the calculated slip errors βlong and βlat, velocity v, heading error ϕ, and the curvature of the machine

κmachine should be enough to produce a new output δ as steering angle for the machine. The

neural network’s output will then be verified through a monitor function, which will check if the intended function will operate reliably and within a safe range.

3.2

Research Goals

The goal of this master thesis is to develop a fully functional and dependable ANN that can steer the hauler so that a reference curvature can be followed. The dependability will be proved by a monitor that ensures that the motion of the hauler is in fact dependable. We also aim to find a way to reliably calculate the slip-error in real-time since that has not been done before on an Autonomous Articulated Hauler.

3.3

Research Questions

• RQ1: Is it possible to decrease the deviation from the articulated body’s path with a neural network and if so, how much of a better motion is achievable?

• RQ2: Can a fault-tolerant architecture be created in a way that the outputs from the selected neural network are monitored within a safe range?

• RQ3: Is it possible to verify the proposed artificial neural network and if so, in what way?

3.4

Limitations

• The hauler may be occupied for other purposes, making testing impossible at certain times. • The deadline can be too strict to fully develop and verify that the neural network is working

exactly as intended.

• If data are gathered when the soil is frozen and/or snowy, the training of the neural network might not be completely accurate compared to when testing during other conditions. • The thesis will only be focusing on the steering part of the motion control, and not the

motion control as a whole (e.g. the throttle).

3.5

Research Contribution

Up to now, there is limited research done on three-wheel-axle articulated machines and together with neural networks, the research is even more limited. In terms of slip-error together with the previously mentioned field of studies, the research available is basically none. Therefore, a fully functional neural network for this specific vehicle is a state-of-the-art research and a contribution to further development of three-wheel-axle autonomous articulated vehicles. Furthermore, a NN together with the structure of the proposed monitor would ensure that a NN thus can be verified and run in a more reliable way.

4.

Research Approach

This chapter will present the method and the methodology which will be used to help answer the research questions.

4.1

Method

Figure 17: Flow Diagram of Multi-methodological Research Approach [6]

In order to achieve a positive result for the problem formulations, a scientific approach known as the “Multi-methodological Research Approach” will be used. This method is applicable for research in engineering where the research questions are complex. The research process is iterative, which means that if one component does not fulfill a requirement, the process can start over or the inadequate element can be adjusted to fulfill the requirement, as illustrated in Figure 17. For training the neural network, a quantitative data collection will be gathered by driving the hauler manually at the site and then downloading data such as velocity, steering angle, and offsets in the motion of the vehicle.

4.2

Methodology

The Multi-methodological Research Approach is common for a typical engineering workflow, start-ing with a literature study to gain a knowledge of the state of the art and thereafter trystart-ing to implement the hypothesis which is drawn from the study. With the help of this research method, results are validated and verified throughout the process. The quantitative data collection is justi-fied in the means that the neural network needs as much data as possible. With a larger amount of valid data used for training the network, higher accuracy of the implemented system is achieved. In other words, the more accurate the system is, the more dependable it will become.

Today a professional driver follows a path more accurately than when the hauler is driven in autonomous mode. If data are collected from a driver who has a very small deviation from the given path, the collected data should help the neural network to make itself more accurate than it would have been to train a network with big deviations from the start. It is also more beneficial to use real data of how the machine behaves compared to e.g. a simulation.

5.

Results

In the results chapter, the most important findings are presented. A deep analysis of the simulated and real machine results, the neural network’s architecture and the monitor is presented as well.

5.1

Dynamics and Odometry

Simulations of the dynamics of the machine were done through Simulink and MATLAB. The simulations were aimed at providing data to be able to calculate errors, they were also used to be able to visualize correlations between different variables. For example, the curvature in correlation with the simulated track. Because of the ease of plotting variables in MATLAB and Simulink, an understanding of the problem with the side-slip was more easily attained. However, no simulated data was later used when training the actual neural network. Calculating βlat and βlong is of

great interest for finding the error compared to a reference path and also as inputs for the neural network. These calculations were done and then tested in simulation before moving on to the actual implementation of the neural network.

Figure 18: Illustration of how Beta long and Beta lat is calculated

Consider x0as the starting position, ˆx as a calculated prediction of the next position, and x1as

the actual position of the hauler at the next time instant. These three-position creates a triangle as illustrated in Figure 18, now with the help of The law of cosines and the Cartesian coordinate system, the lengths and angles of the triangle can be determined. The position of x0 and x1 can

be received from the GPS in a form of coordinates as (x,y), the length C that is between the two coordinates can be calculated as:

c = q (x1− x0) 2 + (y1− y0) 2 (16)

and the distance between x0and ˆx (with The Distance Formula) is:

b = v · t (17)

and the distance between x1and ˆx is given by:

a = q (ˆx − x1) 2 + (ˆy − y1) 2 (18)

By the The law of cosines, the angle α can be determined as follows: α = cos−1 a 2_{+ b}2_{− c}2 2ab  (19) With known lengths and angles a longitudinal and latitudinal error can be calculated in real-time for the articulated hauler. The latitudinal slip will be the height of the triangle βlat and the

adjacent catheter of the right-angled triangle will be βlong.

βlong = a · cos (α) (20)

βlat= a · sin (α) (21)

These real-time calculations works for all given variations of the positioning for x0, x1and ˆx with

the help of some if statements in the code. For example, if the machine has traveled further than anticipated, i.e x1is in front of ˆx, which will result in that α is greater thanπ2. The arccos function

is only defined in the interval [0,π] so π has to be subtracted from α to ensure that the law of cosines is still valid for calculating βlatand βlong.

After the calculations of βlong and βlatwere tested in simulation and verified that they worked

in all conditions, the next step was to calculate the heading-angle and the heading-angle error so they could later be used as input for the neural network. The heading-angle ψ is the angle which the machine should have traveled with at the time instant of x0. The hauler angles are illustrated

in Figure 19. The heading angle is essential for calculating the machine’s movement for the next time instants. With the help of ψ and the position and the velocity of the machine, the next point ˆ

x can be calculated. The angle ϕ(phi) is called the heading-angle error, the angle ϕ is the angle between the points of which the machine actually traveled, i.e. the points x0 and x1 as shown in

Figure 20 where a more clear illustration can be seen. Phi can be calculated from ∆x and ∆y and by using the four-quadrant inverse tangent function, commonly referred to as arctan2(x, y). The tangent function arctan2(x, y) ensures that all different possible scenarios can be handled, since it is defined for values in the interval of [−π, π], i.e. all angles. Since one lap in a circle is 2π radians, the more time the machine has been up and running the number of laps will grow, therefore the heading must be wrapped. Each time the machine has completed a full circle, 2π is automatically subtracted so all the heading angles stay the same. Otherwise, ψ would grow to infinity the longer the machine has traveled. This ensures that as stated both ψ and ϕ stays within a correct range and that the calculations of βlatand βlong has the correct sign for each time instant.

One of the inputs for the neural network is the curvature of the machine, which has to be calculated. The curvature for the machine κmachine, is the curvature of the vehicle at a given

time point. κmachine should not be mistaken with κpath since that is the curvature of the path.

If a target curvature is used, the machine would not travel autonomously, it would just follow a path-following algorithm often referred to as ”automatic”. An illustration of this phenomena with

κmachine and κpath is shown in Figure 21.

To further emphasise, autonomous motion in the context of vehicles is when a vehicle’s motion is determined through algorithms which has no pre-programmed set of values, i.e. it can drive any given path without knowing the characteristics of it in beforehand. An ”automatic” motion in terms of vehicles referees to a vehicle that for example follows a pre-programmed path, i.e. the vehicle can drive the given path, but not anything else outside of that given path. In order to enable an autonomous motion for the hauler, the neural network has to learn the actual kinematics of the real hauler. If the neural network knows the kinematics of the hauler, a better result is achieved which would mean that it works for basically all different paths and not just for following a given path. To clarify, if the difference between κmachine and κpath is zero or close to zero, the neural

network is able to compensate for the slipping behavior during motion on its own. The formula for calculating κmachineis as follows;

κmachine=

∆ψ

∆s (22)

5.2

Artificial Neural Network

In this chapter the steps which was taken to create the ANN will be explained. The first part is about data processing, the second part; is about the development of the ANN and which parameters that will affect the architecture and its performance.

5.2.1 Data Processing

All data that was used was logged and downloaded from the hauler’s on-board computer. The gathered data includes the GPS coordinates(x,y), heading of the machine, velocity and the steering angle and was gathered during three separate occasions during the thesis work. The final training data-set for the neural network included 27.000 samples, which corresponds to approximately 45 minutes of driving since the sample rate was at 10 Hz. The validity of the data was checked to ensure that it did not include any random spikes or obvious errors, such as a GPS loss. In the figures below, examples of different tracks which are included in the data-set can be seen.

x[m]

y[m]

Figure 22: Test Track

x[m]

y[m]

Figure 23: Oval Track

x[m]

y[m]

Figure 24: Max turn Track It is common knowledge that neural networks require a valid and large amount of data to function properly. This means that the NN has to be trained on many different types of motions, e.g. sharp turns, maximum steering angle and straight motion. The total amount of data which was gathered included close to 27.000 samples, where 7000 of these were deemed as unfit. The most of the unfit samples were because of either the machine moving backwards, standing still/coming to a stop or GPS loss. This resulted in a data-set of approximately 20.000 samples as the data-set which was used for training, where the sampling rate was 10Hz. A common fault in the data was when the velocity was at zero, or around zero, the value could jump to negative because of the GPS ripple. At approximately 0.05 meters/second, the velocity was instead set to zero. This was done to increase the performance of the neural network and also because of that the sensor values were fluctuating which meant that, sometimes when the hauler was completely still, the measured velocity could be negative. The same action was taken for the variable which calculates the difference between x0 and x1, which affects the κmachine variable. All data of when the

machine travelled backwards were removed, meaning that no data were included for v < 0. This was expected to increase the performance of the neural network as well since the operating window was more limited, and hence, an increased performance could be achieved.

The input values for the neural network has to be normalized before training because of that all inputs come at different intervals and it will speed up the training process as well. The function for normalizing in the range [0,1] can be found below.

zi=

xi− min(x)

max(x) − min(x) (23)

5.2.2 Neural Network Architecture

The neural network was built in MATLAB, with a tool called ”NNtool” and the ”Deep Neural Network toolbox” [51]. The neural network takes five normalized inputs, βlong, βlat, the velocity

go through the weights and biases which are obtained during training, which enables the NN itself to determine and find how the inputs correlate with each other to output a new steering angle. During training, the data-set of inputs correlates to a steering angle, which is the targeted output during the training process. When the neural network receives the inputs during execution, it will map those to a steering angle which the NN considers as the correct steering angle for the given inputs. See the background chapter 2. for a broader summary.

Figure 25: Simplified NN Architecture 2x4 instead of 10x10

When designing and constructing the architecture for the neural network there is no right or wrong, there is no standardized hidden layer value or amount of neurons. Each architecture has to be optimized and tested for each ANN separate application. In this case, Matlab’s Global Optimization Toolbox was used to find the parameters of the NN. It is a straight forward process beginning with one hidden layer and one neuron. It compares how close the NN output is to the target, then just increase the amount of hidden layer or neurons until the performance stops increasing. The results were verified with a second approach, were manually constructed network’s regression coefficients and performances of each network architecture was compared. The test of the performance for the different architectures can be found in Table 1.

ANN Architecture 5x5 7x7 10x10 15x15

Regression Number 0.961 0.963 0.972 0.974

Performance 0.00403 0.00241 0.00125 0.00109

Table 1: Performance and Regression test for ANN architectures.

Where the regression number(R) is a measurement of the correlation between an output from the ANN and the targeted value. If R=1, an indication of an exact linear relationship has been achieved, were R=0 indicates no linear relation at all. This means that R =1 is a well-performing network, which is desired. Furthermore, the performance is a value which indicates the performance of the network in comparison with the target value. For the performance value, a lower value is desired. All the regression plots can be seen in Figure 26.

The number of hidden layers and neurons became 10 hidden layers and 10 neurons/hidden layers (10x10) for this specific application. An architecture consisting of a 7x7 architecture’s results was pretty close to the performance of a 10x10 architecture. However a 15x15 architecture was not necessarily worse in the correctness of the output, it was instead very slow and inefficient both when training but also during execution. Therefore to be sure that neither performance nor efficiency was compromised in an obvious fashion, an architecture of 10x10 was therefore chosen for this application. The final architecture for the NN can been seen in Figure 27.

Figure 27: Neural Network architecture in MATLAB

The NN type is a Feed-Forward-back-propagation neural network. It is a NN type that only takes in the raw input values in the first hidden layer, after that there are just weights and biases inside the hidden layer architecture which are multiplied and added together. Compared to the cascade feed-forward neural network it inserts the raw input values in each hidden layer, these two different types of NNs were the two final candidates when coming to choose what to go with. The Feed-Forward-back-propagation architecture was both faster to train and gave a better output compared to the cascade feed-forward NN. More epochs were also tested but the back-propagation NN was still better even when the cascade NN had trained for a long time with more epochs.

The ANN uses ReLU as activation function, which is a linear activation function, see equation 8. ReLU was chosen since it has a better gradient propagation, compared to e.g. Sigmoid function the ReLU activation functions handles the vanishing gradient problem better.

The used training function is called Bayesian Regularization, it was used since it is a better training function for noisy input data compared to for example Lavenberg training function that is optimized for heavy Random Access Memory usage. The Bayesian training function is good for this application since an AAH weights about 50 tons depending on if it is loaded or not and there will be lots of vibrations as the machine continues to move it will automatically cause noisy input parameters. After the neural network was trained it was thoroughly tested and used in the simulation, the MATLAB code was then converted into C++ code, because the machine’s code is written in C++.