Datateknik C, Examensarbete, 15 h¨ogskolepo¨ang
Classification framework for
monitoring calibration of
autonomous waist-actuated mine
vehicles
Per Landstr¨
om och John Sandstr¨
om
H¨
ogskoleingenj¨
or Datateknik, 180 hp
Examinator: Johannes Stork
Institutionen f¨or naturvetenskap och teknik ¨
Orebro University ¨
Orebro, Sweden VT 2020
Abstract
For autonomous mine vehicles that perform the ”load-haul-dump” (LHD) cycle to operate properly, calibration of the sensors they rely on is crucial. The LHD cycle refers to a vehicle that loads material, hauls the material along a route and dumps it in an extraction point. Many of these vehicles are waist-actuated, meaning that the front and rear part of the machines are fixated at an articulation point.
The focus of this thesis is about developing and implementing two differ-ent frameworks to distinguish patterns from routes where calibration of the hinge-angle sensor was needed before and try to predict when calibrating the sensor is needed. We present comparative results of one method using ma-chine learning, specifically supervised learning with support vector mama-chine and one optimization-based method using scan matching by implementing a two-dimensional NDT (Normal Distributions Transform) algorithm. Comparative results based on evaluation metrics used in this thesis show that detecting incorrect behaviour of the hinge-angle sensor is possible. Evaluation show that the machine learning classifier performs better on the data used for this thesis than the optimization-based classifier.
Keywords: SVM, NDT, machine learning, scan matching, hinge angle sensor, calibration detection
Sammanfattning
F¨or att autonoma gruvfordon som utf¨or ”load-haul-dump” (LHD) cykeln ska fungera ordentligt s˚a beh¨over sensorerna som de f¨orlitar sig p˚a vara ko-rrekt kalibrerade. LHD cykeln h¨anvisar till ett fordon som lastar material, fraktar materialet l¨angs en rutt och dumpar det vid en extraktionspunkt. M˚anga av de h¨ar fordonen ¨ar midjestyrda, vilket betyder att fr¨amre och bakre delen av maskinen ¨ar fixerad vid en artikulationspunkt.
Fokuset p˚a det h¨ar arbetet handlar om att utveckla och implementera tv˚a olika ramverk f¨or att urskilja m¨onster fr˚an rutter d¨ar midjevinkelssensorn beh¨ovts kalibreras tidigare och kunna f¨orutse n¨ar n¨asta kalibrering av sen-sorn kommer att beh¨ovas. Vi framf¨or j¨amf¨orbara resultat fr˚an en metod som utf¨orts med hj¨alp av maskininl¨arning, specifikt supervised learning med support vector machine och en optimeringsbaserad metod med hj¨alp av skanmatchning genom att implementera en tv˚adimensionell NDT (Normal Distributions Transform) algorithm.
J¨amf¨orbara resultat baserat p˚a evalueringsm˚att som anv¨ants i det h¨ar ar-betet visar att det ¨ar m¨ojligt att detektera avvikande beteende hos mid-jevinkelssensorn. Evalueringen visar att maskininl¨arnings klassifieraren presterar b¨attre p˚a datat som anv¨ants ¨an den optimeringsbaserade klassifieraren.
Nyckelord: SVM, NDT, maskininl¨arning, scanmatchning, midjevinkelsensor, kali-breringsdetektion
Acknowledgements
We would like to start by thanking Helena M¨untzing and our on-site super-visor Fredrik Grahn for the opporunity to write this thesis with the RTE Fleet management and Interoperability group at Epiroc. We would also like to thank our university supervisor Martin Magnusson for his valuable ideas and guidance during the time of this thesis.
Contents
1 Introduction 6 1.1 Background . . . 6 1.2 Project . . . 7 1.3 Vehicle Configuration . . . 9 1.3.1 Hinge Angle . . . 9 1.3.2 Gyroscope . . . 9 1.3.3 Laser Readings . . . 10 1.4 Data . . . 10 1.4.1 Machine Logs . . . 10 1.4.2 Map Files . . . 11 1.5 Requirements . . . 12 1.6 Task Distribution . . . 13 2 Classification Framework 14 2.1 Machine Learning Classifier . . . 142.1.1 Supervised Learning & Unsupervised Learning . . . 14
2.1.2 SVM, Support Vector Machine . . . 15
2.1.3 Data set & Feature Engineering . . . 18
2.2 Optimization-based Classifier . . . 21
2.2.1 Scan matching . . . 21
2.2.2 ICP & NDT . . . 21
3 System Architecture & Implementation 24
3.1 Machine Learning Implementation . . . 24
3.2 Optimization-based Implementation . . . 26
4 Result 29 4.1 Evaluation . . . 29
4.1.1 Evaluation metrics . . . 29
4.2 Evaluation of ML-based Classifier . . . 30
4.2.1 Tests . . . 32
4.3 Evaluation of Optimization-based Classifier . . . 37
5 Discussion 41 5.1 Machine Learning Classifier . . . 41
5.2 Optimization-based Classifier . . . 43
5.3 Comparison of the models . . . 45
5.4 Fulfilment of Project Requirements . . . 47
5.5 Social and Economic Implications . . . 48
5.6 Future Development of the Project . . . 49
6 Reflection of Personal Learning 50 6.1 Knowledge and Understanding . . . 50
6.2 Skills and Abilities . . . 50
6.3 Perception and Approach . . . 50
1
Introduction
1.1
Background
Epiroc develops and produces products for the mining industry, amongst them are autonomous machines that preforms the ”load-haul-dump” (LHD) cycle. The LHD machines are usually used for loading mining materials and transporting it to a specific location where the fragmented rock is dumped. Most of these machines are waist actuated and for these to operate properly it is important that the sensors measuring the hinge angle are calibrated.
The goal with this thesis is to implement and add a classification framework to the LHD automated machines for detecting when calibration of the hingle-angle sensor is needed. The hinge angle sensor measures the angle between the front and rear part of the machine and is mounted on the joint holding the two together at position G in Figure 1.
Figure 1: Overview of a waist-actuated LHD machine, image taken from Marshall et al. [1]
Previous research has been done by Olofsson [2]. Olofsson’s thesis focuses more on the calibration methods of the hinge angle sensor and the laser rangefinders that are used rather than how to detect the need of calibration. As explained in Olofsson’s thesis the calibration of the hinge angle sensor on the machine is today done manually by comparing the laser rangefinder data of the front and the back of the machine (positioning of the laser rangefinders can be seen in Figure 2) while driving approximately 2.5 m/s along a flat surface [2]. But there are no method of determining if the sensor is in need of calibration.
In the work of Him et al. [3] a method to optimize the manufacturing process of a can welding machine by using predictive maintenance is presented. This is a different problem to ours, since they have studied a welder machine and we are working with mine vehicles. We both are working with sensor data and the way that they acquired their data is similar to the way that our data has been recorded. They created their own data set by recording a selection of features from the sensors on the machine. The features that were extracted in their experiments were the current passing the welder, vibration of the machine motor, pressure of incoming cooling water as well as temperature and flow rate of the chilling water in different sections of the machine. They collected data over the span of one month where the operator of the machine recorded failures manually. The sensor data of the machine that operated hours without failures were labeled ”normal”. For each failure, the sensor data of 20 minutes before the failure happened was labeled as ”welding failure”. Lastly they organized their features into different presets and used a decision tree algorithm to identify the feature that contributes the most to failure and predict the failure.
To solve a predictive maintenance problems concerning mechanical, electrical and plumbing (MEP) components to extend the lifetime of facilities. Cheng et al. [4] used an IoT (internet of things) and BIM (building information modeling) architecture together with machine learning algorithms (Support vector machine & artificial neural network) to predict the future condition of MEP components. Susto et al. [5] also used an SVM for predictive maintenance however for a different implementation purpose.
Methods of calibrating sensors mounted on robotic systems has been proposed in Della Corte et al. [6] where the focus is on systems with multiple sensors. They propose methods of automatically calibrating sensors on robotic platforms on-the-fly by constantly updating the kinematic parameters. The implementation allows failure detection from changes in two parameters, platform mechanics and sensors. Usually changes in these two parameters happens gradually due to mechanical wear or all of a sudden due to changes in sensor poses.
1.2
Project
This projects focus has been to implement two methods for predicting the need of calibration of the hinge-angle sensor on Epirocs LHD machines. One method where we look further into machine learning, with focus on supervised learning. The other method is optimization-based and implements scan matching to see if it
is possible to estimate how much the hinge-angle has to be adjusted for two scans to match with sufficient accuracy. The purpose with having two models is that we want to explore different approaches and compare which method that gives the most accurate result.
For the machine learning model we will try to extract valuable parameters that could be used in the classification framework. The classification algorithm that will be used is SVM (Support Vector Machine).
In the optimization-based model we will use the data from the laser rangefinders and the hinge-angle and implement it in an NDT (Normal Distributions Trans-form) algorithm. The NDT algorithm uses point clouds as input and produces a new representation that will be compared to the actual maps that we have been provided with as well. The idea is that the transformation needed to align the point cloud with the actual maps will tell us how wrong the scans from the machine are compared to how they should be if the laser readings are correct.
The functionality in this thesis would be used on the autonomous LHD machines (self-driving loaders and trucks), as there is no operator in the cabin to notice when calibration might be needed. It is useful to be able to evaluate how often calibration is needed, and actually done, and also for fault tracing purposes if the machine is behaving incorrectly.
1.3
Vehicle Configuration
This section will give an overview of the different components that this thesis will focus on (See Figure 2).
Figure 2: Overview of an LHD machine and its sensors, image taken from Marshall et al. [1]
1.3.1 Hinge Angle
The hinge-angle sensor consists of a drive shaft encoder based on inductive sensors, a toothed wheel on the drive shaft axis and a CAN-Open hinge angle encoder [1]. The hinge angle sensor is mounted in the joint where the front and the rear of the machine are held together. The way it is mounted is such that it registers a value in radians that tells how large the angle is between the rear and the front of the machine. The sensor signal is registered in a 12 bit A/D converter with ∼0.09◦ resolution. The standard deviation of the sensor is ±0.1◦ but that is only when the machine is standing still. When the machine is in motion the deviation depends on the velocity. The LHD machine has the capacity to work in the interval [−45, 45] degrees.
1.3.2 Gyroscope
The gyroscope is contained in the IMU (Inertial Measurement Unit) and is manu-factured by AIMS. The IMU has three gyros with 24 bits resolution giving an an-gular speed resolution of 16µ◦/s. The gyroscope measures angular velocity [rad/s] while the machine is driving. The gyroscope has a bias error of 0.06◦/s and a
misalignment of 2 mrad. The noise from the gyroscope is 0.3◦/s but according to Epiroc the noise also increases rapidly when the machine is making sharp turns. The values given from the gyroscope can be compared with the hinge angle sensor values and in this thesis we will only be using the heading gyro.
1.3.3 Laser Readings
The LHD machines are equipped with two laser rangefinders, one is mounted on the front part and one on the rear part of the machine. The scanners are mounted such that we get almost 360 degrees except for the centre part of the machine where the joints are connected, one of them facing backwards and one facing front. The scans are taken in a two-dimensional plane covering 180 degrees with a maximum range of 32 m.
1.4
Data
This section introduces how the data used in the implementations is represented, initially with the machine logs. The following subsection introduces the map files used in this thesis.
1.4.1 Machine Logs
The machine logs that have been provided contains data collected some time before an error has occurred that caused the machine to stop (See figure 3). The data in the log files are registered every 50 ms after the recording starts, it consists of various data such as hinge angle, laser data, gyro data and position along the route. The logs come from different routes in the mine and they vary in length ranging from a few seconds up to several minutes. The data that will be used in the frameworks is extracted from log files from previous routes registered by the LHD machines.
Figure 3: Visualization of an LHD machine registering data for the machine logs, the purple circle is where the machine started logging data, the brown circles represents each time stamp of data collection.
1.4.2 Map Files
The map files for a route consists of an atlas of metric maps which was provided by Epiroc. The metric maps consist of cells which are occupied or not to represent if there are any obstacles. These metric maps are generated using both odometric and range finder sensor data. There are two reasons that an atlas of metric maps are used in Epiroc’s system instead of one single monolithic map, high usage of memory and inconsistencies in the map from when a vehicle closes a loop or crosses its own path [1].
For this thesis it is sufficient to use the metric map cells in each of the atlases to detect the walls, which could then be converted to coordinates to generate point clouds for the optimization-based classifier. The resolution of the map files describes each cell as 10x10cm which the laser reading could be placed in, this introduces a margin error of
arctan(10/3200) = 0.18◦
where 10 is the cell width in cm and 3200 is the maximum range of the laser rangefinders.
To be able to extract only the cells that represents the walls of the tunnel, an edge detection is implemented by checking for each of these cells that are occupied that they had exactly one neighbour cell assigned with a zero as in Figure 4. A visualization of the lookup is represented with the red arrows pointing to occupied cells and the green arrow to an unoccupied cell.
Figure 4: Visualization of the edge detection where red cells represents the tunnel walls and the green cells represents the tunnel
1.5
Requirements
For this thesis to be considered a success there will be two criteria that should be fulfilled.
First of all, both of the methods (machine learning and scan matching) should be able to detect errors in the calibration of the hinge angle sensor of the LHD machines. The methods should be implemented to the extent that a comparison of the two methods can be done using selected evaluation metrics.
Secondly, to determine if the selected methods are precise enough to prove to be useful for Epiroc. According to Epiroc the classifiers should detect errors between 0.5◦ and 1◦ on machines that are driving, atleast 70% of the time to be useful in production.
Lastly, if the best method of the comparison fulfills the constraints given above and if there is enough time. The framework with the best method will be implemented to the RCS (Rig Control System) for testing in real environments.
1.6
Task Distribution
Since we implemented two different methods for solving the specific problem, we divided the work between us. John focused mostly on the optimization-based classifier and Per focused mostly on machine learning. In order to get knowledge about both methods we had daily briefings about what we did during the day so that the other person would have a general understanding on how the other method works. This was done so that when the implementation was made we could analyze and compare the results together.
2
Classification Framework
This chapter provides theoretical background, such as approaches and algorithms for the two implementations (machine learning and scan matching).
2.1
Machine Learning Classifier
This section introduces different machine learning techniques, heavily focused on supervised and unsupervised learning. In the last subsection the different features used in the machine learning implementation is discussed.
2.1.1 Supervised Learning & Unsupervised Learning
Machine learning methods are usually divided into supervised- and unsupervised learning but there are also other categories such as semi-supervised learning and reinforcement learning. The latter two will not be described in this thesis.
Supervised learning is where the model trains on data with known output variable such as ”calibration needed” and ”calibration not needed”. The model then tries to predict the output variable of new data based on the patterns acquired from the training [7]. There are lots of different algorithms that are categorized as super-vised learning e.g. Support vector machine (SVM), Naive Bayes and decision trees. The difference between them is how they separate the training data points in order to classify the new data. For example Naive bayes uses conditional probabilities to calculate the probability that a data point belongs to a certain class[8], while SVM finds the optimal hyperplane that divides the different data point classes. Unsupervised learning on the other hand is a technique that finds patterns in the data without labels. Cluster analysis is one method used in unsupervised learning and this method tries to group data that have commonalities and assign a label to it. Popular algorithms that is used in unsupervised learning are e.g. K-means clustering and DBSCAN.
According to Silvestrin et al. [9] deep learning methods require tremendous amounts of data in order to get decent accuracy and with smaller datasets, classical machine learning techniques work better than the deep learning techniques on scarce sensor data. They show that a selection of classical machine learning algorithms(Decision Trees, Random Forests & K Nearest Neighbours) with simple feature engineering
techniques with a training set of size 400 examples gave a classification error around 3%. The deep learning methods(Long short-term memory & Temporal convolu-tional networks) they tested gave around 40% classification error with a data set of the same size. Silvestrin et al. used the hydraulic system condition monitoring data set developed by Helwig et al. [10]. Since the data provided from Epiroc is relatively small, a little more than 200 logs in total, the classical machine learning techniques will most likely work better than the deep learning techniques.
2.1.2 SVM, Support Vector Machine
Support vector machines try to find the optimal hyperplane that separates the different classes from each other. The optimal hyperplane is the hyperplane that has the maximal amount of margin between the classes. The margin is defined by the data points that are closest to the optimal hyperplane, these data points are called the support vectors [11].
Figure 5: This example shows a 2 dimensional problem where the classes get separated, the support vectors are surrounded by a gray square. Image taken from Cortes and Vapnik [11].
The input for an SVM is a set of (feature, label) pairs such as (x1, y1), ..., (xn, yn).
The feature xi contains of properties that can be found in the data such as
statisti-cal measures. The label yi denotes the data class which xi belongs to. The output
contains a set of weights wi for each feature whose linear combination predicts the
value of y. The weights of the support vectors are nonzero and by maximizing the margin of the hyperplane which are defined by the support vectors, the number of weights that are nonzero gets reduced. The problem of finding the maximal margin is a quadratic optimization problem which can be solved by using Lagrange multi-pliers, a detailed explanation of this can be found in Cortes and Vapnik [11]. This optimization is done so that only the important features matter when deciding
the separating hyperplane. The SVM model therefore consists of the separating hyperplane i.e the support vectors and their weights.[11]
Given a training set on the form of (~x1, y1), ..., (~xn, yn) where yi = ±1 denotes the
class that the data point ~xi belongs to, the optimal hyperplane can be constructed
according to
~
w · ~xi+ b = 0. (1)
Where w is a weight vector, x is the input vector and b is bias. The hyperplanes that the support vectors form (see Figure 5) can be defined according to
~
w · ~xi+ b = ±1. (2)
The distance from the optimal hyperplane to one of the hyperplanes that the support vectors create can be expressed using the following equation
d =| ~w · ~xi+ b| k ~wk =
1
k ~wk. (3)
Therefore the total distance between the hyperplanes that the support vectors create, i.e. the margin (see Figure 5) is thus:
M = 2
k ~wk. (4)
In order to maximize the margin M, k ~wk has to be minimize under the condition yi( ~xi · ~w + b) ≥ 1 that there are no data points between the hyperplanes that the
support vectors create. The problem of minimizing k ~wk under those conditions are a quadratic optimization problem which can be solved using Lagrange multipliers (for a detailed explanation see Cortes and Vapnik[11]).
In the SVM problem the Lagrangian is the following (where ai is a slack variable):
min Lp = 1 2k ~wk 2− l X i=1 aiyi( ~xi· ~w + b) + l X i=1 ai. (5)
by applying a lagrangian stationary condition the following is obtained
w = l X i=1 aiyix~i (6) l X i=1 aiyi = 0. (7)
by solving the dual problem of Lp a solution for solving ai is found. The use
of kernel functions which is explained later also becomes possible by solving the dual problem. When knowing ai the weights w for the optimal hyperplane can be
found. Once the weights are found an unknown point u measured by features xi
can be classified by the sign of the following function
f (x) = w · u + b = (
l
X
i=1
aiyixi· u) + b (8)
By extending the SVM algorithm with a so called ”soft margin”, a hyper-paramter C is introduced which defines how many misclassifications that are allowed. By having a large value to C the optimal margin becomes larger on the expense that some data points gets misclassified. The soft margin therefore allows some errors, this can be helpful since it may produce a more generalized model when applied to a new data set.
For data points that cannot be separated by a linear hyperplane, the so-called kernel trick is used to calculate the optimal hyperplane in a higher dimension where they can be linearly separable without explicitly mapping the data points into that higher dimension feature space. This is done with the help of a kernel function, e.g. an RBF(Radial basis function), and causes the linear hyperplane in feature space to be a nonlinear decision surface in input space (see Figure 6). More detailed explanations of the kernel functions can be found in Boser et al. [12] or Scholkopf et al. [13].
The RBF kernel uses a hyperparameter γ which can be seen as the inverse of the radius of influence of samples selected by the model as support vectors. With other words γ adjust the curvature of the decision boundary. A too large value of γ will result in the support vectors surrounding themselves and a to low value would cause the model to behave as a model with a linear kernel.
Figure 6: φ is the mapping function that maps the data points from input space into feature space. Image taken from Scholkopf et al. [13]
2.1.3 Data set & Feature Engineering
In this section we will present the data set that was used and the different features that will be fed to the ML-classifier.
As previously described in section 1.4 the hinge-angle sensor measures the angle between the front and the rear of the machine in radians. The gyroscope measures angular velocity in radians/s while the machine is driving. Even if the hinge and the gyro is measured in different units it can be seen in section 3.1 Figure 8 that they correlate with each other. Consider the following scenarios described with the help of Figure 7.
• Scenario 1: The machines in Figure 7a & 7b are driving. The impact on the hinge and gyro are the following. In 7a the hinge angle sensor will give a value of 0 as well for the gyro. In 7b the hinge angle sensor will have a nonzero value as well as the gyro assuming that the machine is driving at a constant speed. According to Epiroc the noise in the gyro also increases when the machine makes sharp turns.
• Scenario 2: In this scenario the machines in Figure 7a & 7b are standing still. The hinge and gyro values in 7a will be the same as in scenario 1. In 7b the hinge will have a constant nonzero value but the gyro will be 0 since there are no angular velocity when the machine is standing still.
(a) (b)
Figure 7: Vehicles with different configurations
Because of the problem that arises in scenario 1 & 2 in Figure 7b the data set created from the logs files can not contain sequences where the machine is standing still or when turning sharply due to noisy data from the gyroscope. The data set therefore contains of sequences of 100 time stamps which corresponds to 5 seconds from the log files that were provided from Epiroc when the machine is driving along a straight segment of the mine tunnel. The data sets contains of raw hinge and gyro values. How the data set was created is explained in further detail in section 3.1.
In the list below are the different features which were extracted from the data set presented that will later be fed to the SVM.
1. Hinge angle: Raw hinge-angle values measured in [rad]. As mentioned above, the data set contains of 100 time stamps. Therefore will this feature be a 1x100 vector where every element is the value of the hinge-angle sensor at the specific time stamp.
2. Gyro data: Raw gyro data values measured in [rad/s]. This feature is a 1x100 vector where every element is the value of the gyro at the specific time stamp.
3. Difference of hinge and gyro: The difference of the hinge and gyro values, this idea was formed while discussing with one of the employees about the logs files. The hypothesis is that if the hinge-angle and the gyro have a large difference it could be a sign of a faulty hinge-angle sensor. This feature is based on feature 1 & 2 which both are 1x100 vectors where the difference of each element is calculated, resulting in a new vector that is of the same dimensional size of 1x100.
4. Mean of the differences: From the 100 time stamps from the feature above the mean will be the calculated, resulting in a scalar value.
5. Standard deviation of the differences: under the same time interval as the feature above, the standard deviation of the difference is instead calcu-lated, resulting in a scalar value.
6. Difference2: In this feature the differences of the hinge-angle and the gyro are squared. This comes from the idea that mean squared error is often used as a criteria in optimization problems. This feature is based on feature 3 where every element in the 1x100 vector is squared, therefore this feature will also be a 1x100 vector.
7. (Mean of the Differences)2: for the same reason above the mean of the
differences are also squared. This feature is based on feature 4 which is a scalar, therefore this feature will also be a scalar.
8. Absolute value of Difference: A hypothesis is that it is easier to linearly separate the classes when there only are positive values. This features is based on feature 3 and will therefore have the same dimensional size. 9. Absolute value of Mean of the Differences: for the same reason as
above, the absolute values of the mean of the differences is instead calcu-lated. This features is based on feature 4 and will therefore have the same dimensional size.
More details on how these features are going to be used will be explained in section 3.1 and 4.1.
2.2
Optimization-based Classifier
In this section the implementation of the optimization-based classifier is described, starting with introducing scan matching. The following sub chapters introduces two different algorithms that are used for scan matching, ICP (Iterative Closest Point) and NDT (Normal Distributions Transform).
2.2.1 Scan matching
Scan matching is a technique to compute the transformation between two point clouds, e.g. ones that have been recorded by a laser rangefinder. In this thesis two different algorithms were considered and these are ICP (Iterative Closest Point) and NDT (Normal Distributions Transform). Both of these algorithms use laser data which consists of an angle and the length of each laser beam to which it registers an obstacle. The main difference is that ICP takes each point that is registered into consideration and NDT tries to optimize the scan using the local surface around the points.
2.2.2 ICP & NDT
The ICP (Iterative Closest Point) algorithm, introduced by Besl and McKay [14] attempts to align geometrical scannings where the initial relative pose is known. The ICP algorithm works such that it iterates through all the scans of points that are close and minimizes the sum of all the point-to-point distances. The main problem with ICP is that it is a time consuming algorithm, and the most resource consuming part is the search. Even if the search is made more effective with efficient data structures. Also evaluation from recent studies show that the NDT algorithm provides more accurate results when faced with data with weak geometric structure [15].
2.2.3 NDT, Normal Distributions Transform
NDT (Normal Distributions Transform) was first introduced by Peter Biber and Wolfgang Straßer as an alternative method of representing laser scannings [16]. When using the NDT algorithm we first divide the area to what will be referred to as cells, which will all be a constant size. The result will be a grid structure around
the LHD machine with laser data in the cells. The algorithm then combines the normal distributions in point clouds that describes the probability of containing a surface point or not, at a certain position [16, 15].
An important difference between NDT and ICP is that ICP has to search for the closest point in each iteration in the other point cloud. By using NDT, the representation for the normal distribution cells is calculated once and the lookup to find the closest point in the other scan is done at constant time.
The alignment between two scans is made by mapping the coordinates by describ-ing the translation and rotation between two scans given by
T : x 0 y0 ! = cos(θ) − sin(θ) sin(θ) cos(θ) ! x y ! + tx ty ! . (9)
The translation from one scan to the other is described with tx and ty, the angle
θ describes the rotation between them. To find the parameters to optimize (p = (tx, ty, θ)) Newton’s algorithm is used. This is done by iteratively solving
H∆p = −g (10)
using Newton’s method where H and g is the Hessian and gradient from Equation 11. In each iteration an increment Δp is added such that p ←− p+∆p. A detailed explanation on how the Hessian and the gradient are calculated is described in Biber and Straßer [16].
The approach to match the scans builds on that an NDT of one scan is created and for each sample of the second scan the points are mapped into the coordinates of the first scan. To determine if the matching was done with a good estimate, a score is calculated by summing the normal distributions of each mapped point according to score(p) =X i − exp−(x 0 i− qi)t P i −1 (x0i− qi) 2 (11)
where p is the vector that holds the parameters to estimate, xi represents the
reconstructed point of laser scan sample i in the coordinate frame of the second scan. The point of the first scan mapped according to the parameters in vector p
is represented as x’i. Σiis the covariance matrix and qirepresents the mean of the
normal distribution from the NDT in the first scan.
The scoring function is negated due to optimization problems is generally a mini-mization problem and the goal here is to find the solution with the maximal score. The algorithm described above is called P2D-NDT for the reason that it maps points from one point cloud on to the other, in this thesis an extension of this algorithm will be used which is called D2D-NDT. The D2D-NDT algorithm oper-ates only on NDT models and works such that it minimizes the distances between pairs of distributions in two NDT models. See Magnusson et al. [15] for further details.
3
System Architecture & Implementation
This section begins with introducing how data for each of the models (machine learning and scan matching) were gathered and formatted into suitable formats. Some MATLAB scripts provided by Epiroc are mentioned, these scripts was im-plemented by Epiroc to extract data from logged routes and simulate a playback of the selected route. At the end of each subsection, implementation of the models are described and information about tools and resources used for implementation is introduced.
3.1
Machine Learning Implementation
The machine learning model was implemented in MATLAB using the statistics and machine learning toolbox. The toolbox provides various implementations of machine learning algorithms as well as functions for data analysis and evaluation [17].
For the machine learning approach the data from the logs had to be labeled as “needs calibration” and “does not need calibration”. To be able to assign a label to the data points from the logs, the logs were analysed using the loadplaylog, plotplayload and evaluateLocalization scripts. These scripts were provided by Epiroc. loadplaylog is used to convert the logs from binary files into text files which then get loaded into a MATLAB workspace. plotplaylog displays data from the logs and evaluateLocalization runs a simulation of the chosen log file. To be able to determine if the hinge-angle sensor was in need of calibration the hinge and gyro values were plotted against each other. Some logs files were pro-vided where Epiroc suspected that the hinge angle sensor was uncalibrated and those log files were used as a reference in order to be able to label other log files that was missing a label. This method turned out to be very time consuming and the reference logs that were given were hard to compare to the other logs since no specific reason for why the reference logs assumed to have calibration errors was given. Therefore a hinge angle offset α (α = [−3, −1.2, −0.8, −0.6, 0.6, 0.8, 1.2, 3]) was manually added on logs that were assumed to not have a calibration error as shown in Figure 8. The reason for the selected values of α is explained in section 4. As mentioned earlier the machine had to be driving along a relatively straight section of the tunnel to avoid the problems discussed in Section 2.1.3, Scenario 1 & 2, Figure 7b. While plotting the hinge and gyro against each other, straight sections of the tunnels could also be determined.
Figure 8: Plot of a correctly calibrated hinge-angle sensor shown in blue. The simulated positive & negative hinge-angle offsets are shown in purple. The gyro values are plotted in black. Hinge values are measured in radians and gyro values are measured in radians/s
With the simulated hinge-angle offsets 10 data sets could be constructed using the different values for α. The data sets contains gyro and hinge values from a sequence of 100 time stamps or 5 seconds from 212 logs where the class distribution is even, meaning that half of the data points in the set has the label ”does not need calibration” and the other half has the label ”need calibration”.
To be able to train the SVM model with the training function that was provided from the MATLAB toolbox, the features previously mentioned in section 2.3 were engineered and formatted to fit the training function. The data sets were also divided into a training set and a hold out set. The training set are used for training the model and to be used in the cross validation evaluation and to optimize the hyper-parameters γ and C. The hold out set was used in the final evaluation of the classifier. The cross-validation and the final evaluation will be explained in further detail in section 4.2. A visualization of the classifier can be seen in Figure 9.
Figure 9: Flow chart of the machine learning implementation.
3.2
Optimization-based Implementation
The optimization-based classifier builds on an implementation provided in the framework NDT Core. The original implementation is used to match two point clouds and get the transformation required to map one cloud on to the other. For the implementation in this project, a test program was designed that expected input as a data file containing the configuration of the hinge-angle with coordinates and current angle at the time the scans were registered, followed by a PCD file representing the map of the logged route, followed by two scans, one from the front of the machine and one from the rear, also in PCD format.
A PCD file (Point Cloud Data file) is a simple format for registering point clouds. The PCD files consists of a header to describe the input expected. Values to define are size and type of each value and also how many values to be expected, followed by all the data points necessary to describe the point cloud. In this implementation x and y is sufficient and it will be where each laser beam registers an object i.e, 180 points for each scanner. The reason why only x and y is needed is that the laser rangefinders on the machine only registers data in a two-dimensional plane. [18]
implemented to the systems in this thesis, it supports translations that are not physically possible due to the front and rear of the LHD machines being fixated at the articulation point (i.e the hinge). As an attempt to overcome this, all the scans will be centered w.r.t the hinge-angles world coordinates and angle, which will be origin in the scan matching. The reason for doing the scan matching close to origin is that if a point is, say 200 meters away, even a very small angle can make a big difference which will make the scan matching harder. Also, choosing the hinge as origin makes it possible to compare how much each point cloud has rotated. The idea is that if comparing the front and rear point cloud after scan matching generates a transformation, the hinge had to be adjusted for the scan matcher to be successful. An example flow-chart of how the optimization-based classifier works is presented in Figure 10.
Figure 10: Flow-chart of the optimization-based implementation.
The idea with this implementation is to find the best transformation so that the two scans match the mine walls. If increasing or decreasing the hinge-angle of the machine was the optimal way to do it then it could be assumed that the sensor is in need of calibration.
Figure 11: Visualization of the result from using NDT Core with the imple-mentation developed for this thesis. The red dots represent the map, yellow dots represent the rear scan and green dots represent the front scan. After transforma-tion the rear is represented as bright blue and front as dark blue.
The example in Figure 11 is the visual output expected when the hinge-angle needs to be calibrated, since only one end of the machine needs to be adjusted significantly. This can be hard to tell from just one pair of scans matched, but if matching the following sequences from the route estimates that the same trans-formation has to be done one could say with more certainty that the sensor is in need of calibration.
For the optimization-based classifier the data is extracted from several different routes and saved in PCD files. Since the provided machine logs and scripts to analyze them come in already implemented MATLAB scripts, we could simply add extra features to the scripts for storing the desired data in the folder of the machine log that was currently being analyzed. For the data extraction there were two features that had to be implemented, converting the laser data from each log into PCD format and converting the coordinates of the map walls into PCD format.
4
Result
This section begins by introducing evaluation metrics that will be used to evalu-ate the models (machine learning and scan matching). The following subsections describes how each of the models were evaluated.
4.1
Evaluation
Routes for evaluation
The routes used for evaluation were selected by examining them in the MATLAB script introduced in section 3.1 called evaluateLocalization to be able to see that the machines were actually running and that the laser readings seemed to behave as expected when the hinge-angle sensor is calibrated. As specified by Epiroc, an error of 0.5◦ to 1◦ in the hinge-angle calibration could cause the machine to behave incorrectly, if an error that would be even greater the machines would not function at all. Therefore the log files were modified with an offset α to simulate a log file that were in need of calibration. The offsets that was used was set to α = [−3, −1.2, −0.8, −0.6, 0.6, 0.8, 1.2, 3]. For testing purposes two edge cases of 3◦ and −3◦ was added to be able to evaluate if the classification frameworks would perform the same or better on larger offsets.
4.1.1 Evaluation metrics
The outcome of the tests that were done contains of True positives, False posi-tives, True negatives and False negatives. These outcomes are later used when calculating the score of the evaluation metrics presented further below.
• (TP) True Positive: A sensor that was classified as needing calibration when it needed calibration.
• (FP) False Positive: A sensor that was classified as needing calibration when it did not need calibration.
• (TN) True Negative: A sensor that was classified as not needing calibra-tion when it did not need calibracalibra-tion.
• (FN) False Negative: A sensor that was classified as not needing calibra-tion when it needed calibracalibra-tion.
Accuracy
Accuracy is the proportion of correctly predicted cases i.e. TP & TN and the total number of examined cases and gives an overall effectiveness of the model[19]. The score ranges between 0 to 1, where a score of 0 means that the classifier classified all cases wrongly and a score of 1 means that all cases are classified correctly. An accuracy score of 0 is not necessarily bad since then the predictions simply can be swapped resulting in an accuracy of 1. Therefore the worst possible score of accuracy is 0.5 where the classifier has guessed all cases as either positive or negative, assuming that the data set contains equally many positive cases as negative cases.
Accuracy = T P + T N
T P + F P + T N + F N (12)
Precision
Precision is the number of the correctly predicted positive cases divided by the total number of positive predicted cases [19]. Precision gives a score on how often the model gives false alarms. A high score on precision means that the classifier gives a reliable answer when a positive case is classified.
P recision = T P
T P + F P (13)
Recall
Recall is a score of how many of the positive cases that the classifier catches. A high score on Recall means that the classifier catches most of the positive cases. A high recall often comes at the expense of a lower precision. A low score of Recall means that the classifier missed many of the positive cases because the classifier needs to be certain that a calibration error exists.
Recall = T P
T P + F N (14)
4.2
Evaluation of ML-based Classifier
The machine learning implementation using the SVM algorithm will be cross val-idated in order to see how well the model can handle new data it has not been training on. Cross validation is a technique widely used when evaluating classifi-cation models. A k -fold Cross validation divides the data set into k subsets where
k -1 subsets are used for training and the last subset is used for testing, the model then trains on the training set and evaluates on the testing set. After that the test set gets swapped with one of the training subsets and repeats the same process until all subsets of the data have been used as the test set.[20]
Figure 12: Visualization of k -fold cross validation on different iterations, figure taken from Xiong et al. [20]
Features sets
The features previously mentioned in section 2.1.3 were organized according the list below to create the ”feature sets” that were used in the evaluation of the ML-classifier. The sets are based on the dimensional size. Feature set 1-4 are bases on features that are a vectors of size 1x100. Feature set 5-8 are based on the features that are scalar. Feature set 5-8 also contains of features that are statistical measures such as the standard deviation and mean of the differences between the gyro and hinge values.
• Feature set 1: [Hinge, Gyro]
• Feature set 2: [Hinge, Gyro, Difference]
• Feature set 3: [Hinge, Gyro, Difference, Absolute value of Difference] • Feature set 4: [Hinge, Gyro, Difference, Difference2]
• Feature set 5: [Mean of Difference, Standard Deviation]
• Feature set 6: [Absolute value of Mean of the Difference, Standard Devia-tion]
• Feature set 7: [(Mean of Difference)2, Standard Deviation]
• Feature set 8: [Absolute value of Mean of the Difference, Standard Devia-tion, Mean of Difference]
4.2.1 Tests
The first test to evaluate the machine learning classifier consisted of training a model using each of the 8 different feature set specified above with two different kernels, linear and RBF. The models were then cross-validated with k set to 10. The hyper-parameters γ and C was optimized using functions provided from the Statistics and machine learning toolbox from MATLAB. Accuracy, precision and recall were calculated for each feature set and for both kernels. This test was repeated for each of the data sets that was specified earlier in section 3.1. The results of these tests are presented in Appendix A.
After the first test it was concluded that feature sets 5-8 were the best scoring feature sets for both of the kernels over all of the tested data sets, this is shown in the Appendix A. Therefore the final evaluation which uses the hold out set, only consisted of those four features sets with their respectively corresponding hyper-parameters that were optimized in the previous test. The reason why those features set scored better than the other four sets are probably because features set 5-8 are heavily bases on statistical measures, rather than raw values which feature set 1-4 are heavily bases on. As in the previous test, accuracy, precision and recall were calculated. The results for each individual hinge-angle offset is presented in Tables 1-10.
In order to get an evaluation on which feature set that performs the best. The mean accuracy, precision and recall were calculated over all of the hinge-angle offsets α. The result is shown in Table 11.
Final evaluation of data set 1
Feature set Linear Kernel RBF Kernel
Accuracy Precision Recall Accuracy Precision Recall
5 0.67 0.56 0.71 0.44 0.56 0.45
6 0.61 0.67 0.60 0.56 0.67 0.55
7 0.56 0.56 0.56 0.61 0.67 0.60
8 0.57 0.57 0.57 0.44 0.57 0.45
Table 1: Score of evaluation metrics using the hold out set, hinge-angle offset = 0.6◦
Final evaluation of data set 2
Feature set Linear Kernel RBF Kernel
Accuracy Precision Recall Accuracy Precision Recall
5 0.67 0.78 0.64 0.67 0.67 0.67
6 0.72 0.89 0.67 0.72 0.89 0.67
7 0.67 0.78 0.64 0.67 0.78 0.64
8 0.67 0.78 0.64 0.72 0.89 0.67
Table 2: Score of evaluation metrics using the hold out set, hinge-angle offset = −0.6◦
Final evaluation of data set 3
Feature set Linear Kernel RBF Kernel
Accuracy Precision Recall Accuracy Precision Recall
5 0.89 0.89 0.89 0.83 1.00 0.75
6 0.61 0.67 0.60 0.61 0.67 0.60
7 0.61 0.67 0.60 0.61 0.67 0.60
8 0.72 0.78 0.70 0.72 0.78 0.70
Table 3: Score of evaluation metrics using the hold out set, hinge-angle offset = 0.8◦
Final evaluation of data set 4
Feature set Linear Kernel RBF Kernel
Accuracy Precision Recall Accuracy Precision Recall
5 0.72 0.78 0.70 0.78 0.89 0.73
6 0.72 0.89 0.67 0.72 0.89 0.67
7 0.78 0.89 0.72 0.78 0.89 0.72
8 0.78 0.89 0.72 0.78 0.89 0.72
Table 4: Score of evaluation metrics using the hold out set, hinge-angle offset = −0.8◦
Final evaluation of data set 5
Feature set Linear Kernel RBF Kernel
Accuracy Precision Recall Accuracy Precision Recall
5 0.94 0.89 1.00 0.83 1.00 0.75
6 0.67 0.67 0.67 0.67 0.67 0.67
7 0.67 0.67 0.67 0.67 0.67 0.67
8 0.83 1.00 0.75 0.83 1.00 0.75
Table 5: Score of evaluation metrics using the hold out set, hinge-angle offset = 1◦
Final evaluation of data set 6
Feature set Linear Kernel RBF Kernel
Accuracy Precision Recall Accuracy Precision Recall
5 0.72 0.78 0.70 0.83 0.89 0.80
6 0.78 0.89 0.73 0.78 0.89 0.73
7 0.78 0.89 0.73 0.78 0.89 0.73
8 0.78 0.89 0.73 0.78 0.89 0.73
Table 6: Score of evaluation metrics using the hold out set, hinge-angle offset = −1◦
Final evaluation of data set 7
Feature set Linear Kernel RBF Kernel
Accuracy Precision Recall Accuracy Precision Recall
5 0.94 0.89 1.00 0.89 1.00 0.82
6 0.78 0.89 0.73 0.67 0.67 0.67
7 0.67 0.67 0.67 0.78 0.78 0.78
8 0.89 1.00 0.82 0.89 1.00 0.82
Table 7: Score of evaluation metrics using the hold out set, hinge-angle offset = 1.2◦
Final evaluation of data set 8
Feature set Linear Kernel RBF Kernel
Accuracy Precision Recall Accuracy Precision Recall
5 0.72 0.78 0.70 0.83 0.89 0.80
6 0.78 0.89 0.73 0.78 0.89 0.73
7 0.78 0.89 0.73 0.78 0.89 0.73
8 0.78 0.89 0.73 0.83 0.89 0.80
Table 8: Score of evaluation metrics using the hold out set, hinge-angle offset = −1.2◦
Final evaluation of data set 9
Feature set Linear Kernel RBF Kernel
Accuracy Precision Recall Accuracy Precision Recall
5 0.94 0.89 1.00 1.00 1.00 1.00
6 1.00 1.00 1.00 1.00 1.00 1.00
7 1.00 1.00 1.00 1.00 1.00 1.00
8 1.00 1.00 1.00 1.00 1.00 1.00
Table 9: Score of evaluation metrics using the hold out set, hinge-angle offset = 3◦
Final evaluation of data set 10
Feature set Linear Kernel RBF Kernel
Accuracy Precision Recall Accuracy Precision Recall
5 0.94 0.89 1.00 1.00 1.00 1.00
6 1.00 1.00 1.00 1.00 1.00 1.00
7 1.00 1.00 1.00 1.00 1.00 1.00
8 1.00 1.00 1.00 1.00 1.00 1.00
Table 10: Score of evaluation metrics using the hold out set, hinge-angle offset = −3◦
Summation of feature sets
Feature set Linear Kernel RBF Kernel
Accuracy Precision Recall Accuracy Precision Recall
5 0.65 0.81 0.83 0.81 0.89 0.78
6 0.77 0.85 0.74 0.75 0.82 0.73
7 0.75 0.81 0.73 0.77 0.82 0.75
8 0.73 0.88 0.77 0.80 0.89 0.76
Table 11: Calculated mean of accuracy, precision and recall over all hinge-angle offsets from Table 1-10.
The results of the tests shown in the tables above (1-10) indicates that the classifier has a weakness when it comes to identifying hinge-angle offsets that are small, but as expected gets significantly better as the offset increases to the extreme values of the offsets. In Table 11 it can be concluded that feature set 5 and feature set 8 with the RBF kernel are slightly better than the other feature sets. Feature set 5 has 1% better accuracy and 2% better recall than feature set 8. Those two features set both contains the mean of the differences and the standard deviation of the differences of the hinge and the gyro values. Feature set 8 also included the absolute value of the mean of the differences, which seems like it only made it harder for the classifier to make correct predictions. In Table 1-10 it can be seen that the Feature set 5 scores well on all hinge-angle offsets except α = ±0.6. Therefore the method does not meet the requirements specified by Epiroc to be useful in production. Different approaches to increase the effectiveness of the method are discussed in section 5.1 and section 5.6.
4.3
Evaluation of Optimization-based Classifier
To evaluate the optimization-based classifier we will look at two scans taken at the same time, one from the front and one from the rear of the machine and scan match them to the map. The idea is that if comparing the rotation angles for them is 6= 0, it means that the hinge-angle has rotated for the scan matching to be successful.
The adjustment needed on the hinge-angle is measured in degrees and is calculated as Tf− Tr where T is transformation in degrees for each part of the machine (front
f and rear r ). As we can see in Figure 13, the left image generates zero hinge-angle transformation since the front scan has rotated up and the rear scan has rotated down of equal magnitude thus resulting in Tf− Tr= 0, in this case we have a rigid
transformation. In the right image a small hinge-angle transformation is generated because Tf − Tr6= 0, which is a non-rigid transformation.
Figure 13: The left image show a scan with no transformation of the hinge-angle, the right image show a scan with 0.4◦ transformation of the hinge-angle. Green and dark blue dots represents the front scan before respectively after transforma-tion. Yellow and bright blue dots represents the rear scan before respectively after transformation.
Tests
In an initial test, a sequence of 10 scans from a logged route where the readings seemed to register well to the map walls were selected, and the same sequence of scans but with a modified hinge-angle offset α = −3◦. The adjustment in degrees needed on the hinge-angle for each scan is presented in Table 12.
Initial test Scan Adjustment from the logged
route
Adjustment from the modi-fied route 1 0.16◦ 1.32◦ 2 0.51◦ 1.28◦ 3 0.12◦ 1.05◦ 4 0.13◦ 1.46◦ 5 0.18◦ 1.52◦ 6 0.55◦ 0.94◦ 7 0.57◦ 1.54◦ 8 0.25◦ 1.58◦ 9 0.46◦ 0.84◦ 10 0.37◦ 1.22◦
Table 12: Initial test of the optimization-based classifier on two sequences each containing 10 scans, each from the same route and time stamps. Left column with no offset and right column with −3◦ offset on the hinge-angle.
The expectations of the initial testing was that increasing the hinge-angle offset α by a certain magnitude, the adjustment needed would increase equally. Initial testing show that the adjustment increases, but not with 3◦ which could be due to low precision from the map files, as explained in section 1.4.2. Due to the low precision, the scan matcher could find transformations that gives a good enough scan matching score close to the map walls. Further discussion about sources of error is discussed in section 5.2.
The following evaluation of the classifier will be to investigate several sequences like the initial one presented in Table 12, but covering a larger sequence of scans. The sequences will be 100 scans from several modifications of 24 selected routes, where each route has the different modifications of hinge-angle offset α introduced in section 4.1.
After gathering information needed about the logs, accuracy, precision and recall can be calculated as in section 4.1. These metrics will be evaluated for 24 different logs, all with the specified hinge-angle offsets α. The mean adjustment needed for all the routes with the various hinge-angle offsets α is presented in Figure 14. Each route in the figure is considered a test case, where each test case is the same route, but with various modifications of the hinge-angle. This means that each cell in Table 14 is a sequence of 100 scans modified with a specified hinge-angle offset α from route 1 to 24.
Figure 14: Testing of the optimization-based classifier with mean adjustment for each test case in each cell. Green = true positive, red = false negative, yellow = true negative, blue = false positive.
As previously mentioned, the error that we are trying to classify should be detected somewhere between 0.5◦ and 1◦. Therefore it was decided to classify the sequences where the mean adjustment is ≥ 0.5◦ as positives (needs calibration) and mean adjustment < 0.5◦ as negatives (no need for calibration). Figure 14 shows that it was difficult to classify the sequences with low and no offset modification. One reason that this could be an issue in the implementation, is that there is no certain ground truth to that the logs were correctly classified to begin with.
Evaluation of optimization-based classifier
α Accuracy Precision Recall
+ 0.6 0.52 0.52 0.58 − 0.6 0.58 0.57 0.71 + 0.8 0.63 0.59 0.79 − 0.8 0.58 0.57 0.71 + 1 0.63 0.59 0.79 − 1 0.54 0.54 0.63 + 1.2 0.65 0.61 0.83 − 1.2 0.56 0.55 0.67 + 3 0.71 0.64 0.96 − 3 0.71 0.64 0.96 Average 0.61 0.58 0.76
Table 13: Accuracy, Precision and Recall for each of the hinge-angle offsets used in evaluation of the optimization-based classifier.
The evaluation metrics accuracy and precision is relatively low in this model due to the difficulty of classifying the sequences with low or no hinge-angle offsets α modified. But as seen on recall, it increases significantly when increasing α. This means that the classifier succeeds in classifying the positive cases at its best 96% of the sequences when α = ± 3.
The optimization-based classifier implemented on an LHD machine would work such that it registers the data recorded, with five seconds intervals. Within that five seconds interval, the data recorded is analyzed and the scan matcher algorithm with the current map file that the machine is running in, gets executed. If the mean adjustment needed for the range finder data to match the map is ≥ 0.5◦, send a notification that the hinge-angle sensor might need calibration.
Further testing could be done from data registered from an LHD machine that is newly calibrated. This could prove useful to analyze the actual mean adjustment needed for each sequence. The value recorded could then be stored and used in the classifier. When the following sequences in the route starts to deviate from the assigned value, send a notification that calibration might be needed.
5
Discussion
5.1
Machine Learning Classifier
As seen in the tables from the section 4.2.1, the machine learning classifier have a weakness when the hinge angle-offset is small. As the hinge-angle offset increases the classifier gets better and better. An average score of the evaluation metrics over all of the different hinge-angle offsets is shown in Table 11, where the best scoring feature set was feature set 5 with an accuracy, precision and recall of 0.81, 0.89 and 0.78.
Even if those evaluation scores seems good, there are one big concern with the data set that was created that could have an impact on the result. The concern is that the logs files that were provided were not labeled. As previously mentioned in section 3.1 the data set was created with logs that were assumed to not have a calibration error and adding an hinge-angle offsets to create logs that had a calibration error. Because of the assumptions by us and Epiroc, the ground truth labels are not certain, which could cause the classifier to predict incorrectly. An other thing to consider is that the approach of simply comparing the hinge and gyro values may be a bit naive. For example a low tire pressure could cause incorrect values from the hinge-angle sensor according to our supervisor at Epiroc. An other factor that could have an impact is the time since last calibration. Intu-itively a machine that has been driving for a long time without calibration runs a higher risk of having a calibration error due to mechanical wear. Other statistical measures could possibly also cause the effectiveness to increase. In Zhao et al. [21] a correlation coefficient r is presented that could be used to detect abnormalities earlier compared to using only sensor data from two sensors. Zhao et al. have used data from electric generators in their tests. In Equation 15, x and y are data from two different sensors.
r = Pn i=1(xi− x)(yi− y) pPn i=1(xi− x)2 Pn i=1(yi− y)2 (15)
r in the equation above could possibly be useful as a feature in the ML-method. In short, better feature engineering could prove useful for further testing and de-velopment.
An option to tackle the problem with the unlabeled logs is to simply start to save labels to log files that stopped functioning due to a faulty hinge-angle sensor. This
would not only mean that logs with real calibration errors would exist unlike now that just simulated logs exist, but also logs with true ground truth labels for the logs that does not have a calibration error.
An other option could be to use an unsupervised learning approach instead which don’t need labeled data. For example could an ”one class SVM” for outlier de-tection be implemented. An other option could possibly be an other variant of a clustering algorithm. A clustering method which that uses a density-based ap-proach to find several clusters such as DBSCAN, is probably better suited for the data that we have been provided with. The data points in the data we have been given that probably needs calibration is not centered in a cluster, they are more dispersed around the cluster that is formed by the data points that does not need calibration. Therefore a partitioning method where a number k is set which deter-mines how many clusters the method should create, would be hard to implement since it would be hard to guess how many clusters k that would be needed to separate the classes.
Some initial tests were done using the DBSCAN approach, but due to shortage of time the implementation was never extensively tested. The data set that was used was a smaller sample of 62 data points, where the data class distribution was heavily unbalanced, with a greater amount of data points being negatives (not needing calibration). The same feature sets presented in section 4.2 were used in the tests, where also feature set 5 gave the best results. The parameter minP ts which defines the minimum number of neighbors for a core point, i.e. how many data points are required to form a cluster was set to 2. The parameter ε which defines the radius of neighborhood around a point was set to 0.01. The distance metric used was euclidean distance. With the these settings the DBSCAN algo-rithm created 5 clusters using features set 5, one large cluster and 4 smaller clusters along with data points that did not belong to any cluster. The data points that did not belong to any cluster was categorized as outliers by the algorithm. The outliers and the 4 small clusters were labeled as positives (needing calibration) and the large cluster as negatives (not needing calibration). This yielded in an accuracy of 78%, precision and recall were not calculated in these tests. It would be interesting to further explore this approach in future development.
5.2
Optimization-based Classifier
An average score of the evaluation metrics over all of the different hinge-angle offsets for the optimization-based classifier can be seen in section 4.3 in Table 13. This score resulted in accuracy, precision and recall of 0.61, 0.58 and 0.76.
Both with this average score and as seen in Table 13 in section 4.3, accuracy and precision is not optimal. The main issue if this classifier is implemented on an actual LHD machine is that it would produce many false alarms that tells us that calibration is needed when it actually is not needed. The reason for these two metrics being as low as they are could be because there is no ground truth to that the logs were well calibrated to begin with. With recall however, we can see that increasing the hinge-angle offset α also increases recall significantly. This means that the classifier is effective in finding the positive cases (needs calibration). This metric is only dependant on the positive cases and is therefore not affected by how imprecise the routes with no α modification was classified, as accuracy and precision are.
By using the visual representation of the scan matching we could see that in the majority of the test cases the scans after transformation aligned very well with the map. This indicates that the scan matching is done with a good result but with regard to the low fault tolerance and low resolution maps the implementation is not precise enough to distinguish when the hinge-angle sensor needs calibration. Also the creation of the map files was done by using edge detection from the metric maps provided and resulted in that some of the walls are represented as a clusters rather than a line. In some of the cases during the scan matching the best transformation was found in these clusters which gave us an inaccurate result (As seen in Figure 15).
Figure 15: Failed scan matching due to clusters outside of the map walls.
The implementation as of now allows smaller translations which is not physically possible due to the parts of the machine being fixated at the joint holding the two together. In most cases we overcame this by doing the scan matching around origin which was assigned to be at the hinge. To further ensure that unusual translations would not occur we could filter out the unsuccessful scan matches by adding constraints to the implementation. The constraints for this implementation was done such that transformations for each scan was not allowed to be greater than 20◦ which is where we started to notice unexpected results. The constraint at 20◦ is much higher than the error that would be possible for a working machine since the error we are trying to detect is suspected to be somewhere between 0.5◦ and 1◦, but for testing purposes we decided to only filter out the scans that caused the scan matching to behave incorrectly.
Another aspect that we didn’t take into consideration from the start is scan match-ing for the walls inside the shafts. Since these shafts are the extraction point for minerals, the map coordinates might not match the laser readings if the shafts are full with minerals due to the range finders on the machine will pick up those as well. This could result in that the scan matching algorithm might try to move
the scan towards the wall as it is seen from the map files, thus resulting in an unexpected result that tells us to turn the hinge-angle when it actually wasn’t needed.
5.3
Comparison of the models
Accuracy
The overall effectiveness of the classifiers is measured in accuracy. As seen in Table 14, the ML classifier is more effective in classifying both the positive and the negative cases, with the highest in the edge cases of α = ± 3◦. In these cases everything was classified correctly. The errors that Epiroc specified as where the machine should be starting to behave incorrectly was suspected to be somewhere between 0.5◦ and 1◦. The ML classifier started to detect these with good enough accuracy at ± 0.8◦, the OB classifier started detecting them at 3◦. Good enough accuracy as in useful for the customer, which Epiroc specified to be atleast 70% of the time. α ML classifier OB classifier + 0.6 0.44 0.52 − 0.6 0.67 0.58 + 0.8 0.83 0.63 − 0.8 0.78 0.58 + 1 0.83 0.63 − 1 0.83 0.54 + 1.2 0.89 0.65 − 1.2 0.83 0.56 + 3 1.00 0.71 − 3 1.00 0.71
Table 14: Comparison of accuracy for both models.
Precision
With precision it is measured how often the classifiers gives false alarms. As we can see in Table 15, the ML classifier is less prone to classifying a case which is not in need of calibration wrongly. Also in this case with α = ± 3◦ the ML classifier gives no false alarms. This metric however is not as important to Epiroc due to that if the overall route is proceeding well, a false alarm can simply be ignored.
α ML classifier OB classifier + 0.6 0.56 0.52 − 0.6 0.67 0.57 + 0.8 1.00 0.59 − 0.8 0.89 0.57 + 1 1.00 0.59 − 1 0.89 0.54 + 1.2 1.00 0.61 − 1.2 0.89 0.55 + 3 1.00 0.64 − 3 1.00 0.64
Table 15: Comparison of precision for both models.
Recall
With recall it is measured how well the classifiers detects the positive cases (needs calibration). With this metric both of the classifiers performed well on the edge cases α = ± 3◦. Recall is more important than precision according to Epiroc but as previously mentioned accuracy is the main metric that should be considered. However both of the methods scores well on recall on α ≥ 0.8◦ and α ≤ −0.8◦.
α ML classifier OB classifier + 0.6 0.45 0.58 − 0.6 0.67 0.71 + 0.8 0.75 0.79 − 0.8 0.73 0.71 + 1 0.75 0.79 − 1 0.80 0.63 + 1.2 0.82 0.83 − 1.2 0.80 0.67 + 3 1.00 0.96 − 3 1.00 0.96
Overall
Based on the evaluation metrics we can see that the ML classifier is able to perform well on detecting hinge-angle errors α ≥ 0.8◦and α ≤ 0.8◦. The OB classifier starts detecting errors well on errors at 3◦. This means that the ML classifier performs slightly better on the data used for this thesis. The reason why the ML classifier performs better than the OB classifier, is most likely due to the OB classifier is heavily depended on the map files, which discussed earlier has several problems. Both of the implementations has promising potential for future development which will be discussed in section 5.6.
5.4
Fulfilment of Project Requirements
The project requirements were fulfilled in the essence that we managed implement both methods to the extent so that a comparison could be made. in section 5.3 is was concluded that the OB-classifier did not meet the requirements that Epiroc deemed to be useful. The ML classifier show to be slightly better at lower hinge-angle offsets than the OB-classifier. The ML classifier fulfills the requirements to be useful for Epiroc by correctly classifying 70% of the cases for α ≤ −0.8 and α ≥ 0.8, but fails at correctly classify the lowest hinge-angle offsets of α ± 0.6. As discussed in section 5.1, improvements can be made. Ground truth labels of the data set that are certain, larger data set and better features are improvements that could be made just to name a few. Due to shortage of time and substandard results of the lowest hinge-angle offset, the method was never implemented into the RCS. One of the next step in development would be to integrate it to RCS and run tests in real environments as well as looking further into the options of improvement discussed in section 5.1 and section 5.6
With the machine learning classifier the best scoring feature set in the tests with the hold out set gave an accuracy, precision and recall of 81%, 89% and 78% when calculating the average over all hinge-angle offsets. Even if these results seem quite promising, there are problems as with the ground truth labels and not enough data of the factors that could have an impact on the sensor which are discussed in section 5.1. Because of those problems the classifier has a lot of further development potential.
With the optimization-based model, the visual part of the implementation was used mostly to validate that the scans looks correct, such as layout of the map and placement of the scans. The most useful part of the implementation comes from extracting the angles needed for two scans (front and rear) to be a match
