STOCKHOLM SWEDEN 2018,
Tracking of railroads for autonomous guidance of UAVs
using Vanishing Point detection ANTHONY CLERC
I would like to acknowledge the persons that gave me support under different forms while writing the thesis.
Didem Gürdürfor her support and instructive feedback throughout the thesis.
Matthieu Marchandfor being flexible and open-minded when defining the thesis scope.
Michael Oscarssonfor his continuous support in order to move the thesis forward.
Daniel Thorellfor his help and advices concerning my questions in programming.
Damir Nesicfor his constant support with administrative questions.
Guillaume meurissemy opponent for reviewing in detail my thesis report.
De-Jiu Chenfor examining.
I would like to thank Sofía Navarro Heredia for her emotional support throughout the journey of completing the master’s thesis.
Finally, I would like to express my profound and sincere gratitude to my parents, Diane &
Olivier Clerc, who have always been supporting me in following my goals and with who it wouldn’t have been possible to pursue a master abroad.
Anthony Clerc
Tracking of railroads for autonomous guidance of UAVs
Anthony CLERC
Approved Examiner Supervisor
June 14, 2018 De-Jiu Chen Didem Gürdür
Commissioner Contact person
Bombardier Matthieu Marchand
Abstract
UAVs have gained in popularity and the number of applications has soared over the past years, ranging from leisure to commercial activities. This thesis is discussing specifically railroad applications, which is a domain rarely explored. Two different aspects are analysed.
While developing a new application or migrating a ground-based system to UAV platform, the different challenges encountered are often unknown. Therefore, this thesis highlights the most important ones to take into consideration during the development process.
From a more technical aspect, the implementation of autonomous guidance for UAVs over railroads using vanishing point extraction is studied. Two different algorithms are presented and compared, the first one is using line extraction method whereas the second uses joint activities of Gabor filters. The results demonstrate that the applied methodologies provide good results and that a significant difference exists between both algorithms in terms of computation time.
A second implementation tackling the detection of railway topologies to enable the use on multiple rail road configurations is discussed. A first technique is presented using exclusively vanishing points for the detection, however, the results for complex images are not satisfactory. Therefore, a second method is studied using line characteristics on top of the previous algorithm. This second implementation has proven to give good results.
Keywords: Railroad tracking, Autonomous guidance UAV, Railroad topology, Vanishing point detection, Gabor filters, Edge detection
Spårning av järnväg for autonom styrning av obemannade flygfordon
Anthony CLERC
Godkänt Examinator Handledare
14 juni 2018 De-Jiu Chen Didem Gürdür
Uppdragsgivare Kontaktperson
Bombardier Matthieu Marchand
Sammanfattning
Applikationer med obemannade flygfordon (UAVs) har ökat i både popularitet och antal de senaste åren där applikationsområdena spänner över allt från fritidsaktiviteter till kommersi- ella syften. Den här uppsatsen syftar till att specifikt studera applikationer inom järnväg, där antal tidigare studier är begränsat.
Vid utveckling av en ny applikation eller vid migrering av markbaserade plattformar till UAV-system är kunskap om utmaningar som kan uppstå ofta okända. Den här uppsatsen ämnar således uppmärksamma de viktigaste aspekterna att beakta vid utveckling av dessa system.
I uppsatsen studeras implementering av autonom styrning av obemannade flygfordon ovanför järnvägar med hjälp av extraktion av flyktpunkter ur bilder. Två olika algoritmer presenteras och jämförs: linjeextraktion och gemensamma aktiviteter i Gaborfilter. Resultatet påvisar att de använda metoderna ger goda resultat samt att det finns en stor skillnad mellan de olika algoritmerna gällande beräkningstid.
Ytterligare ett problem som diskuteras är igenkänning av rälstopologier för att kunna generalisera algoritmerna till olika typer av rälskonfigurationer. En första implementation använder enbart flyktpunkten för igenkänning av rälstopologin, och det påvisas att resul- tatet inte är tillfredställande för komplexa bilder. Därför studerades en andra metod där linjekaraktäristik adderats till föregående metod vilket uppvisat goda resultat.
Nyckelord: Rälsspårning, Autonom styrning, UAV, Rälstopologi, Flyktpunktsdetektion, Gaborfilter, Kantdetektion
Abstract v
Sammanfattning vii
List of Figures xiii
List of Tables xv
Nomenclature xvii
1 Introduction 1
1.1 Context . . . . 1
1.2 Problem statement . . . . 3
1.2.1 Purpose . . . . 5
1.2.2 Limitations . . . . 6
1.2.3 Research questions . . . . 6
1.2.3.1 Outcomes . . . . 7
1.3 Ethical considerations . . . . 7
1.3.1 Academic concerns . . . . 7
1.3.2 Implementation concerns . . . . 8
1.3.3 Long term concerns . . . . 8
1.4 Outline . . . . 9
2 Research design 11 2.1 Research methodology . . . . 11
2.1.1 Literature research strategy . . . . 12
2.1.1.1 Challenges encountered when migrating to UAV platform 12 2.1.1.2 Autonomous guidance of UAV over railway tracks . . . . 13
2.1.2 Case study approach . . . . 14
2.1.2.1 Challenges encountered when migrating to UAV platform 14
2.1.2.2 Autonomous guidance of UAV over railway tracks . . . . 14
3 Background 17 3.1 Literature research . . . . 17
3.1.1 UAV applications overview . . . . 18
3.1.2 Migration challenges . . . . 18
3.1.3 Autonomous guidance . . . . 20
3.1.3.1 Vanishing point detection . . . . 20
3.1.3.2 Line detection and tracking . . . . 24
3.1.3.3 Visual railway detection by HSV colour extraction . . . . 27
3.1.3.4 Feature-based tracking . . . . 29
3.1.4 Literature review conclusion . . . . 29
3.2 Technical background . . . . 31
3.2.1 Pre-processing . . . . 32
3.2.1.1 Colour-space conversion . . . . 32
3.2.1.2 Blurring . . . . 32
3.2.1.3 Rescaling . . . . 32
3.2.2 Edge detection . . . . 33
3.2.2.1 Sobel . . . . 33
3.2.2.2 Canny . . . . 34
3.2.2.3 Laplace . . . . 35
3.2.3 Line extraction . . . . 36
3.2.3.1 Hough transform . . . . 36
3.2.4 Feature extraction . . . . 38
3.2.4.1 Gabor filters . . . . 38
3.2.5 K-means clustering . . . . 39
3.2.6 Tracking algorithm . . . . 41
3.2.6.1 Kalman filter . . . . 41
4 Implementation 45 4.1 Approach . . . . 45
4.1.1 Disregarded approaches . . . . 45
4.1.2 Applied approach . . . . 46
4.1.3 Framework . . . . 47
4.1.4 Pre-processing . . . . 48
4.1.4.1 Colour space conversion . . . . 48
4.1.4.2 Blurring . . . . 49
4.1.4.3 Rescaling . . . . 51
4.1.4.4 Extract the railways from the background. . . . 51
4.1.5 Line extraction methodology . . . . 51
4.1.5.1 Edge detection . . . . 52
4.1.5.2 Line extraction . . . . 53
4.1.5.3 Vanishing Point extraction . . . . 57
4.1.6 Gabor filters approach . . . . 58
4.1.6.1 Multiple filters method . . . . 59
4.1.6.2 J. Shi method . . . . 61
4.1.6.3 Vanishing Point determination . . . . 63
4.1.7 Vanishing Point Selection . . . . 64
4.1.7.1 Searching Area . . . . 65
4.1.7.2 Determining the optimal vanishing point . . . . 66
4.1.8 Defining rails topologies . . . . 68
4.1.8.1 Detection of the topologies using the Vanishing Point . . 68
4.1.8.2 Turnout definition . . . . 71
5 Results and Analysis 75 5.1 Migration challenges . . . . 75
5.1.1 Ethical challenges . . . . 76
5.1.2 Technical challenges . . . . 77
5.1.3 Legal Concerns . . . . 78
5.2 Definition and Calculation of the Vanishing Point . . . . 79
5.2.1 Measurements . . . . 80
5.2.2 Parametrisation . . . . 81
5.2.3 Analysis . . . . 81
5.2.4 Comparison with the literature . . . . 85
5.3 Extraction of the rails topologies . . . . 86
5.3.1 3 topologies . . . . 86
5.3.2 5 topologies . . . . 88
5.3.3 5 topologies with detection of turnouts . . . . 90
6 Discussion 93 6.1 Discussion . . . . 93
6.1.1 Limitations . . . . 93
6.1.2 Definition and calculation of the Vanishing Point . . . . 93
6.1.3 Extraction of the rail topology . . . . 94
6.1.4 Improvement of the topology detection . . . . 94
6.1.4.1 Localisation of the railway tracks . . . . 95
6.1.4.2 Secondary camera . . . . 95
6.1.5 Control depending on the topology . . . . 96
6.1.6 Control of the UAV . . . . 96
6.1.7 Extraction of the railways from the background . . . . 97
6.1.8 Robustness of the edge detection filter . . . . 98
7 Conclusion and Future work 99 7.1 Conclusion . . . . 99
7.2 Future work . . . 101
Bibliography 103 Appendix A Algorithms review for tracking application 107 Appendix B Survey about migration challenges - Results 109 B.1 Introduction . . . 109
B.2 Question/Answers 1 . . . 109
B.3 Question/Answers 2 . . . 111
B.4 Question/Answers 3 . . . 112 Appendix C Analysis of the Vanishing Point - Dataset samples 115
Appendix D Analysis of topology - Dataset samples 117
Appendix E Vanishing point localisation - Results 119
Appendix F ANOVA analysis of the accuracy 121
Appendix G ANOVA analysis of the elapsed time 123
Appendix H Three Topologies - Results 125
Appendix I Five Topologies - Results 127
Appendix J Five Topologies with detection of turnouts - Results 129
1.1 Repartition of the tasks for the different stakeholders . . . . 4
1.2 Geotrack system . . . . 4
3.1 Vanishing Point explanation . . . . 21
3.2 Algorithm process for vanishing point definition . . . . 21
3.3 Vanishing point localization principle . . . . 23
3.4 Algorithm of Power Line filters using K-means in the Hough space . . . . . 25
3.5 Multiple cluster algorithm . . . . 26
3.6 Initialisation of the B-Spline . . . . 27
3.7 Flow chart of the methodology for saturation extraction . . . . 28
3.8 Process for tower detection in two phases . . . . 30
3.9 Non-maximum suppression algorithm principle . . . . 35
3.10 Hysteresis thresholding algorithm representation . . . . 35
3.11 Representation of the Hough space . . . . 36
3.12 Representation of two straight lines in the Hough space . . . . 37
3.13 Representation of a Gabor Filter . . . . 39
3.14 Analysis of the "elbow point" . . . . 41
3.15 Kalman filter flow . . . . 43
4.1 Flow diagram of the entire approach strategy . . . . 47
4.2 Pre-processing process . . . . 48
4.3 Image of reference . . . . 48
4.4 Color-spaces comparison . . . . 50
4.5 Comparison of the impact of the resolution . . . . 51
4.6 Railway track cropping operation’s steps . . . . 52
4.7 Line algorithm methodology . . . . 52
4.8 Comparison of edge detection filters . . . . 54
4.9 Extraction of straight lines . . . . 55
4.10 Line representation in frame coordinate . . . . 56
4.11 Distance conversion from camera to real world . . . . 57
4.12 Detection of Vanishing Points . . . . 59
4.13 Gabor algorithm process . . . . 59
4.14 Computation of the energy response for different filter angles . . . . 60
4.15 Best energy responses combined . . . . 61
4.16 Voting scheme principle . . . . 63
4.17 Detection of the optimal vanishing point candidates . . . . 64
4.18 Detection of Vanishing Point - With search area . . . . 66
4.19 Definition of the searching area based on the information from Kalman Filter 66 4.20 Computation of the optimal VP . . . . 67
4.21 Computation of the optimal VP - KMean . . . . 68
4.22 Rails topologies . . . . 68
4.23 Vanishing point evolution for a curve compared to a straight line . . . . 69
4.24 Detection of the Vanishing Points for different perspectives . . . . 69
4.25 Separation of the vanishing points in zones . . . . 70
4.26 Detection of turnouts . . . . 72
5.2 Crowd labelling . . . . 81
5.3 Accuracy comparison between two different algorithms in different environ- ment . . . . 82
5.4 Comparison of the offset . . . . 84
5.5 Elapsed time comparison between two different algorithms . . . . 84
6.1 Extraction of the sleepers with Gabor filter . . . . 95
6.2 Representation of the control of the Drone . . . . 97
6.3 Extraction of the railways from the background . . . . 98
C.1 Samples of the analysis of vanishing point dataset . . . 116
D.1 Samples of the topology dataset . . . 118
2.1 Keywords summary . . . . 13
3.1 Summary of the different papers analysed . . . . 17
4.1 Optimal filters’ parameters . . . . 53
4.2 Optimal Hough Tranform’s parameters . . . . 54
4.3 Optimal Gabor filter’s parameters for multiple angles . . . . 60
4.4 Optimal Gabor filter’s parameters for multiple angles . . . . 62
4.5 Rail track topology related to vanishing point position . . . . 70
5.1 Vanishing point algorithm parameters depending on the image environment 81 5.2 Accuracy comparison - Vanishing point position . . . . 82
5.3 Summary - ANOVA analysis of the accuracy . . . . 83
5.4 Elapsed time comparison - Vanishing point position . . . . 83
5.5 Summary - ANOVA analysis of the elapsed time . . . . 85
5.6 Accuracy comparison . . . . 86
5.7 Time efficiency comparison . . . . 86
5.8 topology algorithm parameters depending on the image environment . . . . 87
5.9 Measurement of accuracy - 3 topologies . . . . 87
5.10 Execution time - 3 topologies . . . . 87
5.11 Measurement of accuracy - 5 topologies . . . . 88
5.12 Execution time - 5 topologies . . . . 89
5.13 Contingency table - Detection of turnouts . . . . 90
5.14 Measurement of accuracy - 5 topologies + turnout detection . . . . 91
5.15 Execution time - 5 topologies + turnout detection . . . . 91
Acronyms / Abbreviations ANOVA Analysis of variance CPU Central Processing Unit GPU Graphics Processing Unit HIS Hue Intensity Saturation
HOG Histogram of Oriented Gradients HSV Hue Saturation Value
IDE Integrated Development Environment IPM Inverse Perspective Mapping
MLP MultiLayer Perceptron
PHT Probabilistic Hough Transform PID Proportional Integral Derivative RGB Red Blue Green
ROI Region Of Interests SD Standard Deviation SWaP Size, Weight and Power UAV Unmanned Aerial Vehicle VP Vanishing Point
Introduction
1.1 Context
The last decade has seen a significant evolution in the UAV industry, not only for military purposes but also with a strong development in industrial/commercial applications[1] and personal use1. Many industries who were traditionally not using airborne solutions, are now in the process of exploring the huge potential that UAVs can offer as far as transforming their products and services and getting a new competitive edge. The range of usability of drones is nowadays gargantuan, it is stretching from leisure use to advance machines built for inspections or supporting human workforce. This trend is mainly due to the improvement in sensor technologies, communication and software[2] as well as the reduction of production cost of vital elements enabling the growth of new application for consumers.
One of the most thriving applications is certainly the use of UAVs in maintenance opera- tions to reduce and speed up laborious human jobs. The benefit of this kind of application is such obvious that an uncountable amount of technologies is available on the market. For instance, one product is used to study the growth of crops to help farmers defining the right time to harvest2, another one is built to apply pesticides3. In a different domain, grid network companies analyse the attrition of power lines normally difficult to access4, which improves the range of action and the deployment time significantly.
There are a plethora of fields where UAVs bring symbolic improvements of productivity.
One case where the number of studies is limited is the railway domain, even though the
1"Commercial drones are the fastest-growing part of the market", The Economist, 2017, available on- line:https://goo.gl/uMRp3f
2sensFly, available online:https://www.sensefly.com/industry/agriculture/
3AGRASMG-1, available online:https://www.dji.com/mg-1
4Asset Done, available online:http://assetdrone.com
applicability of drones for maintenance seems to be large, as the number of maintenance tasks is extensive and the area to cover is broad.
Following this analysis, the thesis aims to focus on investigations related specifically to railways. In this context, the research is accomplished at Bombardier Transportation, which is the rail equipment division of the Canadian firm Bombardier Inc. More specifically, it is performed at Stockholm’s office, that is carrying out the development of rail control solutions, as well as the management of projects related to upgrading existing systems.
Bombardier Transportation has designed a technical solution aimed at automatically capturing the type and precise location of wayside objects placed along the railroad to generate up-to-date digital maps of the tracks. So-called wayside objects are all sorts of controlled or standalone devices used for signaling, powering, controlling, or operating the railways (e.g.: lamp signals, balises, wheel detectors, point machines, object controller cabinets, track circuits, axle counters, level crossing barriers, etc).
Bombardier’s solution is called GeoTrack. From a functional architecture viewpoint, GeoTrackconsists of several processes supported by different hardware and software modules, some of which are mounted onboard a locomotive in order to capture raw images, topological data and GPS coordinates from a train moving at a normal speed. The data is then processed off line and formatted into workable digital maps. The maps can be used for the basic design phase of signaling projects where the type, the amount and the location of wayside objects are determined prior to the installation or modification of a line, as well as for simulation and system integration tests prior to the commissioning of the line.
Previously the inventory of wayside objects was performed manually by means of inter- preting blue prints and records combined with field measurements performed by technicians walking along the track. Even though GeoTrack is a major improvement in itself, it has the disadvantage to be bound to equipped locomotives, which requires planning and authoriza- tions. The migration of the solution onto a UAV platform has the potential to offer tremendous benefits in terms of flexibility, cost, scalability and the ability to digitalize multiple tracks at the same time from the air.
The migration is made up of multiple challenges:
• Modification of the optical vision systems.
• Adaptation of the image recognition algorithm for different point of view (in higher altitude than with a locomotive).
• Research of methods for autonomous guidance of drone5to follow railway tracks autonomously.
However, as explained later in Section 1.2, the task is too large and the scope is redefined in the next section to be adapted for a single master thesis.
1.2 Problem statement
As aforementioned, this project involves several distinct technical challenges and tackling all of them in the scope of a single master thesis is inconceivable. Therefore, the project is separated in different tasks which are handled by multiple stakeholders such as Bombardier, partnered companies or by the author in the context of this thesis.
Whereas the different stakeholders are working on the migration of the Geotrack technol- ogy in terms of optical systems and image processing, as well as on the construction of a tailored UAV dedicated for this application, the thesis student investigates the autonomous guidance of UAVs on railways, in conjunction with the study of the challenges incurred while migrating from an existing system to an unmanned aerial vehicle platform. Bombardier, on its side, establishes the requirements for the project conjointly with the student in order to bring the right technology necessary for unmanned missions with UAVs in the rail industry.
The following paragraph describes the problem statement for each stakeholder. A sum- mary can be seen in Figure 1.1.
As stated before, the general goal of this project is to migrate the GeoTrack system to a UAV platform. The hardware and software for the sensor part of this solution was first developed by the German company Signon. The function is to capture images of the railway, in order to extract wayside objects automatically and create an accurate inventory and a digital map of the track. The system is composed of three cameras (Two classic ones and a high-speed one) as well as an accurate GPS system and a floodlight. The entire system is placed on the front of a locomotive, as depicted in Figure 1.2. The measurements are carried out while the train is running at normal speed.
Signon’s role in the migration is to explore how their sensors could be adapted for the new application. As shown in Figure 1.2, the system is quite heavy and bulky, therefore some major modifications must be carried out. The different challenges lie in the modification of the point of view as well as the reduction of stability of the footage that implies the optimization and weight reduction of the system to be portable by a UAV.
5For this entire report, UAV and drone are considered synonymous and not related in any kind to the military industry
6Image from SIGNON Deutschland GmbH
Figure 1.1 Repartition of the tasks for the different stakeholders
Figure 1.2 Data capture tools of the GeoTrack system mounted on a locomotive6
Concerning the UAV, Cybaero which is part of Drone Center Sweden, has the role of supplying video footages from UAVs as well as providing supervision to establish the requirements. The challenges reside in the fact that the system should be able to cover a large area at high speed and close to the ground, while carrying video equipments.
Finally, the interest of having a UAV is to be able to cover large areas quickly and autonomously. For that purpose, the UAV should be equipped with an autonomous guidance based on a system other than GPS, in order to gain in safety and robustness. Therefore, the master student is studying the problematic of autonomous guidance of the UAV above railway tracks by using computer vision algorithms. Autonomous guidance for unmanned aerial vehicle platforms includes multiple challenges:
• The recognition of the railway tracks must be feasible in any weather/light conditions and be robust enough to guarantee continuous guidance
• The system must work for different background environments and rails type
• The algorithm must be able to detect several rail tracks, as well as turnouts, curves, etc
• The tracking of the rails must be accurate enough to centre the drone on top of it at high speed
Moreover, as the development of UAV application is quite novel, it is arduous to have a great idea of the development process and the challenges that must be faced as few guidelines are available. Consequently, it is of great interest to look at the challenges associated to the migration of an existing system to an unmanned aerial system.
1.2.1 Purpose
The purpose of this thesis can be separated in two distinct parts. The first one is to study what are the challenges that can emerged when migrating an existing system made for ground use (placed on a ground vehicle) to an unmanned aerial system. The point is to increase the awareness, in order to plan such projects in a better manner from the start while tackling the real challenges. The UAV industry is soaring with new applications, therefore it is interesting to analyse what is common for every project in terms of challenges. The idea is not to provide a magic formula that is working for all the projects but more to give a larger view on the subject.
At the same time, the thesis is tackling the problem of detecting and tracking railways.
The purpose of this task is to be able to provide autonomous guidance of drones over railway
tracks. The interest of a such technology is to extend the range of usability of UAVs by enabling the system to be independent and to not have to rely on the supervision of a human.
In more details, the purpose of the algorithm is to primarily detect and extract the railway tracks from an image, and this in any light condition and multiple backgrounds. To track the movement their movements and provide a point of reference to the UAV control algorithm.
Moreover, the thesis aims also to compare different methods to achieve this task and conclude on the weaknesses and strengths of diverse algorithms.
1.2.2 Limitations
To avoid straying from the scope of this thesis, some limitations are specified, that are defined as less relevant for the purpose of the thesis in the allotted time:
• The flying algorithm including stabilization and control of the drone is not going to be studied.
• The obstacle avoidance for the drone is not an aspect developed in this thesis.
• The concerns linked to the computational power or energy consumption for embedded system is not included in this thesis.
• The questions related to regulation when it comes to flying with UAV out of eyesight will not be taken into consideration.
• Safety concerns are of main importance when developing a product for commercial use. However, this project is at the forefront of the development therefore this aspect will be disregarded for now.
1.2.3 Research questions
From the problems explained in the previous sections and the aspects that are going to be explored, three research questions can be written and separated into two groups. The first one is focusing on the challenges for migrating to UAV platform whereas the two others are looking at algorithms’ performance.
• What are the challenges and requirements specific to the migration of a system to UAV platform?
• What is the best technique for autonomous guidance of UAVs over railway tracks?
– What are the most optimal image recognition algorithms for detection of railway tracks, in terms of reliability, position accuracy and robustness (ability to detect rails in different conditions)?
– What are the most optimal tracking methods to enable the UAV to follow accu- rately railways?
– What are the best solutions in order to detect the topology of railroads and provide correct guidance to the UAV?
1.2.3.1 Outcomes
The outcomes of the project are defined here:
• A literature survey concerning migration concerns, image recognition and tracking algorithms.
• A map of technological challenges encountered when migrating to an existing solution architecture to a UAV platform.
• The development of an algorithm defining the trajectory in order to follow railways.
• A validation of the implemented system.
1.3 Ethical considerations
1.3.1 Academic concerns
When writing published documents such as academic papers, it is of paramount importance to take into consideration both plagiarism and responsibility for his own work. It is strenuous to ascertain plagiarism and by consequence it is one of the hardest ethical issue. Plagiarism is described by IEEE as:
“The use of someone else’s prior ideas, processes, results, or words without explicitly acknowledging the original author and source. [3]”
From this definition, it is evident that the separation between referencing, quoting, adopting, and copying [4] is difficult to assess. Moreover, plagiarism is not only in the form of literally copying text, but can be derived from translation or adaptation of a sentence to conform to the author writing style, increasing even more the complexity.
Self responsibility is also of prior importance when writing a thesis. “Ignorantia legis neminem excusat”7or in English: “ignorance of law excuses no one”, relates to the fact that it is the responsibility of the writer to be aware of the law and good practices. Acknowledging the sources is one of the main concern, it starts with giving credits to authors for their researches, ideas, concepts used as inspiration, etc. It is not only mandatory to accredit authors but it also gives credibility and authority to the article written.[5]
1.3.2 Implementation concerns
While developing the project, unethical situations may appear. The testing phase is often carried out while the system is in prototyping phase which means that the security cannot be guaranteed. It is then necessary to have all the equipment to prevent dangers as much as possible. It is also crucial to ensure that the people implicated are aware of the risks, hiding hazards would be unethical.
Another concern is Copyright, that is defined by the World Intellectual Property Organi- zation as:
“The legal right created by the law of a country that grants the creator of an original work exclusive rights for its use and distribution. [6]”.
Copyrighting has to be taken into consideration when using software. For instance, some libraries or free open source software may be licensed with different rights such as attribution, share-alike, non-commercial or non-derivative. Depending on the rights, certain constraints need to be considered and could limit the patentability, publication or marketing of the developed product.
1.3.3 Long term concerns
The project is about autonomously driven UAVs, several concerns may arise about the trustworthiness of such a system. When talking about drones in general, the first thought that comes to mind is military usage. It is then necessary to verify and ensure that the concept is not going to be adapted for unethical operations.
A drone mounted with cameras hoovering over someone’s private field and taking images without consent is violating the protection of the private life.
If there is no human driving the UAV, it means that a computer is taking its own decisions.
However, it may arise that certain situations are unsolvable in the allotted time and the UAV
7Definition by The law dictionary available online: https://thelawdictionary.org/
ignorantia-legis-non-excusat/
system shall take a decision that could jeopardize human life. In this context, if an accident happened, it would be really difficult to define responsibilities.
1.4 Outline
In Chapter 2, the research design is presented by first defining the research methodology applied for this thesis and then more precisely for the literature review. The approach for the case study is also explained.
In Chapter 3, the outcomes of the literature survey are exposed. By first over-viewing the different UAV applications and then focusing on the migration challenges from a ground system to an aerial system. Afterwards, the technical literature is reviewed and the relevant articles are presented in detail. The second part of this chapter is giving a background knowledge to the reader by describing and explaining the principal techniques used in the implementation. It ranges from computer vision algorithms to clustering methods.
The implementation is broken down in Chapter 4, starting by a extensive explanation of the approach, followed by a small section about the framework. Afterwards, every part of the implementation is described starting with the pre-processing operations. Then, two methods are presented to extract the Vanishing Point, the first one is using edge detection and line extraction while the latter is applying a specific kind of filters (Gabor). Subsequently, the clustering method applied in this report is described, followed by the tracking technique.
Finally, the methodology for the definition of the rail topology is depicted.
Chapter 5 presents the results obtained. While the first section compares the position accuracy of the vanishing point obtained with the two implemented methods, the second section is defining the veracity of the implemented method to define the rails topology.
A discussion about the results and feasible implementation is carried out in Chapter 6.
The report is concluded with Chapter 7 containing also some advices for future work.
Research design
2.1 Research methodology
The thesis comprises two distinct questions, the first one related to establishing challenges regarding the migration of an existing system onto a UAV platform. While the second is more technical and looks at the implementation of a vision processing algorithm in order for a UAV to track and to follow railroads. The questions are titled respectively: What are the challenges encountered when migrating to UAV platform? and What is the best method for autonomous guidance of UAVs over railway tracks?
The methodology used to answer the first question starts with a literature review to assess the existing methodologies tackling migrations and what it includes in terms of constraints and requirements. After this overview, a survey is carried out with multiple experts in the UAV industry and as much feedback as possible is collected.
The first question is studied in a qualitative manner. And the completeness and correctness are evaluated. This assessment is performed by the drone manufacturers and Bombardier to verify that all the stakeholders are on common ground.
Apropos of the two remaining questions, they deal with more technical subjects and it is more relevant to carry out a quantitative approach for the research. As previously, a literature survey is achieved by exploring existing articles, books and patents tackling the problem of line detection and more generally object tracking. This study provides a better understanding of the range of algorithms usable for this task.
The first step is to study more carefully the different algorithms to define their applicability in the project. The most relevant are implemented software-wise and the performances are evaluated on template images of simple railways. This allows to have a benchmark to compare with the literature. The most proficient ones are then implemented on complex images of railway tracks and fine-tuned to achieve the highest accuracy. Finally, the algorithm
fulfilling the requirements in terms of positioning accuracy and robustness is used in the final implementation.
The performance can be evaluated in a quantitative way, several indicators can be mea- sured: the accuracy of the detection and tracking in terms of absolute and relative positioning, the robustness of the algorithms in terms of percentage of misclassification, and this for different background conditions and noises, as well as the rapidity of execution.
This iterative process has the advantage of covering the study of multiple algorithms. It enables also to understand the functioning gradually. And finally, it ensures that the final implementation is based on optimal algorithms and parameters.
2.1.1 Literature research strategy
This section defines the strategies employed to carry out the literature survey. The objectives of such a research is to establish a review of the existing articles, books, studies, etc. on a specific subject and highlight the major findings, contradictions and conclusions as well as certify that the research does not duplicate already existing studies.
At the beginning of a research, it is essential to have an overview of the subject and to gain a better understanding of the context. In fact, it is often the case that the first idea is erroneous due to limited knowledge on the subject. Therefore, it has been decided for this literature research to explore first the resources in a broad manner, before narrowing it down to specific technical fields.
The databases used as resources are detailed below:
• Science Direct [7]
• IEEE Xplore Digital Library [8]
• Springer[9]
• digital-library [10]
The keywords utilised for the research are broken down in Table 2.1.
2.1.1.1 Challenges encountered when migrating to UAV platform
Concerning the literature exploration about the different challenges encountered when mi- grating a system to a UAV platform, the initial articles’ investigation was made using specific keywords: ["Migration", "Challenges", "Strategies", "UAV","Drone"]. The range of scientific articles about the subject of challenges with drones is large but for most of them are not associated to the migration but mainly to the restrictions related to UAVs. Less than a dozen
Table 2.1 Keywords summary Migration
Challenges Strategies UAV Drone
Trajectory Planning Path Planning Terrain Following UAV
Drone
Railway Maintenance Inspection Tracks UAV Drone
Tracking
Visual Servoing UAV
Drone
Vanishing Point Feature Tracking Line Tracking
of papers were selected as relevant, which was not enough to have a clear picture. Therefore, the research was extended to scientific websites. Multiple articles tackling the migration problem were found. Finally, a survey was carried out with experts to obtain wise feedback.
2.1.1.2 Autonomous guidance of UAV over railway tracks
The starting point of the research concerning the technical question of autonomous guidance was about trajectory planning and path planning as it seemed to be the normal way to solve the problem. Nonetheless, the research was narrowed down to UAV application. The keywords used for the first exploration were ["Trajectory planning", " Path planning", "UAV", "Drone",
"Terrain following"]. And as for the rest of the research, it was carried out using "google scholar" and mainly two databases: "ScienceDirect" and "IEEE". For these first keywords, a plethora of articles matched. A quick review was carried out reading the abstracts to extract the most relevant articles for the subject of this thesis. Approximately 10 articles have been studied more carefully and the conclusion was that the keywords were not in phase with the goal of this thesis and did not provide existing methodology correlated with the research questions.
Therefore, a new set of keywords were defined. This time the approach was different and focusing on the maintenance application for railways to get an understanding of the needs and use cases of UAVs in the railway industry as well as exploring already existing methods developed to tackle the challenges of hovering on railway tracks . The keywords were this time ["Railway", "Maintenance", "Inspection", "Tracks", "UAV", "Drone"]. Unfortunately, few results came out from this research, therefore it was extended to websites. Multiples example were found for drone application in maintenance, but none of them was tackling the autonomous guidance over railways. After deeper research, one article appeared to be a perfect match: [Railway track following with the AR. Drone using vanishing point detection].
After reading this article, it was concluded that the first assumptions were biased and that the solution to resolve the problem were not in trajectory or path planning but more in tracking algorithms or visual servoing.
From these unexpected inputs, a new set of studies was carried out with the following keywords ["Tracking", "Visual servoing", "UAV", "Drone"]. This set gave a lot of results tackling the task of tracking for multiple sorts of applications. It has been decided that only three types of applications were relevant for the research as similar to the problem: Pipeline, Powerline and Railways. This reduces the results to fifteen relevant articles.
The study of these articles highlights common methodologies and computer vision algorithms used for tackling such tasks. The next step was to explore new articles using the same methodologies for different application. The investigation was made using predominant keywords ["Vanishing Point", "Feature tracking", "Line tracking"]. The outcomes of this research were beneficial and gave a large set of articles to work on.
The next step was then to study in more details the specific algorithms developed in the articles to understand them and be able to use them on the problem of this thesis.
2.1.2 Case study approach
This part of the report is describing the different case studies that have been carried out during the experimentation phase and presenting the process with the failures and the successes that leaded the final result. As previously, the section is separated in two parts for the two different research questions.
2.1.2.1 Challenges encountered when migrating to UAV platform
Concerning this question, the case study is straightforward. After gathering information about the different challenges from the literature review, a discussion is carried out with different experts from the field. As it is difficult to get into contact with numerous people for a direct talk, a survey is established that is then sent to a massive group of people related to UAV industry. The survey is focused on obtaining general thoughts and ideas about migration and not assessing the reliability of what is known. The next step is to combine the different sources of information to develop a map of the challenges while migrating to a UAV platform.
2.1.2.2 Autonomous guidance of UAV over railway tracks
The case study about autonomous guidance starts by analysing different computer vision algorithms existing to define how they work and their influences on an image. In order to have a good grasp on the subject and be able to apply the different filters on multiple images, a "tester" was developed. It enables to apply easily different kind of filters (edge detection, blurring, etc) onto an image that is uploaded. The parameters of each filter can be modified in real time and individually with sliding buttons. The objective behind the development
of this program is to gain a thorough understanding of each filter and see the real impacts on the images as well as choosing the best combination of parameters to obtain the highest efficiency.
Once the knowledge about the filters reached a sufficient level, the second step of the development was to implement different methodologies that were presented in articles tackling the same problem, namely railway detection and tracking. The algorithms from different papers have been implemented successfully and in order to reach a better robustness and accuracy, algorithms from different sources were implemented together.
This implementation gave some interesting results, however, it is based on proven techniques. It is thus also interesting to implement analogues methods never implemented in the field of research of this thesis. For the sake of research and progress, it has been decided to implement Gabor filters to detect railway tracks. These filters have been used for roads or detection of defects on rails but never (as far as the author knows) for rail tracks detection.
As previously, a tester has been developed to have a better understanding of the influence of the multiple parameters. Then, an implementation has been done from an inspiration of a use case for unstructured roads. It did not prove to work totally for this specific application and therefore deeper development were necessary to adapt the methodology to railroads.
After having proven that the two filters implemented (line detection and Gabor filter) gave good results in extracting railways, the second step was to develop the tracking algorithm that enables the UAV to follow the railway tracks. For this purpose, the literature was reviewed and the predominant technique that had the best chance to work is chosen for implementation.
Once implemented, the algorithm is tested on different data to assess the reliability and robustness.
The rails are made of several topologies, there can be straight lines, curves, turnouts all this with different density of rails. To have a system fully autonomous, it needs to cope with situations where there multiple paths are available. In order to deal with this, an algorithm has been developed to define the rail typology between three categories: straight lines, curves and turnouts. The tests are carried out by defining the accuracy of classification.
Background
3.1 Literature research
In this chapter, a literature survey of the existing research is presented. It starts with an overview of UAV applications for maintenance in different fields. Then, it continues with a review of research related to migration challenges for UAVs. And it finishes with an extended study of the existing algorithms for tracking, covering different approaches and use cases.
An overview of the review is depicted in Table 3.1, it separates the articles depending on the subject treated between "Migration challenges" and "Autonomous guidance". Under
"Inspection" are the articles presenting inspection or maintenance operations. On the other hand, the column "Tracking" is resourcing the articles describing methods for autonomous tracking/guidance. Finally, the "Algorithms" row shows different papers related to specific techniques for image filtering, tracking, clustering, etc.
Table 3.1 Summary of the different papers analysed Migration challenges
[1], [2], [11], [12], [13], [14]
Autonomous guidance
Application Inspection Tracking
Railways [15], [16], [17], [18] [19], [20]
Power lines [21], [22] [23], [24]
Pipelines - [25], [26]
Roads - [27], [28], [29], [30], [31]
Others [32] -
Algorithms
[33], [34], [35], [36], [37], [38], [39], [40], [41]
3.1.1 UAV applications overview
Following an outdoor object in order to provide surveillance, maintenance or measurements is a common task for UAVs. An overview of the different types of applications are presented in this section. For instance, [32] explores the possibility to use UAV for inspection of solar farms to identify defects. [21] and [22] demonstrate that it can be used for power line inspection in order to analyse the attrition or defaults. [23] presents a method for autonomous inspection of electric towers with UAVs, using only computer vision and machine learning.
A very common application for UAVs related to object following is the inspection of pipelines as explored in [25] and [26]. An analogy can be made between infrastructure tracking and road following, this latter has been studied deeply by multiple scientists as it is a necessary feature for autonomous driving of cars [[27], [28], [29], [30]] . The previous research, even if not directly related to railways, is used as inspiration for further investigation and several applied methods can be adapted to be used on railway tracks.
When it comes to railway applications particularly, the number of research papers is very limited. The main field explored is infrastructure surveillance to monitor civil structures such as viaducts, bridge, etc., as discussed in [15]. [16] presents a railroad detection algorithm tailored for high altitude drones. [17] surveys a number of vision and control methods applicable on UAVs and depicts two use-case scenarios for railway inspections, the first defines an approach using feature extractions for detection of target object before inspection, such as light signals or others. The latter concerns the tracking of railroads in order to follow track lines. [19] tackles the same problem using, as previously, vanishing point detection technique.
As shown, the number of applications is large for UAVs in the domain of maintenance and surveillance this section aims to give an overview of the different fields and applications.
However, the papers cited previously are only a subset of all the articles that have been published about this subject.
3.1.2 Migration challenges
Concerning UAV technologies, the development of new applications is often related to the migration of an existing ground-based system to a UAV platform. The challenges encountered during this process have been investigated by several researchers. [2] is looking at this problem from a project management point of view. They establish a list of different challenges broken down into two main fields: project resources management and technical challenges. The principal restrictions concerning the resource management are resource allocation, provisioning and adaptation. The resource allocation challenges come from the
trade-off between the efficiency and cost versus the functionalities of the UAV. For the provisioning concern, it is related to the allocation to different applications of the power (computational, energy, etc...). About the adaptation, it involves the adjustment of the resources depending on the application. Concerning the technical challenges, they are more related to communication, security, integration, energy and autonomy. The authors state that communication and safety issues are tightly related on different aspects, such as the communication loss, interruption or latency that could result in a mis-transmission of vital information. The integration is also a difficult task as UAVs are composed of multiple separate systems that have to work all together. The energy consumption matter is maybe one of the most known concern about UAVs as it defines the range of applications and usability and therefore is a major bottleneck. As discussed in [42], this problem can be tackled from different sides, either by reducing the power consumption of the system, with the drawback of certainly reducing the performances, by increasing the energy density of the batteries in order to decrease the payload, or by adding an ancillary power system (Fuel cells, Photovoltaic, etc).
Different other aspects related to technological challenges are stated in [11]. They establish a list of domains including as previously mentioned communication, energy and integration as well as several other concerns: sensing, signal processing, data and information fusion, collision, propulsion, etc. They highlight also the development of autonomous UAV systems, which includes different challenges related to reliability and supervision.
Other articles [1, 12] are focusing more on the problem of communication between the UAV and the ground station. In fact, if the communication is unstable it leads to multiple problems and possibly to a crash of the drone. However, as pointed out by [11], the increase in autonomy reduces the need for a robust communication, as it can guide itself, the hazard of a communication loss is decreased. The principal communication challenges are frequency allocation, size, weight and power (SWaP) and security, as stated in [1]. Indeed, the frequency allotted for specific applications is strictly regulated and the communication distance is tightly related to frequency. About SWaP, the size and power consumption of the antenna is directly related to the travelled distance. For high power antennas, the size is so large that it needs to be integrated into the fuselage which reduces the aerodynamic. The last point stated concerns the communication security, another article [12] is exploring the hijacking of drones, which is a main concern for safety and reliability.
One of the aspect that has been raised by different articles [13, 14] is the challenges related to the regulation. In fact, as written in [14], "A lack of comprehensive government regulations necessary to safely facilitate the accelerate integration of UAVs into the national airspace system" is a primary safety concern in addition to the scarcity of technological
standards. The root cause described is the rapid expansion of UAV applications that did not let time to the administration to keep up.
It can be seen that the concerns about UAV migration have been inventoried, studied and analysed thoughtfully and relate to the fact that it is a current and real problem.
3.1.3 Autonomous guidance
The problem of autonomous guidance has been explored in a plethora of research papers, with applications ranging from autonomous driving to surveillance. It is concluded after the analysis of several articles that feature tracking is the most used technique for autonomous guidance. This section summarizes the different algorithms explored to tackle the problem of tracking applied in infrastructure inspections.
Several steps are necessary in order to track an object, the main ones are detecting features on an image and then defines the movement of these features over continuous frames (video).
Appendix A depicts an overview of techniques that can be used for tracking algorithms. The panel of algorithms can be sundered in two main classes, the first one uses edge detection and line extraction before applying a tracking algorithm (3.1.3.1, 3.1.3.2) whereas the latter uses global features extraction to describes the scenery (3.1.3.3, 3.1.3.4).
The section starts with an analysis of papers tackling tracking problems using vanishing point detections, three papers are described thoroughly. Then, another technique using lines to track pipelines, power lines or roads is described. An article focusing on the detection of railways is also studied. Finally, a technique using feature-based tracking is presented.
3.1.3.1 Vanishing point detection
Vanishing point detection is based on the physical principal that straight lines parallel in 3D space are converging in 2D to a single point called vanishing point [20] (See Figure 3.1).
Moreover, several vanishing points (VPs) arising from an object on a common 3D plane form a Horizon line. This property can be used to define the movement of an image by tracking the displacement of VPs as they are constant parameters of the environment. This method has been explored by many researchers ([27], [19], [28]). The most relevant literature is detailed in this section.
Railway track following with the AR Drone using vanishing point detection
[19] is presenting a method for railway track following using the principle of VP tracking.
It extracts the vanishing point of the railways and then uses a Kalman filter[33] for tracking and finally applies a PD controller to keep the drone centred on the rails. The entire process is depicted in Figure 3.2 and explained in detail afterwards.
