Boundary Representation
Modeling from Point Clouds
OSKAR ARONSSON
JULIA NYMAN
KTH ROYAL INSTITUTE OF TECHNOLOGY
Examiner: Raid Karoumi
School of Architecture and the Built Environment Host company: Tyréns
Abstract
Inspections of bridges are today performed ocularly by an inspector at arm’s lengths distance to evaluate damages and to assess its current condition. Ocular inspections often require specialized equipment to aid the inspector to reach all parts of the bridge. The current state of practice for bridge inspection is therefore considered to be time-consuming, costly, and a safety hazard for the inspector.
The purpose of this thesis has been to develop a method for automated modeling of bridges from point cloud data. Point clouds that have been created through pho-togrammetry from a collection of images acquired with an Unmanned Aerial Vehicle (UAV). This thesis has been an attempt to contribute to the long-term goal of making bridge inspections more efficient by using UAV technology.
Several methods for the identification of structural components in point clouds have been evaluated. Based on this, a method has been developed to identify planar sur-faces using the model-fitting method Random Sample Consensus (RANSAC). The developed method consists of a set of algorithms written in the programming lan-guage Python. The method utilizes intersection points between planes as well as the k-Nearest-Neighbor (k-NN) concept to identify the vertices of the structural ele-ments.
The method has been tested both for simulated point cloud data as well as for real bridges, where the images were acquired with a UAV. The results from the simulated point clouds showed that the vertices were modeled with a mean deviation of 0.13 − 0.34 mm compared to the true vertex coordinates. For a point cloud of a rectangular column, the algorithms identified all relevant surfaces and were able to reconstruct it with a deviation of less than 2 % for the width and length. The method was also tested on two point clouds of real bridges. The algorithms were able to identify many of the relevant surfaces, but the complexity of the geometries resulted in inadequately reconstructed models.
Keywords
Point Cloud, Photogrammetry, RANSAC, Modeling, Bridge Inspection, Boundary Representation
Sammanfattning
Besiktning av broar utförs i dagsläget okulärt av en inspektör som på en armlängds avstånd bedömer skadetillståndet. Okulär besiktning kräver därmed ofta speciell ut-rustning för att inspektören ska kunna nå samtliga delar av bron. Detta resulterar i att det nuvarande tillvägagångssättet för brobesiktning beaktas som tidkrävande, kostsamt samt riskfyllt för inspektören.
Syftet med denna uppsats var att utveckla en metod för att modellera broar på ett automatiserat sätt utifrån punktmolnsdata. Punktmolnen skapades genom fotogram-metri, utifrån en samling bilder tagna med en drönare. Uppsatsen har varit en insats för att bidra till det långsiktiga målet att effektivisera brobesiktning genom drönar-teknik.
Flera metoder för att identifiera konstruktionselement i punktmoln har undersökts. Baserat på detta har en metod utvecklats som identifierar plana ytor med regressions-metoden Random Sample Consensus (RANSAC). Den utvecklade regressions-metoden består av en samling algoritmer skrivna i programmeringsspråket Python. Metoden grundar sig i att beräkna skärningspunkter mellan plan samt använder konceptet k-Nearest-Neighbor (k-NN) för att identifiera konstruktionselementens hörnpunkter.
Metoden har testats på både simulerade punktmolnsdata och på punktmoln av fy-siska broar, där bildinsamling har skett med hjälp av en drönare. Resultatet från de simulerade punktmolnen visade att hörnpunkterna kunde identifieras med en me-delavvikelse på 0, 13 − 0, 34 mm jämfört med de faktiska hörnpunkterna. För ett punktmoln av en rektangulär pelare lyckades algoritmerna identifiera alla relevanta ytor och skapa en rekonstruerad modell med en avvikelse på mindre än 2 % med av-seende på dess bredd och längd. Metoden testades även på två punktmoln av riktiga broar. Algoritmerna lyckades identifiera många av de relevanta ytorna, men geo-metriernas komplexitet resulterade i bristfälligt rekonstruerade modeller.
Nyckelord
Preface
This master thesis project was written in collaboration with Tyréns AB as the last step for our master’s degree in Civil and Architectural Engineering at The Royal Institute of Technology (KTH) in Stockholm, Sweden.
The topic for the thesis was suggested by Tyréns, as preparatory work for a research and development project lead by Mahir Ülker-Kaustell and Chunling Shan.
We would like to sincerely thank our supervisor Mahir Ülker-Kaustell for his contin-uous support and assistance through invaluable insights and discussions. His interest and dedication towards the project have been greatly encouraging and motivating for us. We would also like to thank Juan Avendaño for helping us with the UAV image acquisition.
Stockholm, June 2020
Acronyms
2D Two Dimensional 3D Three Dimensional
ASCII American Standard Code for Information Interchange BaTMan Bridge and Tunnel Management
BIM Building Information Modeling BREP Boundary Representation CAD Computer Aided Design Deg Degree of the Vertex DSLR Digital Single-Lens Reflex GPS Global Positioning System k-NN k-Nearest Neighbor
KD-Tree K-Dimensional Tree
LiDAR Light Detection and Ranging NNS Nearest Neighbor Search PC Point Cloud
PCD Point Cloud Data PLY Polygon File Format
PPE Personal Protective Equipment RANSAC Random Sample Consensus RGB Red-Green-Blue
RGBA Red-Green-Blue-Alpha RMSE Root Mean Square Error
SfM Structure from Motion SHM Structural Health Monitoring TLS Terrestrial Laser Scanning TOF Time of Flight
UAV Unmanned Aerial Vehicle
Symbols
A Matrix of Planar Coefficients n Normal Vector
p Point in n-Dimensional Space q Point in n-Dimensional Space r Parametric Line
t Directional Vector of a Line x0 Point in 3D Space
∩ Statistical Intersection
R3 Real Coordinate Space of Dimension 3 Rn Real Coordinate Space of Dimension n µ Mean Value x Mean Value σ Standard Deviation a, b, c, d Plane Coefficients d Euclidean Distance [m] E Edge G Graph I Intersection
M d-Dimensional Vector Space n Number of Neighbors
P Plane
p Orthogonal Distance to Origin q Query Point
r Radius [m] S Set of Points
s Scalar
sx Sample Standard Deviation V Vertex
1 Introduction 1
1.1 Background . . . 3
1.2 Aim and Scope . . . 5
2 Literature 7 2.1 Generating Point Clouds . . . 7
2.1.1 Laser Scanning . . . 8
2.1.2 Photogrammetry . . . 8
2.2 Data Acquisition with UAV . . . 9
2.3 Segmentation . . . 11
2.4 Principles for Model Reconstruction . . . 12
2.4.1 Geometrical Shape Descriptors . . . 12
2.4.2 Hard-Coded Knowledge . . . 12
2.4.3 Machine Learning Approaches . . . 13
2.4.4 BIM Alignment Approaches . . . 13
2.5 Structural Health Monitoring . . . 13
2.5.1 UAVs in Structural Health Monitoring . . . 14
3 Theoretical Background 15 3.1 Photogrammetry . . . 15
3.1.1 Structure from Motion . . . 16
3.1.2 Digital Photography . . . 17
3.2 Point Cloud Data . . . 18
3.2.1 Point Cloud Defects . . . 19
3.3 PCD Processing Techniques . . . 21
3.3.1 Voxelization . . . 21
3.3.2 Noise and Outlier Filtration . . . 22
3.4 RANSAC . . . 23
3.5 Nearest Neighbor Search . . . 24
3.6 K-Dimensional Tree . . . 24
3.7 Spatial Geometry . . . 25
3.8 Boundary Representation and Graph Theory . . . 27
4 Method 31 4.1 Python . . . 32
4.1.1 Open3D . . . 32
4.1.2 NumPy . . . 32
4.2 Rhinoceros . . . 32
4.3 Simulation of Test Data . . . 33
4.4 Photogrammetry . . . 33
4.4.1 Data Acquisition . . . 34
4.4.2 Pix4D . . . 35
4.5 Point Cloud Pre-Processing . . . 36
4.5.1 Downsampling of PCD . . . 36 4.5.2 Outlier Removal . . . 37 4.6 Identifying Vertices . . . 37 4.6.1 Planar Identification . . . 38 4.6.2 Intersection of Planes . . . 39 4.7 Identifying Edges . . . 41 4.8 Modeling of Edges . . . 42 4.9 Evaluation of Results . . . 43 5 Results 45 5.1 Simulated Data . . . 45 5.1.1 Simulated Structures . . . 45
5.1.2 Results from Simulated Data . . . 47
5.2 Photogrammetrical Data . . . 51
5.2.1 Photogrammetrical Models . . . 51
5.2.2 Results from Generated Point Clouds . . . 57
6 Discussion 61 6.1 Further Research . . . 63
Bibliography 65
A Simulated Point Clouds 73
A.1 Algorithm settings . . . 73
A.2 Deviation of Vertices . . . 76
B Real Point Clouds 79 B.1 Column . . . 79
B.1.1 Measurements . . . 79
B.1.2 Calculations . . . 80
B.2 Portal Frame Bridge . . . 84
1.1 Bridge inspection vehicle [8]. . . 3
1.2 Stages of a bridges’ information model throughout its lifetime. . . . 5
3.1 Structure from motion principle [56]. . . 16
3.2 Working mechanism of cameras. . . 17
3.3 Example of PLY-file. . . 19
3.4 PCD defects. . . 21
3.5 Voxel downsampling. . . 22
3.6 Outlier filtration. . . 23
3.7 Model fitting to a data set with outliers. . . 24
3.8 Data structure of a KD-tree. . . 25
3.9 Edge connecting the vertices A and B. . . 28
3.10 Non-directed graph. . . 28
3.11 Connectivity of graphs. . . 29
4.1 Flowchart of the overall process. . . 31
4.2 Flowchart of the test data simulation. . . 33
4.3 Flowchart of the photogrammetric process. . . 34
4.4 Flight trajectory pattern. . . 35
4.5 Camera positions during image acquisition of a column. . . 35
4.6 Rectangular block with identified planes and plane indices. . . 42
5.1 Structure A. . . 46
5.2 Structure B. . . 46
5.3 Structure C. . . 47
5.4 Results from the stages of calculation for Structure A. . . 47
5.5 Results from the stages of calculation for Structure B. . . 49
5.6 Results from the stages of calculation for Structure C. . . 50
5.7 Point cloud generated of a column at Uppsalavägen. . . 52
5.8 Filtration techniques applied to the column, red indicates outliers. . 53
5.9 Point cloud generated from photographs of a portal frame bridge. . . 54
5.10 Statistical filtering applied to the portal frame bridge, red indicates outliers. . . 55
5.11 Point cloud generated from photographs of the beam bridge. . . 56
5.12 Statistical filtering applied to the beam bridge, red indicates outliers. 57 5.13 BREP model created from the point cloud of the column at Upp-salavägen. . . 58
5.14 Attempt to create a model from the point cloud of the portal frame bridge. . . 59
5.15 Attempt to create a model from the point cloud of the beam bridge. . 59
B.1 Measurements from the point cloud. Green lines indicates the dis-tances measured. . . 79
B.2 Measurements of the column based on the point cloud. . . 80
B.3 Planes identified for the column. . . 81
B.4 Planes identified for the portal frame bridge. . . 85
B.5 Intersections after Alg. 3 for the portal frame bridge. . . 86
B.6 Calculated vertices for the portal frame bridge. . . 86
B.7 Calculated edges for the portal frame bridge. . . 87
B.8 Filtration of the beam bridge. Red indicate outliers. . . 88
B.9 The beam bridge point cloud after the filtration. . . 88
B.10 Planes identified for the beam bridge. . . 90
B.11 Intersections after Alg. 3 for the beam bridge. . . 91
B.12 Calculated vertices for the beam bridge. . . 92
2.1 Steps for planning the data acquisition [21]. . . 10
3.1 Properties of different PCD file formats. . . 18
4.1 Equipment used for image acquisition [83]. . . 34
4.2 Number of points generated in relation to image scale and point den-sity [84]. . . 36
4.3 Pre-processing functions applied from Open3D 0.9.0 [61] . . . 37
5.1 Summary of the results for Structure A. . . 48
5.2 Summary of the results for Structure B. . . 49
5.3 Summary of the results for Structure C. . . 51
5.4 Settings used in Pix4d. . . 51
5.5 Summary of the results obtained for the column PC. . . 53
5.6 Summary of the results obtained for the portal frame bridge PC. . . 55
5.7 Summary of the results obtained for the beam bridge PC. . . 57
5.8 Comparison between the measurements of the model and point cloud. 58 A.1 Algorithm settings for Structure A. . . 74
A.2 Algorithm settings for Structure B and C. . . 75
A.3 Deviation of vertices for Structure A. . . 76
A.4 Deviation of vertices for Structure B. . . 77
A.5 Deviation of vertices for Structure C. . . 78
B.1 Measurements from the point cloud of the column. . . 80
B.2 Planes identified for the column. . . 81
B.3 Algorithm settings for the column. . . 82
B.4 Measurements from the derived model. . . 83
B.5 Deviations for the column, between model and point cloud. . . 83
B.6 Deviations for the column, considering deviations in measurements
from the PC. . . 83
B.7 Algorithm settings for the portal frame bridge. . . 84
B.8 Planes identified for the portal frame bridge. . . 86
B.9 Results for the portal frame bridge. . . 87
B.10 Algorithm settings for the beam bridge. . . 89
B.11 Planes identified for the beam bridge. . . 91
Introduction
Bridges often provide an essential link in the transportation system, by enabling transportation over depressions or obstructions, it is therefore of the highest impor-tance that they are kept in a condition ensuring that the bridge can remain operational and safe. Bridges are generally large structures and a collapse would undoubtedly risk injuries or even causalities. A bridge that no longer can remain operational can create a severe bottleneck in the transportation system. The Swedish Transport Administration manages nearly 21000 bridges [1], where 79 % are road bridges, 19 % are railway bridges and the remaining are other types of bridges. To ensure the structural safety and serviceability of these bridges, the Swedish Transport Ad-ministration requires bridges in Sweden to be routinely inspected every sixth year following their guidelines [2].
The deterioration of load-bearing structures is affected by many variables. Some of these variables being, time, load capacity, load frequency, and environment [3]. The inherent uncertainties in the variables affect the assessment at which rate the bridge is deteriorating. Visual inspections are essential to determine the deterioration patterns and mechanisms, and therefore a key action for the overall assessment whether the bridge is deteriorating at the expected rate. Assessing the current state of the bridge is fundamental for the management of the structure, planning of maintenance, and remediation work.
Visual inspections of bridges are traditionally performed ocularly at close range by an inspector. This type of traditional inspection is often considered to be expensive concerning the time consumption, operation equipment needed, safety measures to be taken, and the obstruction of traffic [4]. Furthermore, the bridge inspector who
works within proximity of traffic as well as at potentially fatal falling heights is exposed to a safety hazard. Considering these intrinsic problems with traditional inspections, further development of the inspection procedures can increase its effi-ciency both concerning time and cost, as well as to reduce the risk the inspector is exposed to.
Advances in technology have allowed for new techniques of visual inspection. Un-manned Aerial Vehicles (UAVs) or, drones, can maneuver around the bridge reach-ing locations where the inspector in a traditional ocular inspection would struggle to reach. UAVs allow for inspections without obstructing traffic as well as eliminating the requirement of employing equipment such using under-bridge inspection units (see Figure 1.1). A UAV equipped with a high-resolution camera can fly within close vicinity of the bridge, taking pictures of the structural elements with a high level of detail. From the collection of photographs, methods of photogrammetry can be used to obtain a 3D model of the bridge, both as a point cloud or as a mesh. The images can be further analyzed to identify and measure cracks, corrosion, and chemical damage to the structure. Through the utilization of machine learning tech-niques, the damage detection could be performed in an automated manner, greatly increasing the efficiency of the inspection process [5].
An efficient bridge management system relies on the identified damages to be stored and documented in a structured manner, to enable inspection-data from consecutive inspections to be used to track the damage propagation, allowing for assessment of the deterioration rate. A key for creating a more efficient inspection procedure is to identify structural members from point clouds. Structural members that are to be stored in the management model for the bridge and creates the foundation for which the inspection-data can be managed and analyzed. Identified damages and their locations can be mapped within the management model to the locations where they were found, so the propagation of damages efficiently can be compared with data from previous inspections. This sort of mapping requires the structural members to be modeled efficiently based on the point cloud data. Modeling of an object in terms of its limits and shape is known as boundary representation (BREP) modeling [6].
An efficient inspection technique in combination with a management system that allows for reliable data management and assessment, can over time potentially save money and resources by providing more knowledge about the deterioration rate and damage propagation.
1.1
Background
According to Trafikverket [7], a main bridge inspection is to be conducted at least every sixth year, to ensure the structural safety of the bridge. The first main inspec-tion is conducted before the bridge is taken into service. These inspecinspec-tions are per-formed according to guidelines in the Swedish management system BaTMan (Bridge and Tunnel Management). In which it is stated that during the main inspection, all structural elements, with only a few exceptions, shall be inspected at an arm-lengths distance. This suggests that the damages on the bridge are to be detected through oc-ular inspection. The damages are then measured using a set of standardized methods stated in the BaTMan handbook, depending on the type of structural member and type of damage. For cracks, this can include measuring the maximum crack width, length, as well as recording its placement.
Inspecting at an arm-lengths distance can require many safety precautions. Namely, personal protective equipment (PPE) against falling from heights and wear to in-crease the visibility of the personal as well as closing trafficked lanes on the bridge, all to ensure the safety of the personal. Inspecting larger bridges may further increase the need for safety precautions and equipment, special bridge inspection vehicles, as seen in Figure 1.1, may need to be employed, professional climbers equipped with PPE or other specialized measures. Understandably, these measures have a high op-erational cost at the same time as the cost of disturbing the transportation system can not be neglected.
Bridges are structures with a long expected technical lifetime of up to 120 years [9]. The need for robust and reliable documentation from inspections and maintenance is therefore essential. Since 2004, the documentation from the majority of the Swedish bridge inspections is organized in the BaTMan database provided by Trafikverket. This is done using the organizing tools that are incorporated into the system [10]. BaTMan contains over 30000 bridges and it is used both for adding new information from inspections as well as for supervision and planning for future measures.
Standardization of the documentation is important to be able to assess the status of the bridge over time. Although, documentation from traditional bridge inspections is mainly done manually and often consists of sketches, photos, notes, and voice recordings [11]. The documentation should be continuous and explicit, nevertheless, records are often incomplete and not extensive enough for interpretation of others. This results in difficulties in the assessment of degradation and deterioration rates. This type of assessment requires thorough documentation from previous inspections for comparison and evaluation for damages, such as cracks and their propagation. It is, therefore, a necessity that information is transferred properly throughout time to maintain the bridge stock throughout their expected lifetime.
For bridges that are reaching the end of their expected lifetime, structural drawings created decades ago may not be in good condition or not resemble a complete set for the structure. Alternative and efficient techniques of measuring structural elements are therefore necessary.
Within recent years, building information modeling (BIM) has been recognized as a good practice for structural management, throughout the lifetime of the structure, and is therefore often requested by clients for new structures [12]. A model rep-resenting the actual implemented structural choices is known as an as-built model [13]. For cases where drawings of the structure are not complete or retrofit and rebuild work is not documented properly, creating an as-built model retroactively can be difficult. Under these circumstances, as-built models can be created from point clouds acquired from photogrammetry or laser scans. The work of creating the model from the point cloud is a manual and time-consuming endeavor [14]. The current workflow for this procedure is that the modeler traces the point cloud to form the geometries, using the point cloud as a mold [15]. This known as scan-to-BIM and is recognized as being time-consuming, subjective, tedious, and requires a skilled modeler [13]. Lastly, models that are enriched with information of damages are re-ferred to as as-damaged models. However, as-damaged models are not commonly created nor up to date [16].
As-designed model As-built model
3D Reconstruction
As-damaged model
Damage Detection
Bridge
Figure 1.2: Stages of a bridges’ information model throughout its lifetime.
Transferring fragmented information throughout the lifetime of the bridge is asso-ciated with increasing costs and time consumed for validation of information [17]. As-damaged BIM models updated with information from multiple inspections allow for the possibility to thoroughly track the changes of damages over time. Managing documentation from bridge inspections in collaboration with a BIM model can make the validation of information more apparent.
1.2
Aim and Scope
This master thesis aims to initiate the work of a platform for identification and au-tomated modeling of structural components from point cloud data acquired from photogrammetry. As an effort towards the long term goal of creating a reliable and cost-effective bridge management system, where inspection-data can be managed, updated, and stored throughout the lifetime of bridges.
The scope of this thesis has been to:
• Develop a method for identification and BREP modeling of structural compo-nents using Python programming.
• Evaluate the reliability and accuracy of the method through testing on syn-thetic and real point clouds.
Literature
A literature search is presented in this chapter focused on providing relevant back-ground information for the thesis. The search for literature has mainly been con-ducted in the material published in scientific journals. The current state of practice, the methodological choices, and the different approaches used in previous work are summarized in this chapter.
The disposition of this chapter has the following order. Firstly, different types of data acquisition techniques for acquiring point clouds are summarized. Secondly, different approaches for segmentation and model reconstruction is presented. Lastly, the field of structural health monitoring is briefly described, and its utilization of UAVs.
2.1
Generating Point Clouds
There are several applications for using point cloud data within the field of construc-tion. Some of these applications being as-built modeling, BIM verification, quality control, and progress estimation [18]. There are mainly two approaches for the ac-quisition of point cloud data, photogrammetry/videogrammetry, and laser scanning. In this section, the principles of theses acquisition techniques will be presented, along with their advantages and disadvantages.
2.1.1
Laser Scanning
A terrestrial stationary laser scanner (TLS) is used for 3D point cloud acquisition of objects or structures. A TLS can capture thousands to millions of points per second, with a high level of accuracy, which results in dense and accurate point clouds [19]. This type of measuring equipment is known as LiDAR (light detection and ranging) and it relies on the time of flight (TOF) principle [20]. The TOF principle is that a photon source emits a laser light onto an object and the time for the light to return to the receiver is recorded. Based on the time for the light to make the round-trip, the distance can be calculated since the speed of light is known.
Scanning using a TLS requires a clear line of sight since it only can capture fea-tures that are directly visible from the scanners’ location [21]. This results in occlu-sions from objects being located in between the scanner and the structure. Another phenomenon is self-shadowing, which is when the geometry of an object causes a blockage of the line of sight to itself, resulting in an area that will not be covered in the scan. To avoid these types of issues with missing data, multiple scanning lo-cations must be used. Assembling a complete point cloud of a bridge using a TLS is not always possible since the technology is bound to a stationary location on the ground. With bridges often spanning over physical obstacles, the adopting of the TLS technology can prove difficult, since the locations to mount the TLS are lim-ited. The complex geometry of certain structural elements or features may require multiple scans from different locations which may not be possible or feasible.
2.1.2
Photogrammetry
Photogrammetry is the process of deriving 3D information from a set of images of an object [20, 22]. The photogrammetric process generally generates a point cloud, where the points describe the position of features matched in the set of pictures [20], the theory behind photogrammetry is further explained in Section 3.1. The term videogrammetry is used for the process of applying photogrammetry to the sequence of photographs in a video. Videogrammetry can in some aspects prove advantageous to photogrammetry, mainly because the sequence of images is known and the overlap of sequential images is large. The downside is that images that are of poor quality or blurry, are more likely to exist in a video and must, therefore, be filtered out before going through the photogrammetrical processing.
2.2
Data Acquisition with UAV
The images used in the photogrammetric process can be acquired with an unmanned aerial vehicle (UAV). It is a type of aircraft that can be remotely controlled, driven autonomously or semi-autonomously [23]. Bridge inspections with UAV’s can reach areas of the structure that would be difficult to observe during an ocular inspection. Nevertheless, structural components having complex shapes with narrow passages are troublesome or sometimes not possible to inspect with the UAV.
The photogrammetrical process requires a complete set of images of the object with a certain overlap to match the images together. The data acquisition is therefore cru-cial for the overall outcome of the model. During autonomous as well as remotely controlled flights, the flight path must be planned properly to collect images of all structural components from different angles. An unstructured approach to the im-ages acquisition and flight path can prove to be very disadvantageous for the outcome of the model [22]. The accuracy of the reconstructed 3D model increases with the number of different images taken of the target object.
According to Chen et al. [21], the most important aspect of data acquisition is the planning of the flight trajectory. Since this step can have the greatest impact on the quality of the generated point cloud. This is because it highly relates to parameters such as image distances, lighting conditions, camera angles, and overlap of the im-ages. Chen et al. [21] has identified the following six steps to ensure a successful data acquisition using a UAV, presented in Table 2.1.
Hallermann and Morgenthal [24] has identified some limitations when applying UAVs for visual inspections. UAVs suitable for close-range inspections have lim-ited payload, hence the camera and battery systems must be of smaller caliber. A small battery restrains the flight time, as a consequence, inspections of large bridges must be partitioned into several flights. In addition to this, the fact that the vehicles often are lightweight makes them sensitive to situations of strong winds and other harsh weather conditions. Disturbing signals or unexpected objects can also cause confusion or failure. An autonomous flight might require manual adjustments in such situations or when GPS connection is lost.
Table 2.1: Steps for planning the data acquisition [21].
Survey objective setting Decide which information is of interest and ascertain which areas need to be inspected. For general bridge inspections, this often includes an overall analysis of the entire structure. In other cases, a critical area could require particular monitoring of how damages change over time.
Site pre-checking Site conditions need to be preparatory investigated. This includes locating nearby or disturbing objects such as trees and buildings, as well as traffic flow on or under the bridge.
Flight planning The UAV trajectory requires planning to cover all rel-evant areas. This includes minimizing blockage from the nearby objects and self-shadowing from the struc-ture itself due to the camera angle as each image is limited by the line of sight.
Risk assessment Risks that the operator is exposed to should be identi-fied, and the risk of unforeseen events that can cause problems for the flight should be planned for. Safe distance to the target, surrounding objects, and an as-signed spot for emergency landing are some important safety precautions.
Permission application Some flights require permission from landowners or authorities. Permission should, therefore, be obtained if needed.
Data collection The data acquisition should be performed with regard to the surrounding. This may require the population to be notified or areas to be restricted.
2.3
Segmentation
Identification and recognition of structural components in point clouds is often an initial step before applying a reasoning framework for model reconstruction pro-cesses [25]. Segmentation of data implies that the data is to be arranged and labeled into clusters composed by points with similar properties or characteristics [25, 26]. The goal of segmentation is to create workable sets of data which is an essential part of the interpretation and further processing of point clouds. Segmentation processes are recognized as one of the most important processes for automatized procedures with large unstructured point clouds [25, 27]. Segmentation is a challenging task due to the unstructured nature of point clouds and for numerous applications the vast size of the data set. Segmentation techniques have applications throughout various fields, such as computer vision, robotics, and remote sensing [28].
Segmentation algorithms employ a range of different approaches based on vari-ous methods. These algorithms are grouped depending on what they are based on. The main categories of segmentation algorithms are region growing-based methods, model fitting-based methods, and clustering feature-based methods [27].
Region growing algorithms examine the boundary points around an initial seed point to determine whether they should be added to the segment or not, depending on if there are similarities with the seed point [27]. The similarities between seed and neighboring points can be derived from information about the point’s normal, color, or position [25]. The region is grown from adding the boundary points that fulfill the criteria for similarity, when none of the neighboring points can be added to the region a new seed point is selected. This process is repeated until all points belong to a region, making region growing an iterative method.
For the model fitting-based segmentation, there are two widely used algorithms, Hough Transform developed by Ballard [29] and Random Sample Consensus (RANSAC) developed by Fischler and Bolles [30]. Hough Transform has been used to locate lines [31], planes [32] as well as other geometrical objects like spheres and cylin-ders in point clouds. RANSAC is an iterative method for fitting a model to a data set. Points are sampled randomly from the data set followed by the implementation of a voting scheme to reach a consensus for which fitting is optimal concerning the num-ber of inliers [30]. The method has since it was first developed by Fischler and Bolles [30] been adapted to work for the segmentation of 3D point clouds. RANSAC has been applied to identify a range of geometrical shapes like planes, cones, cylinders, and spheres from point cloud data [33]. Anagnostopoulos et al. [12] implements a
version of RANSAC for point cloud shape detection to segment planar surfaces for identifying and modeling of walls, floors, and ceiling in buildings.
Furthermore, the segmentation process can also be based on information about the normals to the points in the PCD, as the method proposed for the segmentation of arch bridges by Riveiro, DeJong, and Conde [34]. The method consists of normal determination by principal component analysis (PCA), which is then converted into spherical coordinates, where the elevation and azimuth angle then are used for the classification of points.
2.4
Principles for Model Reconstruction
There are numerous different methods applied for reconstructing 3D models from point cloud data. In a literature review, Wang and Kim [35] summarizes the tech-niques used for the reconstruction of semantic models from point clouds. These techniques are divided into four different categories, geometric shape descriptor, hard-coded knowledge, supervised learning, and lastly alignment with BIM. Recon-struction of semantic models primarily consists of geometrical modeling and object recognition. Where the objective of the geometrical modeling is to form the geome-tries of building elements from the point cloud data. And object recognition aims to label the objects found in the point cloud into appropriate object categories.
2.4.1
Geometrical Shape Descriptors
Recognition approaches based on geometric shape descriptor uses a library with de-scriptors of various objects and are therefore suitable in cases of when the object to be recognized is already modeled in a CAD/BIM model. The steps of such a method proposed by Sharif et al. [36] follow the steps: (1) Model Library Generation, (2) Scene Representation, and (3) Matching. In the library generation, the descriptors of the objects are calculated based on their models and stored. The scene represen-tation stage consists of a search for target objects in the point cloud, using the shape descriptors in the library as a comparison. Lastly, the potential matches in the point cloud are aligned with the CAD/BIM model to evaluate if its a good match.
2.4.2
Hard-Coded Knowledge
A hard-coded knowledge-based recognition approach employs pre-existing knowl-edge of structural elements from which the objects are classified [35]. The prior
knowledge used to differentiate between features can be information about size, position, orientation, topology, and point density [37]. This type of hard-coded knowledge-based approach has been implemented to identify building components such as walls and roofs from point cloud data [37, 38]. Identification of variant- or deviant-shaped objects remains a challenge for the hard-coded approaches as well as when the complexity of the geometries increases [35].
2.4.3
Machine Learning Approaches
Supervised learning is a machine learning task of training an algorithm to classify an input to output, based on labeled training data [39]. The strategies identified by Wang and Kim [35] involve either point-based or segment-based classification. Point-based classification uses local features, such as color and geometry, of each separate point to determine its object category. For segment-based classification, the point cloud is first segmented into clusters of points representing purposeful fea-tures of the structure, thereafter the segments are classified into an object category. Kim, Son, and Kim [40] identifies structural components of concrete in point cloud data by applying a machine learning algorithm, a support vector machine (SVM), utilizing the Red-Green-Blue (RGB) colorspace that is stored for each point in the PCD. Sacks et al. [41] has developed a machine learning-based technique that em-ploys a repository of bridge component shapes from which geometries in the PCD are matched.
2.4.4
BIM Alignment Approaches
BIM alignment-based approaches require a reference BIM model from which the point cloud is aligned, commonly using georeferencing [35]. The subsequent step includes assigning the points to elements in the reference model. Lastly, the points assigned to the same element is used in the modeling of the element. These types of alignment approaches are generally used for compliance control and assessing the progress of construction activities, as seen in the work of Zhang and Arditi [42] and Bosché [43].
2.5
Structural Health Monitoring
Structural health monitoring (SHM) is the process of assessing the structural in-tegrity and safety of a structure using an automatic monitoring system [44]. By
applying an SHM strategy to monitor a structure, damages can be detected in an early stage allowing efficient maintenance planning and management for infrastruc-ture [45, 46]. An SHM strategy generally consists of data acquisition through sen-sory systems, such as accelerometers for measuring structural response, followed by data transmission, data processing, data management, health evaluation, and lastly, decision making [44]. The development of data-driven techniques such as machine learning and computer vision has gained more interest in their applicability in the field of SHM, as the techniques are becoming more reliable and accurate for analyt-ical tasks such as pattern recognition and classification [47].
2.5.1
UAVs in Structural Health Monitoring
Unmanned aerial vehicles (UAVs) has as of late been applied for several types of tasks within the field of structural health monitoring. Within SHM, UAVs are most commonly used in conjunction with a high-resolution camera to acquire photographs used for visual defect detection. Defect detection such as identifying cracks can be assessed in an automated manner using pre-trained machine learning algorithms [48]. To facilitate non-visual assessments, UAVs have been suggested to be used for the placement of measuring equipment onto civil engineering structures. Na and Baek [49] has proposed to use a UAV to attach piezoelectric transducers onto struc-tures for data acquisition. Escobar-Wolf et al. [50] has employed a UAV equipped with a thermal camera to detect and map concrete delamination of concrete bridge decks. A broad spectrum of different sensors can be attached to a UAV, allowing for a wide range of possible applications within bridge inspection, e.g. RGB cameras, thermal cameras, laser scanners, or hyperspectral cameras [51].
Theoretical Background
This section provides some theoretical background to the techniques and concepts used in this master thesis. Firstly, fundamental principles within photogrammetry and point cloud data are described. Thereafter, relevant theory to the different meth-ods used in the algorithms is explained.
3.1
Photogrammetry
Photogrammetry comprises methods for measurement and interpretation of images, to derive the shape and location of an object [22]. The main principle of photogram-metry is triangulation, where points in space are reconstructed from the convergence from 2D target points in different pictures. Photogrammetry is generally divided into two categories, terrestrial and aerial photogrammetry [52]. The principle of terres-trial photogrammetry is that the photos are collected from the ground surface, at a fixed terrestrial position. It is, therefore, normally performed with stationary cam-eras, where images are captured from different angles relatively close to the object to provide detailed information about its dimensions. In aerial photogrammetry, im-ages are instead taken from aerial vehicles. This includes imim-ages taken overhead and is often used for creating topological maps and providing information on land usage. Aerial photogrammetry is the processing of aerial photographs taken from a distance of more than 300 away from the object in question [22]. Whereas, the processing of photographs taken closer than 300 m is referred to as close-range photogrammetry.
The UAV operator must collect images that fully cover the object given certain pho-togrammetrical requirements [53]. To be able to triangulate the position in 3D space
it is a requirement that each point on the structure is captured in at least two photos [11]. Furthermore, to ensure the robustness of the image matching, there is a need for a 50 % overlap of images [54].
When using a UAV for image acquisition, each image can be georeferenced. A built-in GPS built-in the UAV stores its position for each taken image. This position is embed-ded in the image metadata. The georeference generally describes the position by its latitude, longitude, and altitude. When processing the images in the photogramme-try mapping software, these coordinates assist the mapping process. It is also used to store the geographical location of the object.
3.1.1
Structure from Motion
The process of generating a 3D point cloud from the 2D images is based on the structure from motion (SfM) principle, first presented by Ullman [55]. The method utilizes a series of 2D images to identify matching features. It generates automatic tie points on the target object that are identified in as many separate images as possible. Tie points can be chosen randomly or formed where the algorithm anticipates to detect them in other images, like edges or corners. An overlap of images is therefore essential to ensure that each point is captured in multiple images. The concept of structure from motion is illustrated in Figure 3.1:
The cameras’ georeference for each image is used to calculate the position of the point on the object such that corresponding viewing rays intersect. The point po-sition is transformed to object 3D coordinates which are used to generate a dense point cloud.
3.1.2
Digital Photography
Capturing an image electronically rather than through analog methods is known as digital photography. A digital camera uses an array of electronic photodetectors, an image sensor, where light is converted into electronic signals which are stored on a memory card [57]. The lens of the camera collects and focuses the light onto the image sensor allowing the information to be detected and conveyed.
For digital cameras, a distinction between two different systems is generally made, cameras that are mirrored or mirrorless [58]. Digital single-lens reflex (DSLR) eras are commonly referred to as mirrored cameras. The principle of mirrored cam-eras is that the light first enters the lens, where it is redirected through a mirror located in front of the image sensor onto a pentaprism. A pentaprism is a five-sided prism where the light is redirected by a constant 90◦, independent on the angle of the entry beam. The light is reflected twice to not invert the image. Lastly, the light reaches the optical viewfinder, where the photographer can view the scene. When an image is to be captured the first mirror is lifted and the light hits the image sensor. For a mirrorless camera, there is no mirror to redirect the light coming through the lens, the light is therefore directly focused onto the image sensor. The viewfinder in mirrorless cameras is therefore electronic instead of optical, as for mirrored cameras.
The quality and accuracy of the photogrammetry are highly influenced by the quality of the photos [59]. The quality of the photos is in turn affected by parameters such as lighting conditions, image distance, movements as well as weather conditions [60]. For images collected with a UAV, the movement of the camera becomes a highly influential parameter for the image quality [60, 24]. Since the UAV is an aerial vehicle with low weight, its stability in the air is affected by changes in wind speed and direction, both horizontally and vertically. This type of movement can result in motion blur in the images.
(a) DSLR camera (b) Mirrorless camera Figure 3.2: Working mechanism of cameras.
3.2
Point Cloud Data
A point clouds consist of a collection of points in a 3-dimensional space where each point is defined by their coordinates. Depending on the file format of the point cloud, the points can contain additional information such as color and normal vectors. The purpose of the point cloud is to provide an accurate representation of the objects dimensions, geometry, and orientation.
Several different PCD file formats exist. They differ in storage-size and in what in-formation they can contain. Some of the common file formats and their properties are summarized in Table 3.1. The formats can be in either American Standard Code for Information Interchange (ASCII) or binary format. ASCII files are commonly larger but are easier to interpret. Normally, all points are represented by their Carte-sian coordinates (x, y, z). If the file includes color, each point is further defined with the individual Red-Green-Blue (RGB) or RGBA-value, where alpha (A) represents the transparency. Surface normals can be incorporated as vectors (nx, ny, nz) [61].
The polygon file format (PLY) can store both point clouds and surface meshes since it can contain both element vertices and faces [22]. An example of an ASCII PLY-file containing 3D points as element vertices with coordinates and RGBA-color is presented in Figure 3.3.
Table 3.1: Properties of different PCD file formats.
Content XYZ PLY LAS
Points X X X
Polygons X
Coordinates X X X
Color X X X
Surface normals X* X
ASCII/Binary ASCII Both Binary
Compatible with
Pix4D X X X
Open3D library X X
format ascii 1.0 element vertex 4 property float x property float y property float z property uchar red property uchar green property uchar blue property uchar alpha element face 0
property list uchar int vertex_indices end_header
1 1 1 127 102 91 255 -1 1 -1 127 102 91 255 -1 -1 1 133 104 88 255 1 -1 -1 133 104 88 255
Figure 3.3: Example of PLY-file.
3.2.1
Point Cloud Defects
Several parameters will have a strong impact on the point cloud quality. The most influential parameters for 3D reconstruction are according to Berger et al. [62], non-uniform density, noise, misalignment, outliers, and missing data. The different im-perfections are described below and illustrated for 2D data in Figure 3.4. These defects may result in errors in later processing steps [19]. Therefore, if possible, they should be avoided or filtered out.
Non-uniform Density
Sampling density describes the distribution of points in the point cloud as seen in Figure 3.4 (b). Depending on how the data collection is performed there will be more information in some regions, resulting in a higher density of points. Parameters such as the sampling distance and the shape of the structure can influence the density of the point cloud [62]. A non-uniform density could result in some regions having an insufficient number of points.
Missing Data
Gaps in the PCD, as in Figure 3.4 (c), are referred to as missing data. This is com-monly occurring for complicated structures as the data acquisition is limited by the line of sight. A complex geometry with narrow regions can result in self-shadowing of the structure or insufficient data for image matching [21]. Another reason for missing data in PCD is errors in the photogrammetrical processing. Increasing the image overlap can reduce the amount of missing data.
Outliers
Outliers in the point cloud data are points or clusters of points that lie outside the actual surfaces of the object that is represented. This is illustrated in Figure 3.4 (d). For point clouds created through photogrammetry, outliers can be caused by several reasons. The main reasons being; errors due to internal camera parameters, imaging setting, and the software used for processing. The internal camera parameters are; the principal distance, the principal point, and the lens distortion coefficients. The imaging setting refers to parameters such as shooting distance, percentage overlap of photos, as well as the number of overlapping photos [63].
Noise
The point cloud data can be distorted by points scattered close to the surface. These points are usually defined as noise, which is illustrated in Figure 3.4 (e). Noise can be caused by several reasons. The SfM process tends to be disturbed by textureless surroundings, causing the reconstruction algorithm to misinterpret sky-points as part of the structure [21]. Other causes of noise can be, shadows, camera properties, and images taken at a large camera tilt angles.
Misalignment
Defective image acquisition or processing could result in layers of points being offset from the actual surface. Misalignment of surfaces can result in a distorted model reconstruction with measurements deviating from the original geometry, as in Figure 3.4 (f). This tends to be caused during the photogrammetrical mapping process where initial configurations of image locations are not optimally aligned [62].
(a) Original shape (b) Non-uniform density (c) Missing data
(d) Outliers (e) Noise (f) Misalignment Figure 3.4: PCD defects.
3.3
PCD Processing Techniques
This section presents techniques for the processing of PCD to prepare the point cloud for later processing steps. Firstly, a technique for downsampling the point cloud is described. Secondly, the techniques used for filtering out defects in the form of deviating points are presented.
3.3.1
Voxelization
Point clouds of small structures can contain millions of points. Initially, the point clouds contain a large number of redundant points compared to what is sufficient for model reconstruction. Before further processing of the PCD, a reduction of the num-ber of points is typically performed to save computational resources. Such processes are known as downsampling [34]. One technique for downsampling a point cloud is to apply a voxel-grid filter, this process is known as voxelization. A voxel represents a value in 3D space, similar to a pixel in 2D space. Voxelization is essentially the process of converting geometric or volumetric objects into volumetric data in the form of voxels [64]. The stepwise procedure of voxelization is described in Figure 3.5. The points in the point cloud are initially grouped into a 3D voxel grid of appro-priate size, see Figure 3.5 (a) [38]. For the voxels that contain multiple points (b),
the downsampling procedure is to derive a single point that represents the collection of points in that voxel (c). The centroid for the points within the voxel is commonly used as that representation [61]. An additional benefit of voxel downsampling of point cloud data is that the point-density, which can be varying throughout different locations, will become more uniformed throughout the point cloud.
(a) (b) (c)
Figure 3.5: Voxel downsampling.
3.3.2
Noise and Outlier Filtration
As described in chapter 3.2, point clouds can contain defects as noise and outliers. There are several methods for noise and outlier filtration. These techniques have different traits and are based on various techniques. The methods have therefore been classified into different categories, whereof two of the main ones are statistical-based and neighborhood-statistical-based which are further explained below [65].
Statistical-based methods utilize statistical and probabilistic concepts to filter outly-ing and noisy data. By performoutly-ing statistical analysis, points with deviatoutly-ing charac-teristics can be removed from the data set. One statistical approach is to compute the distribution of the euclidean distance between points to a predefined number of its closest neighbors [61, 66]. This is done by assuming a Gaussian distribution and calculating the mean value, µ, and standard deviation, σ, for the distances to neighboring points within the given neighborhood for the point cloud. Points that are within a distance of a selected number of σ to neighboring points are regarded as inliers and kept in the data set, whereas points further away are regarded as outliers and removed from the data set. This is illustrated in Figure 3.6 (a).
Filtration techniques that have its basis in finding deviations within its neighbor-hood to determine the filtering position are known as neighborneighbor-hood-based filtration
methods [67]. For points in point cloud data, these deviations can be based on their position, normal, or region. A procedure for this is to define a neighborhood sur-rounding a point by radius. Points with too few neighbors within the radius are considered outliers, as illustrated in Figure 3.6 (b).
(a) Statistical filtration. (b) Radius filtration. Figure 3.6: Outlier filtration.
3.4
RANSAC
Fischler and Bolles [30] first introduced Random Sample Consensus (RANSAC) for the fitting of a model to a data set. RANSAC aims to perform the model estimation for data sets containing outlying data points, without having these points influencing the estimated model. The RANSAC algorithm has an iterative approach, it initiates by selecting a number of points at random, from the input data set. The number of points is governed by which type of model that is to be fitted to the data set. E.g. the number of points needed to fitting a straight line is two and for a plane, three points are required. The algorithm then compares the coherence between the fitted model and the data points, granted a threshold to what is considered a sufficiently good fit. Data points found to belong to the model, inliers, are considered to be part of the consensus data set and are counted. Points that do not belong to the consensus data set are regarded as outliers (see the cluster of points in Figure 3.7). The procedure is then repeated for a predetermined number of times, n, before termination. The model parameters with the largest consensus data set, the largest number in inliers, is considered to be the best model estimation to the data set. Comparing model fitting with RANSAC to other techniques such as the method of least squares, the main difference is that these methods consider all the available data points for parameter estimation. Therefore, it is also considering data points that are deviating from the
data set, outlying points, which in term affect the model estimations. In contrast, RANSAC comprises the ability to disregard from these outlying points in the data set. This is illustrated in Figure 3.7 with a comparison between RANSAC and linear regression.
Figure 3.7: Model fitting to a data set with outliers.
3.5
Nearest Neighbor Search
Nearest neighbor search (NNS) is a form of proximity search. Given a set of points, S, the closest adjacent point to the query point, q, are to be searched for [68]. The set of points, S, belongs to the d-dimensional vector space M (S ∈ M ) and q ∈ M . The similarity between the points is usually measured using the Euclidean distance or other distance measures. The k-nearest neighbor (k-NN) search applies the same reasoning principle but the search is performed for the k nearest points [69]. The k-NN algorithm is a supervised machine learning algorithm, applicable for both clas-sification studies for the determination of class membership as well as regression problems. For efficient nearest neighbor searches, the concept of k-dimensional trees is explained in Section 3.6.
3.6
K-Dimensional Tree
A k-dimensional tree (KD-tree) is a multidimensional binary search tree for the struc-turing of k-dimensional data [70]. The concept is based on the data being partitioned into a binary structure where every leaf node is a k-dimensional point. Given a set of k-dimensional points, the node is selected as the point in the middle of the data set with regard to one of its dimensions, e.g. x. From this node the remaining points
are partitioned into two subsets, one being the points larger than the node point with regard to the dimension in question and the other the points being smaller. These two subsets of points are then represented by a left and right subtree. The process of growing the tree is continued by repeating the process of selecting the middle point of each individual subset with regard to the next dimension, e.g. y. The process is continued until all points are nodes as illustrated in Figure 3.8. This type of data structure is useful in range searches and nearest neighbor searches. For the applica-tions related to point cloud processing nearest neighbor searches often prove to be faster compared to an exhaustive or brute-force approaches [20]. Searching for the closest data points can be done efficiently because large portions of the tree can be eliminated, hence reducing the search space and the search time [71]. The process for nearest neighbor search using the KD-tree is initiated at the root node of the tree, where the distance is calculated from the target point to the root and this point is held as the current best. The algorithm then recursively processes down to the left side of the tree, calculating the distance and comparing if this point is the current best. If this point is the new current best the distance to its children is calculated. If none of those distances is better than the previous best point, the children node to those points can be discarded or pruned. The algorithm then returns to the root and takes the right subtree, following the same procedure.
Figure 3.8: Data structure of a KD-tree.
3.7
Spatial Geometry
This section presents how planes and lines are described and represented, as well as, some algebraic operations and their implications.
A plane in 3-dimensional space can be described in vector form as Equation 3.1 or by computing its dot product and writing it as a scalar equation, as Equation 3.2.
Where the coefficient d is calculated from Equation 3.3. a b c · x y z − x0 y0 z0 = 0 (3.1) a · x + b · y + c · z + d = 0 (3.2) d = −(a · x0+ b · y0 + c · z0) (3.3)
Furthermore, a plane can also be described in the so called Hessian normal form, where n = (nx, ny, nz) is a unit normal vector to the plane and the constant p is the
orthogonal distance from the plane to the origin, given by Equation 3.5 [72].
n · x = −p (3.4)
p = √ d
a2+ b2+ c2 (3.5)
A line in 3 dimensional space with the direction t = (t1, t2, t3) that intersects the
point, x0 = (x0, y0, z0), is represented by Equation 3.6 given that t1, t2, t3 are non-zero real numbers.
x − x0 t1 = y − y0 t2 = z − z0 t3 (3.6)
Lines can similarly be represented in parametric form as in Equation 3.7, where the parameter s is a real value.
r(s) = x0+ s · t (3.7) For solving the intersection between the two planes, let the normal vectors of the planes be n1 and n2. The intersections between two planes in R3 always occurs along a line, as long as n1 × n2 6= 0, meaning that the planes are not parallel to
each other [72]. The intersection will occur perpendicular to the normals, resulting in the line having the direction t, where t is the cross product of the plane normals, see Equation 3.8.
To specify the line, a point x0, on the line is necessary. This implies, for the case
of a planar intersection, that the point will be on both planes simultaneously, see Equation 3.9. This system is underdetermined since x0 can be any of the points along the line [73].
n1 n2 | · x0 = − p1 p2 (3.9)
To avoid underdetermination of this system of linear equations, an additional crite-rion can be added to where the point is to be located on the line. By stating that the vector to the point, x0, is to be orthogonal to the line direction, t, the dot product
will be zero. Hence resulting in a system of equations (Equation 3.10), where the point, x0, has an solution corresponding to a single coordinate in R3.
n1 n2 t | · x0 = − p1 p2 0 (3.10)
The determinant of a n × n matrix is equal to zero, if one of the rows or the columns are linear combinations of another [72]. Resulting in the system of equations not having a unique solution. For the intersection of three planes in R3, let A be the matrix of coefficients from the scalar plane equation (Equation 3.2).
A = a1 b1 c1 a2 b2 c2 a3 b3 c3 (3.11)
The criteria for a unique intersection point of three planes in R3is, therefore det(A) 6= 0, where A is the matrix presented in Equation 3.11. For the case when the determi-nant is equal to zero, the intersection of the planes will occur along one or two lines, or not at all if all planes are parallel.
3.8
Boundary Representation and Graph Theory
A description of a solid object in terms of its shape and limit, its boundary between model and non-model, is known as a boundary representation (BREP) of the object [6]. A BREP is typically composed of two types of information: topological and
geometric [74]. The topological entities in a BREP are vertices, edges, and faces. Where the faces are composed of its bounding edges and vertices. An element com-posed of multiple connected faces is known as a shell. The geometric information describes the topological entities in terms there shape and location in space. The data structure of a boundary representation is in mathematical terms a graph [6].
A graph is a mathematical structure used to describe relations between objects and it consists of points and connections between these [75]. A graph, G = (V, E), is composed out of a set of vertices, V , and edges, E. A vertex, also known as a node, is a point where one or multiple edges meet. The connection between two vertices is referred to as an edge.
Figure 3.9: Edge connecting the vertices A and B.
The number of edges that connect to a vertex is denoted as the degree of the vertex (deg). A graph can be directed in different ways depending on whether the edges only allow for movement in a specified direction or not. Graphs without any specified directions are known as non-directed graphs whereas a directed graph has restric-tions to the direction of edges. Figure 3.10 represents a non-directed graph, where the degrees of its vertices are deg(A) = deg(D) = 3 and deg(B) = deg(C) = 2.
Figure 3.10: Non-directed graph.
Graphs can have different connectivity throughout its vertices. For a graph, G, to be called connected all of its vertices must be possible to reach through a path of edges from any vertex [76], see Figure 3.11 (a). For the case when a vertex or a
group of vertices are not reachable through a path of edges the graph is said to be disconnected, see Figure 3.11 (b).
(a) Connected graph
(b) Disconnected graph
Method
This chapter describes the approach used in our attempts to solve the problem of identifying relevant points for reconstructing a model from a point cloud (PC). The choices of programming language, libraries, and modeling software are motivated. Moreover, the overall process illustrated in Figure 4.1 is described. Firstly, the gen-eration of point clouds is described. In order to test the method on simplified cases, synthetic point clouds were first generated. Thereafter, point clouds of real-life struc-tures were generated through UAV image acquisition and photogrammetry. Lastly, the approach for extracting vertices and edges of the structures is described together with the approach for modeling of these.
Creating Point Clouds Pre-Processing
Segmentation Extracting Vertices and Edges
BREP Model
Figure 4.1: Flowchart of the overall process.
4.1
Python
Python is a programming language, first released in 1991 [77]. Python is used in a wide range of applications such as software development, web development as well as scientific computing. For this thesis, Python 3.6 has been used for handling and processing of point cloud data, as well as for writing modules for extraction of geometrical information and visualization. There is a wide range of open-source libraries for the Python language. A library is a collection of modules, classes, and functions. The main libraries that have been used in this thesis are described below.
4.1.1
Open3D
Zhou, Park, and Koltun [78] has developed an open-source library for 3D data han-dling and processing. The library, Open3D, contains data structures and algorithms for both the programming languages Python and C++. Features for noise filtration, segmentation, and visualization of PCD are implemented in the library. These fea-tures in the Open3D version 0.9.0 have been used for processing and visualization of PCD.
4.1.2
NumPy
NumPy is a library for scientific computing. It contains tools for computation with N-dimensional arrays, linear algebra, as well as other mathematical operations [79]. NumPy version 1.18.1 has been used for a majority of the computational tasks as well as for storing data in arrays.
4.2
Rhinoceros
Rhinoceros, also known as Rhino, is a computer-aided design (CAD) application de-veloped by Robert McNeel & Associates [80]. Rhino supports two types of scripting languages, Rhinoscript (which is based on VBScript) and Python [81]. Applications with scripting possibilities allow for more or less automated and efficient execution of repetitive tasks, such as modeling of geometries derived from the point cloud data. Using a CAD application with Python scripting capability allowed for good interaction with the computations performed in Python.
4.3
Simulation of Test Data
Testing has an essential role throughout the development. It allows for the creation of specific cases where the outcome is predetermined or easily predictable. By pro-ducing simplified test cases, it is possible to evaluate if the program is behaving as intended.
The test data was in the form of point clouds representing various geometries and simple bridges. The simulation of the test data was performed in three stages, as illustrated in Figure 4.2. Firstly, the intended structure was modeled in Rhinoceros as a solid geometry. Secondly, the model was exported as a polygonal mesh in the PLY file format. The PLY format specifies data by assigning elements with a set of properties, mainly vertex and face data [82]. Lastly, the mesh-model was converted into and saved as a point cloud using the Open3D library. The number of points in the simulated point cloud was predetermined to represent an appropriate density corresponding to real conditions.
Rhinoceros
Solid Geometry
Polygonal Mesh
PLY-format
Open3D
Mesh to Point Cloud
Point Cloud
Figure 4.2: Flowchart of the test data simulation.
4.4
Photogrammetry
The photogrammetrical data used for the analysis were acquired with a UAV and processed with the Pix4D software (see Section 4.4.2). This process is illustrated in four steps in Figure 4.3 and described further in the following sections.
Flight Planning Image Acquisition
UAV with digital camera
Pix4D
Mapping process
Point Cloud
Figure 4.3: Flowchart of the photogrammetric process.
4.4.1
Data Acquisition
For the image acquisition, a consumer UAV equipped with a mirrorless camera was operated manually by remote control. The equipment used is described in Table 4.1.
As mentioned in Section 2.2, it is important to have a well-planned flight, to capture the entire structure with sufficient image overlap. The flight trajectory, therefore, followed the pattern illustrated in Figure 4.4. To be able to photograph the bottom of geometries, the camera gimball was rotated upwards. Because of the limited flight time of the UAV, the acquisition had to be divided into multiple flights with a change of battery in between.
Table 4.1: Equipment used for image acquisition [83].
Specifications UAV Weight: 907 g
DJI Mavic 2 Pro Max tilt angle: 35◦
Max Flight Time: 31 minutes
Camera Sensor: 20 MP 1"
Hasselblad L1D-20c Photo format: JPEG / DNG (RAW) Image size: 5472 × 3648
