USING A SMARTPHONE
Joakim Wiklund
A thesis submitted for the degree of
Master of Science in Engineering Physics
Geometry measurements using a smartphone Joakim Wiklund, joakim.wiklund@hotmail.se
Supervisors: Tobias Nilsson Knightec
Christian Larsen Department of Physics
Examiner: Magnus Andersson Department of Physics
Copyright c 2020. All Rights Reserved.
Acknowledgements
I would like to begin by thanking the people at Knightec for their help during this project, and especially my supervisor Tobias Nilsson who provided me with many ideas and insights during our weekly meetings and discussions in the office. I would also like to thank Pontus Gjertz for his generous help with 3D-printing a smartphone stand.
1 Introduction 1
1.1 Background . . . 1
1.2 Deflectometry . . . 1
1.3 Purpose and aims . . . 4
2 Theory 5 2.1 Triangulation . . . 5
2.2 Surface integration . . . 9
2.3 Patterns and distance L . . . 9
2.4 Gap and flush . . . 12
3 Method 13 3.1 Performing tests and setups . . . 13
3.1.1 3D-printed smartphone holder . . . 13
3.1.2 Stationary tests without holder . . . 14
3.1.3 Handheld tests . . . 15
3.2 Gathering the necessary data . . . 16
3.3 Image processing . . . 17
3.4 Stabilizing images . . . 19
3.5 Approximation of the height h . . . 21
3.6 Checking for and determining gap and flush . . . 23
3.7 Producing the heightmap . . . 24
4 Results and discussion 25 4.1 Flat mirror parallel to smartphone . . . 25
4.2 Flat mirror angled relative to smartphone . . . 26
4.3 Partially blocked mirror . . . 28
4.3.1 Height variation study . . . 30
4.4 Offset mirrors . . . 31
4.5 Concave mirror . . . 32
4.6 Flat mirror parallel to phone, handheld . . . 35
4.7 Flat mirror with tape, handheld . . . 37
4.8 Gap and flush on a car . . . 39
4.9 General discussion . . . 41
5 Conclusions 43
A Examples of test data 45
1 INTRODUCTION
1
Introduction
1.1
Background
Quality Assurance is an important step in almost all types of production, and in many cases the methods of measuring the quality of manufacturing are less than optimal. Even some large car manufacturers still use their employees’ hands and eyes to check whether or not the panels are smooth enough, or if the various doors and panels are mounted correctly[1]. The quality control in factories is evolving with new technology, and that technology itself is a growing industry.
Shape measuring techniques of different types are already popular in many industries for the purpose of inspection[2][3]. One common measuring tech-nique is structured light, a method based around projecting a known pattern onto the object that is to be measured and taking photos of the resulting pattern on the object. By knowing the geometry of the setup this informa-tion can be used to produce a 3D-model of the object with high accuracy. Even relatively simple versions of such methods can reach an accuracy of significantly less than a millimeter[4].
These methods have many uses, but they also have some limitations. One of these limitations is that they require a projector of some kind to project the patterns onto the object in question. They also require the object to be diffuse enough, that is, the object cannot be reflective or the method simply does not work. A method that could handle both of these limitations is deflectometry.
1.2
Deflectometry
with some other geometric properties we can create a 3D-model of what the object looks like on the part which contains the reflected pattern.
Figure 1: An illustration of the idea behind deflectometry.
This technology has many potential areas of application in various industries where portable and solutions are valuable. One of these potential applications is within the car industry where we may be able to use it to determine what is known as gap and flush after installation of car doors. Gap and flush are qualities that are commonly used to determine how well the doors are mounted by measuring the distance between the two panels as well as the offset between them.
The reason patterns are shown on the screen is in order to figure out what point on the screen was reflected into what pixel in the camera. In order to determine this, there are several different types of pattern that can be shown on the screen. Two common pattern methods for this are phase-shifting and centroiding [5].
1 INTRODUCTION
point on the screen, for each pixel on the captured images. The pattern is most commonly shifted four times by π/2, but in order to reduce errors methods with up to seven shifts have been suggested [6]. These methods are however sensitive to changes and inconsistencies in the surrounding light and environment.
Figure 2: Example of a pattern that is used in phase-shifting algorithms.
The second common method, centroiding, is based on lighting up only a small part of the screen at a time to and perform calculations for that single pixel and doing that for the entire area which should be calculated for. There are several ways of doing this, either a single pixel is lit, a small square is lit, or more commonly, a vertical or horizontal line is lit. The benefit of centroiding is that they are insensitive to light variations in the environment[5], but a large downside is that this method requires a very large amount of patterns, and thus also a large amount of images.
1.3
Purpose and aims
The main purpose of this project is to investigate the possibilities and the potential of developing a program for smartphones that utilizes only its own screen and front-facing camera to perform measurements for deflectometry without the use of any external tools. This is done by implementing an application for a smartphone to gather all the test data by itself as well as implementing a program that performs the necessary calculations to perform a surface measurement.
2 THEORY
2
Theory
2.1
Triangulation
The end goal for the calculations is to find h, the distance from the smart-phone to each point on the reflection on the object. This is done by first gathering information about some other variables that can be used to pro-duce a heightmap. The principle behind these calculations is triangulation, a principle explained in detail in [8]. Each relevant variable is marked in figure 3, all of which are individually calculated for each pixel in the photos from the camera.
As mentioned, the end goal is to find the height h from the screen to the object for each point on the reflection, and to do this we have a few known properties to work with. These properties are all dependent on the smartphone used, such as the geometry of the smartphone, the pixel density of the display, and the field-of-view (FOV) and resolution of the camera.
Figure 3: Schematic drawing of a general setup with a smartphone, including its screen and camera, as well as the specular object that is to be measured.
The first of the relevant variables is α, the angle at which the light entered the camera to the specific pixel in the image we are looking at. This angle is determined by
α = atan x · tan(F OV ) resolution
, (1)
2 THEORY
x
resolution
α FOV
Figure 4: Schematic drawing of the internal properties of the camera used to calculate the entry angle α for a specific pixel in the image.
Knowing this angle we can calculate d, the distance between the center point of the camera to the point on the reflection which was reflected in to our camera pixel.
d = h · tan(α), (2)
where h is the approximated distance from the smartphone to the object.
n^ α η β η π/2-α L d h (a) δx δy (b)
Figure 5: (a) shows a schematic drawing of a light ray and its angles in the general setup. (b) shows an illustration of how to measure δ on a general smartphone.
Knowing these variables we can use them to calculate the angles shown in figure 5 according to η = atan h L − d (3) and β = π − π 2 − α + η 2 = π 4 + α − η 2 . (4)
With η and β known as well we can, for each camera pixel, calculate the normal angle ˆn for each point on the reflection using
ˆ
n = η + β = π 4 +
α + η
2 . (5)
2 THEORY
determined its geometry for the region in which a reflection is captured by the camera.
2.2
Surface integration
To produce the final heightmap we use surface integration. Knowing the normal angles ˆn and distances d between each pixel it is straight-forward to integrate a surface. From a pixel (i, j) with a known height hi,j, the height for pixel (i + 1, j) can be calculated using
hi+1,j = hi,j + tan ˆ nx− π 2 (dx(i + 1, j) − dx(i, j)) , (6)
where ˆnx is the x-component of the surface normal and dx is the distance from the camera center in the x-direction. The height for pixel (i, j + 1) can be calculated similarly using
hi,j+1 = hi,j+ tan ˆ ny − π 2 (dy(i, j + 1) − dy(i, j)) (7)
where ˆny is the y-component of the surface normal and dy is the distance from the camera center in the y-direction.
2.3
Patterns and distance L
Figure 6: An example sequence of patterns that are shown to determine which point on the screen the camera is photographing for an individual pixel.
2 THEORY
Figure 7: A sequence of binary patterns with one pixel marked in gray. The binary sequence for the gray pixel is determined by what color it covers in the pattern, the sequence for this pixel is [1,1,0,1,1,0,1,1] where 1 is white and 0 is black.
Since this sequence is unique, it can be used to match each pixel in our sequence of photos to a point on the screen and thus help us determine the distance L, which is the distance from the camera to the point on the screen which is reflected in to the given camera pixel.
For practical purposes, we can decide to end the pattern sequence before we reach a pattern with squares of 1 pixel length and instead have the final pattern contain larger squares. This leads to less accurate calculations of L as each camera pixel is matched to a region on the screen as large as the size of the squares in the final pattern.
Determining the distance L after matching the binary sequence to a point on (or part of) the screen is done separately for x and y using
L = p · pitch + δ (8)
2.4
Gap and flush
Gap is defined to be the perpendicular distance from the edge of one surface to the point where it intersects the other surface, see figure 8. If the value for the gap varies between different points a median is considered to be the gap.
Gap
Figure 8: An illustration describing how gap is defined for two surfaces.
The definition for the flush is instead defined to be the perpendicular distance from the line stretching from one surface to the edge point of the second surface, see figure 9. Like with the gap, if the flush varies over the different points a median is taken as the flush.
Flush
3 METHOD
3
Method
3.1
Performing tests and setups
To speed up calculations and to simplify programming of an Android app, the decision was made to perform all tests on a smartphone and then transfer the images captured to a computer. Once on the computer all calculations were performed using MATLAB.
3.1.1 3D-printed smartphone holder
Figure 10: A setup for testing with custom designed holder.
3.1.2 Stationary tests without holder
3 METHOD
Figure 11: A setup for testing without the custom holder.
3.1.3 Handheld tests
Figure 12: A setup for handheld testing.
3.2
Gathering the necessary data
To collect the needed information, a requirement is that we can show a sequence of patterns on a smartphone screen and for each pattern take a photo with the front-facing camera on the same phone. To do this efficiently an app was created for Android phones using Android Studio. The resulting app has simple functionality, it shows the patterns in order and captures a sequence of photos while the patterns are being shown.
The pattern sequence was implemented in the form of a video where each pattern is shown for a duration of 100 ms each, and the video starts playing when the user clicks one of the volume buttons. This sequence begins by showing an all white image followed by an all black image that will be used to determine the region to consider for calculations.
3 METHOD
our purposes. This likely occurs because of the shutter speed of the camera or the compression in the video shown on the screen, but since this was not thoroughly investigated there is a possibility that the reason is something different. In order to make sure that every real pattern is captured at least once in an image without this problem, three images are taken for each pattern. In most cases, one of these is the transition image and two images are the real pattern. When only taking two images per pattern, there is still a chance to not capture a good image of the real pattern without interference from the previous or upcoming patterns which is the reason to take three images per pattern.
The app saves these photographs uncompressed as .yuv files on the smart-phones internal storage that need to be transferred to the computer on which the calculations are to be performed.
3.3
Image processing
Figure 13: Example of a raw-image captured by the camera before any pro-cessing has been done.
The entire process of reading and processing images is described in algorithm 1, and in further detail below.
Algorithm 1 Read and process images 1: Load images from directory
2: Use all white and all black images to find area of interest 3: for pattern images do
4: Undistort image based on camera calibration 5: Stabilize image with previous image
6: Adjust intensities to utilize full intensity range 7: Binarize image based on threshold value
The first calibration is to adjust the pictures based on a calibration of the used device’s camera. This includes adjusting the images based on variables such as radial and tangential distortion and skew.
3 METHOD
threshold value to binarize the image. The four corners of the relevant area are then found in the binary image and the largest rectangle between these four corners will be used as the area to be calculated.
At this point, each image should only contain the reflection of its corre-sponding pattern in a grayscale image. In order to find a binary sequence for each pixel from this sequence of grayscale images we must first convert the grayscale to binary. This is done using a threshold value based on how the image looks, and in this case a median of the intensities is used as the threshold since each pattern is approximately half white and half black.
The images should at this point contain only the reflection of the patterns shown on the screen and in binary form. This means that each pixel in the image sequence has a binary sequence, and thus each pixel can be matched with a point on the screen. An example of the first four images in the sequence after all processing is done is presented in figure 14.
Figure 14: Example of the first four images in the sequence after all image processing has been done.
image sequence.
The first step in this algorithm is to pick two images, one to be stabilized and one to be stabilized against. This means that the image we want to stabilize will be adjusted to match the features of the second image. The features of these two images are extracted using the FAST (Features from Accelerated Segment Test) algorithm that is well described in [9]. This algorithm will extract corners from the images shown as green crosses in figure 15. The area from which the features are extracted is set to be only in a small region around the smartphone itself in order to force stabilization around the phone instead of the surrounding area.
(a) (b)
Figure 15: Two images two be stabilized with their features extracted and shown as green crosses. (a) shows the base image that the other image (b) will be stabilized against.
The next step in the stabilizing process is to match the points extracted from each image to each other to figure out which points in image one correspond to which points in image two. This is done by extracting the features for each point using FREAK (Fast Retina Keypoint) descriptors[10] and matching descriptors between the images.
3 METHOD
show the features from the second image, while the yellow lines show which features are matched.
(a) (b)
Figure 16: Both images superimposed on each other, one in cyan and one in red. The red circles show the features from the first image and the green crosses show the features from the second image. (a) shows the matched features before alignment with yellow lines linking each pair, and (b) shows the same after alignment, excluding matched points with too much distance between them. The red line along the hand in (a) indicates that the two superimposed images are misaligned.
3.5
Approximation of the height h
Figure 17: The pattern shown on the screen for height approximation.
Since we know the width of the line shown on the smartphone, and we can calculate the values of α for each end of the line according to equation 1, it is simple to approximate the height h.
h = ∆L
2 (tan(α2) − tan(α1))
, (9)
where ∆L is the width of the line shown on the screen, and α1 and α2 are the angles from the camera to the left and right edges of the line.
This approximation can either be done for the entire length of the line and use different heights for different points of the reflection or a single value can be used for the entire reflection. Generally, the approximation is best closest to the center of the image, and therefore a median of the values closest to the center of the image is typically used in this project.
3 METHOD
3.6
Checking for and determining gap and flush
The most common case when there is a gap involved in the data it is au-tomatically discovered by the initial determination of relevant data, but it is not guaranteed in all cases. An example of a case where it would not be automatically found is when we have two overlapping but offset mirrors. In this case the reflection would be more or less continuous even if it covers two different mirrors.
For cases like this it is desirable to be able to find this gap and manually create a small gap in the data in order to be able to perform calculations. This gap can be found in different ways, one of which is implemented in this program. The method chosen for this is to look through the approximated height curve and look for very drastic differences over a small distance, which indicates some form of discontinuity in the object or objects.
When a gap exists in the data, and thus also in the heightmap, we can cal-culate the size of the gap and flush. To do this, the program goes through each column in the heightmap and finds where the gap begins and ends for each column. Separate linear interpolations are then created for both the beginning and end of the gap and the gap is considered to be the perpendic-ular distance from one interpolation to the other, similar to the line shown in figure 8.
In order to calculate the flush of the data we instead go through all columns in the heightmap and create a linear interpolation of all heights before the gap. The flush for this column is then the perpendicular distance between the first interpolation to the first data point after the gap, as demonstrated in figure 9.
Since car doors are generally designed with curved edges, measuring with deflectometry on these will make it slightly more difficult to determine the gap in a good way. The curved edges will reflect the light away from the camera, which will lead to the camera not seeing any pattern on a part of the door which is not supposed to be considered a gap. With this in mind, a different way of determining gap on cars was found.
part inside the gap will not get much light. Using this dark area along with the average physical distance between each pixel in the reflection we can determine the gap of a car door in a much better way.
3.7
Producing the heightmap
The algorithm used to produce a heightmap given a set of images is described in algorithm 2. Each part of this algorithm is described in detail in previous sections. The iterations in this algorithm are the amount of times to refine the heightmap, with one iteration to go from one approximated height to a heightmap and each consecutive iteration utilizing the previous heightmap to create a more accurate heightmap.
Algorithm 2 Producing the heightmap 1: Read and process images
2: for each iteration do
3: if heightmap exists then 4: heights ← heightmap
5: else
6: Approximate heights 7: for each pixel do
8: Match pixel with pattern sequence 9: Calculate normal angles
4 RESULTS AND DISCUSSION
4
Results and discussion
4.1
Flat mirror parallel to smartphone
For the simplest possible case, we have a flat mirror positioned a set distance from the smartphone used to take pictures with the mirror and smartphone being parallel to each other. This is done with the 3D-printed holder shown in section 3.1.1. Examples of how the test data can look both before and after image processing has been done can be found in appendix A.
In this case we would expect that every normal angle to the surface, in this case the mirror, would be perfectly 90 degrees. This is almost true for the data that can be seen in figure 18, which shows that we have angles at around 89.5◦ for the x-direction and 89.9◦ for the y-direction. The reason that these angles are not perfectly 90 degrees is almost certainly that the physical setup from which these values were gathered was not perfect.
mirror, or at least close to it, we should have normal values that are constant in each direction. In this case the difference from largest to smallest angle is 0.15◦ in x-direction and 0.12◦ in the y-direction.
Looking at the heightmap for this dataset it is clear that is not perfectly flat, but the variation in height from highest to lowest is around 0.3 mm. This is a direct consequence of the angles not being perfectly 90 degrees, but one would expect the heightmap represents an almost perfectly flat surface.
While these values are not exactly the same as the values we were expecting, they are very close. The normal angles not being perfectly 90◦ is caused by the holder itself not being perfect and they are close enough for it to be believable that the offset is the same as the imperfection in the holder. The heights for each point on the reflection are all very close to the 15.5 cm we expect since we are using the 3D-printed holder described in section 3.1.1.
The variation in normal angles is small, and will not contribute to any large errors in the resulting heightmap. A small variation in normal angles is also a good indicator that the height approximation is sufficient since a height approximation that is incorrect will lead to the normal angles varying heavily over the surface.
4.2
Flat mirror angled relative to smartphone
4 RESULTS AND DISCUSSION
Figure 19: Calculated normal angles and heightmap from a test with a flat mirror angled relative to the smartphone.
Similarly to the case with the parallel flat mirror the normal angles in x-direction is not perfectly 90◦, with it being around 89.8◦ for this case. As expected the normal angles for the y-direction are far from 90◦ at around 103◦, that is of course 13◦ off parallel.
In this case the differences between smallest and largest angles are slightly larger than for the parallel mirror. The angles in x-direction this time have a difference of 0.48◦ and in the y-direction the difference is 0.14◦. Still, these differences make for small differences in the heightmap.
the height approximation in this case is slightly off, but since the variation is relatively small the height approximation is still close to reality.
This is also evident in the comparison of the height difference over the heightmap. A difference in height of 0.2 mm would likely be acceptable in many applications and a slight tweak of the approximated height could make it even better.
4.3
Partially blocked mirror
In an effort to simulate the calculation of a gap in the simplest form we go back the smartphone being parallel to a flat mirror, but this time have the reflection being partially blocked by a piece of tape placed on the mirror. This will introduce a gap in the data and therefore we can measure the width of this piece of tape, similarly to how we would like to measure the gap in a car. This test was also done with the holder from section 3.1.1.
4 RESULTS AND DISCUSSION
Figure 20: Calculated normal angles and heightmap from a test with a par-tially blocked flat mirror parallel to the smartphone.
Looking at the differences in normal angles for this case we have a difference of 0.15◦ in the x-direction and 0.27◦ in the y-direction. This is again on the scale of the previously shown tests, and will not cause any large disturbances in the heightmap.
For this case we also have something relevant to compare to a real world measurement. That is of course the width of the tape, or the "gap" between these two parts of the mirror. When measuring the physical tape with a ruler the result was that the width was around 1.85 cm. With this dataset the program calculated a gap of 1.87 cm, giving a difference of 0.2 mm between the measured gap and the calculated gap.
4.3.1 Height variation study
To determine the importance of an accurate approximation or measurement of the height h we can look at how the results change when adjusting only the approximated height in different amounts. For this analysis the case with a flat parallel mirror partially blocked by tape was chosen. The main reason for this is that we can analyze the gap value calculated by the algorithm as well as the normal angles. Table 1 lists the results from some height differences between −1 cm to +1 cm that were applied after the height approximation was performed.
Table 1: Difference in results when shifting the approximated heights by certain amounts. Offset (cm) Gap (cm) ∆nx(◦) ∆ny(◦) -1.0 1.734 0.7026 1.0781 -0.2 1.842 0.2647 0.4135 -0.1 1.856 0.2301 0.3664 -0.05 1.862 0.2240 0.3430 0 1.869 0.2208 0.3198 +0.05 1.875 0.2277 0.3030 +0.1 1.883 0.2521 0.3392 +0.2 1.896 0.3094 0.4367 +1.0 2.005 0.7636 1.2854
4 RESULTS AND DISCUSSION -1 -0.5 0 0.5 1 Height offset (cm) 0.2 0.4 0.6 0.8 1 1.2 1.4 Angle (°)
Figure 21: Normal angle differences depending on the change in approxi-mated height.
4.4
Offset mirrors
To test the performance of calculating the flush between two surfaces we used a setup with two mirrors that were positioned offset from each other. In this case the holder shown in section 3.1.1 was placed with one side on each mirror. Like in the case with a partially blocked mirror, the gap between the two mirrors will introduce a gap in the data which is used to calculate the flush of these two mirrors. The properties of the holder is used to calculate the height difference between the two sides of the holder which is required to calculate the flush.
Figure 22: Calculated normal angles and heightmap from a test with two offset mirrors.
Once again, the results for a case with mirrors look very good. The value for the flush is not perfect, but like with the previous cases the value is within the resolution of the measuring instrument. While this result is good, it is hard to say from this alone how well the method will translate to a less perfect environment than mirrors. For example, it is harder to gather good data on a painted car body which is less reflective and also has an imperfect surface. Nevertheless, these results indicate that the method works well for determining both gap and flush on, or between, two reflective objects.
4.5
Concave mirror
4 RESULTS AND DISCUSSION
Unlike the cases with a flat mirror, the normal angles are not expected to be constant over the reflection in both directions. However, since the mirror is curved to be close to a cylindrical shape, we can expect that the normal angles in one direction are around 90◦ and a leaning plane in the other direction. This is almost true, as can be seen in figure 23.
Figure 23: Calculated normal angles and heightmap from a test with a con-cave mirror.
The average of the normal angles for the x-direction is extremely close to the expected 90◦, but the angles vary significantly more in this case compared to the flat mirror cases. The difference from smallest to largest normal angle is 1.01◦, which can at least in part be explained by the fact that the mirror was not curved to a perfectly cylindrical shape.
the measured mirror was calculated to be 0.31 m based on measurements with a ruler and the radius of curvature for the heightmap generated by the program was calculated to be 0.30 m. This difference in curvature means that the generated radius of curvature is smaller than the measured radius of curvature by around 2.4%.
A comparison between the center column of the heightmap and a circle with the curvature of the measured mirror can be seen in figure 24.
-2 0 2 4 6 8 10 y (cm) 16 16.5 17 Height (cm) Calculated heightmap Measured curvature from setup
Figure 24: The center column of the heightmap plotted together with part of a circle with the same curvature as the measured mirror.
The difference between the measured and calculated curvatures can be seen more clearly in figure 25. The largest difference between the two curves is 0.23 mm. -4 -2 0 2 4 6 8 10 12 y (cm) -0.025 -0.02 -0.015 -0.01 -0.005 0 0.005 Height difference (cm)
4 RESULTS AND DISCUSSION
When looking at the normal angles in the x-direction for this case it is clear that they are not as flat as the previous cases, which can be explained by two different things. The first of these is that it is possible that the height approximation is slightly off because of the shape of the object, and the second explanation is that the mirror plastic was not perfectly shaped.
The method of shaping the mirror has likely contributed to the bent shape of the normal angle graph, but even this difference in normal angles does not cause large errors in the heightmap as shown by the curvature comparisons. The 2.4% error in the curvature could be caused by a few different factors. One of these is a miscalculation in some part of the program, one is an error in the measurement of the curvature in the real mirror, and one is that the mirror may be shaped more like a parabola than a circle.
4.6
Flat mirror parallel to phone, handheld
For the first test with a handheld device, we once more look at the simplest possible case with a flat mirror parallel to the smartphone. Because of the nature of the smartphone being held in the hand, the mirror will not be perfectly parallel to the phone but it will be close enough to be able to compare with the non-handheld equivalent case.
Figure 26: Calculated normal angles and heightmap for a flat mirror parallel to a handheld smartphone without stabilization.
4 RESULTS AND DISCUSSION
Figure 27: Calculated normal angles and heightmap for a flat mirror parallel to a handheld smartphone with stabilization applied.
Comparing these results with the non-handheld equivalent case it is clear that this case has less smooth data, especially in the unstabilized case. Despite the increase in normal angle differences the resulting heightmap only has a height difference 0.04 mm larger than the case with a holder. This is a good indication that the algorithm is fairly stable for some disturbances in the measuring process.
4.7
Flat mirror with tape, handheld
angles is mostly caused by the patterns not being perfectly aligned with each other. For this we have normal differences of 0.27◦ and 0.45◦ for x and y, respectively. With this data the gap was calculated as 1.85 cm, which is the same as the measured width of the tape.
Figure 28: Calculated normal angles and heightmap for a partially blocked flat mirror parallel to a handheld smartphone without stabilization applied.
4 RESULTS AND DISCUSSION
are relatively small it has little effect on the gap value, which was the goal.
Figure 29: Calculated normal angles and heightmap for a partially blocked flat mirror parallel to a handheld smartphone with stabilization applied.
Like the previous case, these results show more instability and "waviness" than its stable counterpart but it still results in good values for the important properties. In this case the gap is calculated as almost exactly the same as the measured width of the tape, but since the tape was measured with a ruler it is hard to say if it is exactly correct or not. The values are once again well within the ruler’s resolution of 1 mm.
4.8
Gap and flush on a car
Despite these difficulties the program produced results that look very rea-sonable, as can be seen in figure 30. Using the method for determining the gap of a car door described in the theory section, the gap of this particular car door was found to be around 4.71 mm. The gap in the real car door varied depending on where it was measured, but in the region this test was performed the gap was measured to be around 4.5 mm.
Figure 30: Calculated normal angles and heightmap for a real car body. Each side of the data is part of a car door.
For the flush on this particular door it can be considered to be close to perfectly flush in the real world, meaning that we would expect a flush of 0 mm. The program instead calculated the flush to be 0.49 mm.
Considering that the gap varies slightly over the measuring region, a differ-ence of 0.21 mm between the (imperfect) measured gap and the calculated gap can be considered to be good since the difference is well within the 1 mm resolution of the measurement instrument. It is harder to confirm what the exact real flush should be because of the difficulty measuring it, but even here the result is well within a 1 mm error compared to what was expected.
4 RESULTS AND DISCUSSION
the camera had some trouble with capturing the finer patterns in a good way which results in a larger than normal amount of noise in the data since the processed images for the later patterns were not exactly representative of the actual reflected pattern. Luckily the earlier patterns are the most crucial for the overall results, and thus the results are decent despite this problem.
4.9
General discussion
Throughout this project there are some factors that have played a role in how the results look and how they could have been done better. One of these is more or less exclusive to the handheld tests, and that is the fact that the stabilization of the photos is done around the reflection of the smartphone itself. The main reason behind doing this is that it is very difficult to stabilize around the object itself, especially when it is as reflective as a mirror. The effect of this is that the patterns are lined up with each other very well, but if the smartphone moves relative to the object there is a chance that the different patterns are measuring slightly different parts of the object. This is a difficult thing to fix as long as the object is taking up most or all of the image and is very reflective. However, if the object is taking up a smaller part of the image or is not as reflective it is possible to stabilize around the object itself. That would lead to measuring the same part of the object for each pattern, rather than the same part of the phone reflection.
The biggest problem with the current stabilization is if there are large areas inside the patterns that are not reflective, as the part that is counted as all black would move within the data region. In general, assuming that the phone is held relatively stably, this is a small problem and does not cause major errors but it is something that could be improved.
that does not produce something close to a rectangular reflection. There are ways to get around this problem and consider every data point in the images, and such methods were implemented in this project as well. These methods were however eliminated because they required significantly more processing power for a small benefit in most cases. Naturally, if the area of use would require a more free-form reflection it is possible to use a free-form data region.
Through all of the tests shown in this report, there was some median filtering done on some of the data. After the normal angles were calculated for each iteration a median filter was applied in order to remove major noise and any small errors in the image processing. This leads to a result that is easier to analyze, but because the noise is generally quite small and not biased in any direction this has a very minor effect on the resulting heightmap.
5 CONCLUSIONS
5
Conclusions
In general, the results of this project are quite promising. The accuracy of many of the qualities measured is well within the measuring device’s resolu-tion to measure the real-world equivalents. However, a weakness is the low accuracy obtained when measuring the experimental setups. The limitation of using a ruler to measure the setups, as well as the difficulty of creating perfect setups limits how accurately we can determine the precision of the calculations. We obtained an accuracy of 1 mm, and likely even less in most cases, but due to the limitations we cannot guarantee a precision of any less than 1 mm. Despite this, many of the results indicate that the precision of the calculations are accurate to a degree significantly smaller than 1 mm since the calculated quantities are often within less than 0.5 mm of their measured equivalents.
Most of these tests were made with mirrors, which is an easy case to use deflectometry measurements and thus it is hard to say how well this method will transition to other materials. One example of other materials is the car body measurement, and in that case we had good accuracy of the measure-ments of both gap and flush despite the poor testing environment with a large amount of surrounding light. This is of course a good indication that the method can handle less than optimal testing situations, but it would be interesting to see an even wider variety of test objects to verify that the method can handle many different surfaces.
The method does require at least a decent testing environment to produce good results. If the object being measured is something not perfectly reflec-tive, such as a car door, and is being measured in direct sunlight it can be hard to produce any results at all. A decent environment is thus needed for this method if used on less than optimal surfaces.
A EXAMPLES OF TEST DATA
A
Examples of test data
Figure A.3: One of the test images for the concave mirror from section 4.5 before and after image processing.
REFERENCES
References
[1] Toyota, Quality Assurance - Toyota. [Online]. Available: https : / / blog.toyota.co.uk/factory- to- forecourt/quality- assurance (visited on 06/02/2020).
[2] J. Leopold, H. Günther, and R. Leopold, “New developments in fast 3D-surface quality control”, Measurement, vol. 33, no. 2, pp. 179–187, 2003, issn: 0263-2241. doi: 10.1016/S0263-2241(02)00056-8. [3] T. Bothe, W. Li, C. Von Kopylow, and W. Jüptner, High-resolution
3D shape measurement on specular surfaces by fringe reflection, 2004. doi: 10.1117/12.545987.
[4] F. Wang, Z. Zeng, L. Wang, Y. Fu, G. Jiang, and H. Zhang, “A fast 3D shape measurement method based on sinusoidal and triangular fringe projection”, Journal of Modern Optics, vol. 65, no. 2, pp. 136–144, 2018, issn: 0950-0340. doi: 10.1080/09500340.2017.1380853.
[5] P. Su, R. E. Parks, L. Wang, R. P. Angel, and J. H. Burge, “Soft-ware configurable optical test system: a computerized reverse Hart-mann test”, Applied optics, vol. 49, no. 23, p. 4404, 2010, issn: 1559-128X. doi: 10.1364/AO.49.004404.
[6] J. Novák, P. Novák, and A. Mikš, “Multi-step phase-shifting algo-rithms insensitive to linear phase shift errors”, Optics Communications, vol. 281, no. 21, pp. 5302–5309, 2008, issn: 0030-4018. doi: 10.1016/ j.optcom.2008.07.060.
[7] G. Butel, Analysis and new developments towards reliable and portable measurements in deflectometry, J. H. Burge, R. Liang, G. Smith, and P. Su, Eds., 2013.
[8] T. Luhmann, S. Robson, S. Kyle, and J. Boehm, Close-Range Pho-togrammetry and 3D Imaging, 2nd edition, T. ( Luhmann, S. Rob-son, S. Kyle, and J. Boehm, Eds., ser. De Gruyter textbook. Berlin: Walter de Gruyter GmbH & Co. KG, 2019, isbn: 3-11-030278-0. doi: 10.1515/9783110607253.
[9] E. Rosten and T. Drummond, Fusing points and lines for high perfor-mance tracking, 2005. doi: 10.1109/ICCV.2005.104.
