Thermal Impact of a Calibrated Stereo Camera Rig

(1)

Institutionen för systemteknik

Department of Electrical Engineering

Examensarbete

Thermal Impact of a Calibrated Stereo Camera Rig

Examensarbete utfört i Datorseende vid Tekniska högskolan vid Linköpings universitet

av Elin Andersson LiTH-ISY-EX–16/4980–SE

Linköping 2016

Department of Electrical Engineering Linköpings tekniska högskola

Linköpings universitet Linköpings universitet

(2)

(3)

Thermal Impact of a Calibrated Stereo Camera Rig

Examensarbete utfört i Datorseende

vid Tekniska högskolan vid Linköpings universitet

av

Elin Andersson LiTH-ISY-EX–16/4980–SE

Handledare: Andreas Robinson

isy_{, Linköpings Universitet}

Jimmy Jonsson

saab d_{ynamics AB}

Examinator: Klas Nordberg

isy, Linköpings Universitet

(4)

(5)

Avdelning, Institution Division, Department

Computer Vision Laboratory Department of Electrical Engineering SE-581 83 Linköping Datum Date 2016-06-09 Språk Language Svenska/Swedish Engelska/English Rapporttyp Report category Licentiatavhandling Examensarbete C-uppsats D-uppsats Övrig rapport

URL för elektronisk version http://www.ep.liu.se

ISBN — ISRN

LiTH-ISY-EX–16/4980–SE Serietitel och serienummer Title of series, numbering

ISSN —

Titel Title

Temperaturpåverkad av en kalibrerad stereokamerarigg Thermal Impact of a Calibrated Stereo Camera Rig

Författare Author

Elin Andersson

Sammanfattning Abstract

Measurements performed from stereo reconstruction can be obtained with a high accuracy with correct calibrated cameras. A stereo camera rig mounted in an outdoor environment is exposed to temperature changes, which has an impact of the calibration of the cameras.

The aim of the master thesis was to investigate the thermal impact of a calibrated stereo camera rig. This was performed by placing a stereo rig in a temperature chamber and collect data of a calibration board at different temperatures. Data was collected with two different cameras and lenses and used for calibration of the stereo camera rig for different scenarios. The obtained parameters were plotted and analyzed.

The result from the master thesis gives that the thermal variation has an impact of the accuracy of the calibrated stereo camera rig. A calibration obtained in one temperature can not be used for a different temperature without a degradation of the accuracy. The plotted parameters from the calibration had a high noise level due to problems with the calibration methods, and no visible trend from temperature changes could be seen.

Nyckelord

(6)

(7)

Sammanfattning

Mätningar gjorda från stereorekonstruktion kan erhållas med hög precision, för-utsatt att en korrekt kalibrering för kamerorna finns. En stereokamerarigg mon-terad i en utomhusmiljö påverkas av temperaturförändringar, vilket i sin tur på-verkar kalibreringen. Syftet med denna studie var att undersöka vilken påverkan temperaturförändringar har på kalibreringen av en stereorigg. Detta gjordes ge-nom att en stereorigg placerades i ett temperaturskåp och bilder på ett kalibre-ringsobjekt samlades upp under olika temperaturer. Data samlades in med två olika kameror och linser. Dessa data användes sedan för att kalibrera stereoriggen under olika förhållanden och resultatet plottades upp i grafer och analyserades. Slutsaten av studien visar att temperaturförändringar har en inverkan på en ka-librerad stereokamerarigg. En stereokamerarigg kaka-librerad i en viss temperatur kan inte användas för stereorekonstruktion i en annan temperatur utan att det kraftigt påverkar noggrannheten. Parameterna som plottades hade en hög brus-nivå och ingen modellerbar, temperaturberoende trend kunde utskiljas.

(8)

(9)

Abstract

Measurements performed from stereo reconstruction can be obtained with a high accuracy with correct calibrated cameras. A stereo camera rig mounted in an outdoor environment is exposed to temperature changes, which has an impact of the calibration of the cameras.

The aim of the master thesis was to investigate the thermal impact of a cali-brated stereo camera rig. This was performed by placing a stereo rig in a temper-ature chamber and collect data of a calibration board at different tempertemper-atures. Data was collected with two different cameras and lenses and used for calibra-tion of the stereo camera rig for different scenarios. The obtained parameters were plotted and analyzed.

The result from the master thesis gives that the thermal variation has an im-pact of the accuracy of the calibrated stereo camera rig. A calibration obtained in one temperature can not be used for a different temperature without a degra-dation of the accuracy. The plotted parameters from the calibration had a high noise level due to problems with the calibration methods, and no visible trend from temperature changes could be seen.

(10)

(11)

Acknowledgments

I would like to thank the people whom made this masters thesis happen, without your help and support it would have been a lot harder. First of all, I would like to thank my supervisors, Andreas Robinson and Jimmy Jonsson, and my examiner Klas Nordberg for always having time for questions and discussions. I would also like to thank saab for believing in me and giving me the opportunity to do my master thesis with them. A special thanks to Gösta Huldt at saab for interesting inputs and discussions that has helped pushing this thesis forward.

Last but not least I would like to thank my boyfriend Martin for the moral support and endless encouragement that has helped keeping my spirit up.

Linköping, June 2016 Elin Andersson

(12)

(13)

3.4.1 Processing of data . . . 20 3.5 Evaluation of results . . . 21 3.5.1 Evaluation scenarios . . . 21 3.5.2 Visible trends . . . 22 3.5.3 Re-projection error . . . 22 3.5.4 Triangulation . . . 22 4 Results 23 4.1 Outline . . . 23 4.1.1 Full calibration . . . 23 4.1.2 Partial calibration . . . 25 4.1.3 Triangulation . . . 25 4.2 Dataset A . . . 25 4.2.1 Full calibration . . . 25 4.3 Dataset B . . . 31 4.3.1 Full calibration . . . 31 4.3.2 Partial calibration . . . 36 4.3.3 Triangulation . . . 38 4.4 Dataset C . . . 39 4.4.1 Full calibration . . . 39 4.4.2 Partial calibration . . . 44 4.4.3 Triangulation . . . 46 4.5 Dataset D . . . 46 4.5.1 Full calibration . . . 47 4.5.2 Partial calibration . . . 52 4.5.3 Triangulation . . . 54 5 Discussion 55 5.1 Method evaluation . . . 55

5.1.1 Using a 3D calibration object . . . 55

5.1.2 Using a calibration board . . . 56

5.2 Evaluation of dataset . . . 57

5.2.1 Dataset A . . . 58

5.2.2 Dataset B . . . 58

5.2.3 Dataset C . . . 59

5.2.4 Dataset D . . . 60

6 Summary and Conclusion 61 6.1 Summary . . . 61 6.1.1 Future work . . . 62 6.2 Conclusion . . . 62 A Appendix A 67 B Appendix B 71 Bibliography 77

(15)

1

Introduction

This is a master thesis in computer vision done on behalf of saab Dynamics ab and the Department of Electrical Engineering (isy) at Linköping University. It aims to investigate the thermal impact of the calibration of a stereo camera rig, as a first step of an on-line calibration algorithm.

The target readers of this report are representatives at saab and the university, along with my fellow students and other interested. A certain basic knowledge of image processing and computer vision is assumed of the reader.

1.1 Motivation

When a timber truck enters a sawmill today, the size of the logs are measured manually. This volume measurement is inaccurate and a margin is needed when cutting the logs into planks. With this method there is a lot of waste of the logs and a profit lost for the sawmill. Moreover, there is usually a stock of logs at the sawmill, where there is of interest knowing its volume. The same method for volume estimate for the stock as for the logs is applied, which gives the same accuracy as previously. The stock should never run out of logs and since a modern sawmills operates around the clock more logs than necessary are kept due to uncertainty in the measurement methods.

saab dynamics wants to replace this manual measurement method with a stereo camera rig that can perform a stereo reconstruction of the logs. Placing the rig where the trucks are passing by gives an opportunity to automatically reconstruct the truck-loads of logs. From this reconstruction, volume measurements can be performed and reduce the error the human factor introduces. For the stereo

(16)

construction to perform accurate measurement it is necessary for the stereo cam-era rig to be calibrated.

1.2 Background

For this application the stereo camera rig is mounted outside for year-round us-age, in an environment with temperature changes and other external forces, such as vibrations in the ground and rig. These disturbances in the surrounding en-vironment have an impact of the calibration of the stereo camera rig since they cause the cameras to move. A calibration of a stereo camera gives the mathemati-cal relation of the mapping from the 3D world to the image plane, internal cam-era parameters, and the relationship between the two camcam-eras, external camcam-era parameters. The movement of the cameras relative each other, and temperature changes of the cameras which affects the camera components makes the calibra-tion less accurate which has an impact of the volume measurement.

Traditionally, a calibration of the camera rig is performed off-line with a calibra-tion object before usage of the system. The rig is then re-calibrated if something happens to the calibration. This is not possible in the intended application and therefore an on-line stereo camera calibration algorithm is needed to calibrate the system autonomously. As a first step in the process to develop such algo-rithm, this master thesis will investigate the thermal impact of a calibration on the stereo camera rig.

1.3 Purpose

This investigation is a first step of an on-line calibration algorithm that can han-dle changes in the environment where the camera rig is used. It is of interest to evaluate if and how the camera parameters are affected due to changes in temper-ature and if there are any parameters that are not affected. Furthermore it is of interest to know if any changes of parameters are correlated with each other.

1.4 Problem Description

The following question will be examined as an investigation of the thermal im-pact.

Is it possible to model the change of the internal camera parameters as a func-tion of temperature?

This includes examining if the changes are reproducible for several cycles of tem-perature variations.

(17)

1.5 Restrictions 3 stereo rig?

It is of interest to examine if the result of the investigation is exclusive for the cameras and lenses used in the particular stereo rig, or if the results can be used with different lenses or camera houses.

Which internal parameters are affected in the calibration?

Are there some parameters that are unaffected of the temperature changes and can be kept fixed in the calibration?

If there is a trend, does it need to be compensated for or can it be ignored in the calibration?

The purpose of this statement is to investigate if a trend found is big enough to cause trouble in the calibration, of if it can be ignored.

What are the behaviors of the parameters of the stereo rig due to thermal im-pact?

Do the parameters follow the expected heat expansion of the material of the stereo rig? Can the changes be modeled?

1.5 Restrictions

Calibration algorithm

A constraint on the calibration algorithm from saab was that the calibration should be performed with an additional geometric estimate of the calibration object, described in Section 2.5. Previous internal studies at saab has shown that geometric variation of the calibration board has an impact of the accuracy of the calibration. In the calibration the board is assumed to be planar, which is not always valid. An investigation on how if affects the calibration is out of scoop for this project.

1.6 Outline

This report is composed of six chapters and describes how the master thesis work was carried out. Chapter 2 covers the theory behind camera calibration and Chap-ter 3 describes the method used. ChapChap-ter 4 presents the results while ChapChap-ter 5 covers the discussion around the results and method. In Chapter 6 the conclu-sions are presented.

(18)

(19)

2

Theory

This chapter presents the theory used in this master thesis. To get understanding of the problem a short explanation of the camera model used is presented as well.

2.1 Related work

There is some previous work done in the area of thermal impact of a calibrated camera. In [1] a study investigates the impact of camera warm-up on the image acquisition. The study shows a thermal impact from the camera warm-up of the internal camera parameters for a single camera. The conclusion shows the need for further investigation of larger temperature variations than the internal from the camera warm-up. In [2] an analytical camera model for compensation of changes in internal and external camera parameters due to temperature changes is described. This is developed for a camera on a building site, and assumes the camera is mounted on a bearing structure which is under thermal impact. It is only investigated for a single camera and therefore it is on interest to investigate the impact of a stereo camera, where the relative motion between the cameras is affected as well.

2.2 Camera Model

The camera model used is a pinhole camera with a radial lens distortion correc-tion model. The projeccorrec-tion from homogeneous 3D coordinates x to homogeneous coordinates in the image plane y is mathematically described as the 3x4 camera matrix C. The camera matrix can be decomposed into internal camera

(20)

ters K and external camera parameters R and t [3].

y ∼ Cx= K (R|t) x (2.1)

Here K transforms the coordinates from the internal coordinate system of the camera to a pixel based coordinate system and R is a rotation matrix and t a 3D vector describing a translation. The geometric interpretation of the external cam-era parameters is found in Fig. 2.2. Theperspective division projects the 3D points

in the image plane by dividing the coordinates with the z-component which re-sults in Z = 1 and is applied before the K matrix.

2.2.1 Internal camera parameters

The internal camera parameters are defined by the 3x3 matrix K as

K=           fx γ cx 0 fy cy 0 0 1           (2.2)

and describes the transformation from C-normalized image coordinates to pixel coordinates. The parameters fxand fyare the focal length measured in pixels and cxand cyare the coordinates from the image origin to the principal point, which

is the point where the principal axis intersects with the image plane. The image origin is defined in the middle of the image. The parameter γ expresses the skew-ing of the pixels and is set to zero in this project for a simpler camera model. The assumption fx= fyindicates that the distance between two pixels are the same in

vertical and horizontal direction and is applied in the project. Fig. 2.1 shows the geometric interpretation of the focal length f and the principal point P , which is expressed in cxand cyand the camera center n. Y is the coordinate system in the

(21)

2.2 Camera Model 7

Figure 2.1:Geometric interpretation of the focal length and principal point.

2.2.2 External camera parameters

The external camera parameters gives the pose of the camera in the world coor-dinates system, a coordinate system in 3D space relative an origin. The pose is described as a rotation R and translation t in the world coordinate system to the origin of the camera centered coordinate system, a coordinate system where the origin is in the camera center. The rotations can be defined as Euler angles and are then represented in a 3x3 rotation matrix where the rotations are in relation to the world coordinate system, see Fig. 2.2 for geometric interpretation.

(a)Rotation and translation of

cam-era relative origin.

(b)Definition of Euler angles.

(22)

2.2.3 Lens distortion model

The pinhole camera model in Eq. 2.1 describes the mapping from a 3D point onto the image plane, assuming a linear projection model. A linear projection model is not always valid as many lenses produces a both radial and tangential distor-tion in the image. The distordistor-tion can be simplified to be only radial. The radial distortion is compensated for using a polynomial equation, with coefficients κ1 and κ2. It is applied after the perspective division but before the multiplication with the K matrix [3]. The compensation do not cancel out the distortion, but Eq. 2.3-2.5 reduces it by an approximation of the compensation needed.

xd= xc(1 + κ1rc2+ κ2rc4) (2.3)

yd= yc(1 + κ1rc2+ κ2rc4) (2.4)

rc2= x2c+ y2c (2.5)

Here xc and ycare the coordinates after perspective division and xd and yd are

the distortion compensated coordinates. This is true assuming the center of the radial distortion is coinciding with the principal point, an approximation applied in this project. The final image coordinates are obtained by multiplying xd and ydwith the K matrix.

          x y 1           = K           xd yd 1           (2.6)

2.3 Camera Calibration

In the following section the theory of camera calibration is treated.

2.3.1 Homography between two planes

A homography describes the mapping between two plane surfaces projected through a fixed point n. In the case of camera calibration between 3D points (X, Y , Z) placed on a plane, and 2D points (u, v) in the image plane through the camera center. Given a world coordinate system with origin placed in the calibration board, the board is found at Z = 0. This combined with Eq. (2.1) gives:

          u v 1           ∼_Kh_r₁ _r₂ _r₃ _ti                X Y 0 1                = Khr1 r2 t i           X Y 1           (2.7)

(23)

2.3 Camera Calibration 9 where r1, r2, r3are the columns in the rotation matrix. Since Z = 0 always is true due to the definition of the coordinate system, the 3D points x and the image points y will always be related by the homography H, which is determined up to a scale factor:

y ∼ Hx with H= Khr₁ r₂ ti (2.8)

Here the K matrix is related to the H matrix by denoting H = hh₁ h₂ h₃i, which together with Eq. (2.8) results in:

h

h₁ h₂ h₃i∼_Kh_r₁ _r₂ _ti_. _(2.9)

From the knowledge that r1and r2are orthonormal, the following constrains can be determined on the internal camera parameters, given one homography.

h₁TK−TK−1h₂ = 0 (2.10)

h₁TK−TK−1h₁ = h2TK −_T

K−1h₂ (2.11)

These are the fundamental constrains on the intrinsic parameters in K. Only two constrains can be obtained since r1and r2are the columns in a rotation matrix. H is estimated independently of these poses and each one of them contributes with two constrains on K. By using several poses and rotations of the calibration board the K matrix can be estimated, as each pose contributes with two constrains. The entire solution and a geometric interpretation can be found in [4].

2.3.2 Camera Calibration

Camera calibration is performed to obtain metric information from 2D images and can be done in several ways, for example with a known pattern printed on a plane surface or with a known calibration object. A common method is the usage of a known pattern on a plane surface, presented by Zhang [4], where the idea is to find the corresponding points between 3D points and image points, both found on a plane surface. The following steps are suggested by Zhang and will be covered in the following section.

(24)

• Collect images of the calibration plane in different poses relative to the cam-era

• Detect corners in the images

• Estimate internal and external camera parameters using a linear method • Refine all parameters using a non-linear method

Calibration pattern

A calibration pattern is a known pattern with a high contrast, as seen in Fig. 3.1. This makes it possible to detect theinterest points, the corners, with a high

accu-racy of the position and connect them to their corresponding 3D points. One of the most frequent used calibration pattern is the black and white checker-board pattern, attached to or directly printed on a plane surface.

In [5] a layout is developed for the checker pattern, with different codes in the squares. Each interest point is surrounded by four individual codes which al-lows the matching between the corresponding 2D and 3D points to be performed with high certainty. With a pre-knowledge of the pattern the scale factor can be determined from the calibration.

Image collection

The images of the calibration board are collected under different orientations by moving the calibration board. A minimum number of images l = 3, where l is the number of images used, are required if γ , 0 and l = 2 if γ = 0. This since the H matrix obtained from each image contributes with two constrains for the estimation of K in Eq. (2.9) [4]. Even though three is the minimum number of images the estimation becomes better if more images are used. The pose of the calibration board should vary as much as possible in both the rotation and orientation in each image.

Corner detection

Corner detection is used to obtain the pixel coordinates of the interest points in an image. A high precision of the corner detection improves the accuracy of the camera calibration as the 2D and 3D points are connected with higher accuracy. The detection of interest points can be performed in several ways, but with equivalent practical approach.

Feature points can be extracted by investigating the intensity in the image, as corners represent a variation in the gradient [6]. A small region around each pixel is examined to determine if the pixel contains a interest point. If a corner is found this coordinate can be used as a starting point for a second method with sub-pixel accuracy of the the corner detection.

(25)

2.4 Stereo Cameras 11

Parameter refinement

Camera calibration is performed to obtain an optimal K matrix, by minimizing

there-projection error [4]. The re-projection error is measured as the euclidean

distance of a re-projected 3D point and its corresponding detected point in the image plane. The minimization is a non-linear optimization of the difference of

k re-projected 3D points ˆy and their corresponding 2D points y in all images l,

where yi,jare the image coordinates of the 3D point j projected into image i. Here

ˆyis a function of the calibration parameters and 3D points; K is the matrix of the internal camera parameters, κ1, κ2 are the distortion parameters, Ri and ti are the rotation and translation for the camera corresponding to image i and xjis the j-th 3D point in the calibration board.

 = l X i=1 k X j=1 k_y_ij−_{ˆy(K, κ}₁_{, κ}₂_{, R}_i_{, t}_i_{, x}_j_)k2 _(2.12)

The optimization in Eq. (2.12) requires an initial guess of the parameters, which can be obtained from estimates of the different homographies. These estimates can be obtained from a Direct Linear Transform, (dlt) with the corresponding 2D and 3D points as input [7].

When the initial estimated camera parameters are obtained they are used to minimize in Eq. (2.12). To solve the non-linear optimization the Levenberg-Marquardt algorithm can be applied [8].

2.4 Stereo Cameras

Two cameras arranged to be facing the same scene from different viewpoints are calledstereo cameras and together they constitute a stereo rig.

2.4.1 Calibrated epipolar geometry

Calibrated epipolar geometry is the geometry of stereo vision with known K ma-trices of the cameras.

Given a pinhole camera with the camera matrix as described in Eq. (2.1), the coordinates of the projected points are measured in pixels and referred to as stan-dard image points, ys. With anormalized camera the internal parameters are set

to K1= K2= I, which is possible when the cameras used are pre-calibrated and K1and K2are known.

C0₁= (R1|t1), C 0

(26)

The normalized camera matrix C0transforms the 3D points toCamera-normalized

coordinates yn, i.e. the coordinates are given relative a camera centered coordinate

system, as

y_n∼_K−1_y_s_. _(2.14)

The 3D coordinate system used for the stereo cameras can be aligned with one of the cameras, since the epipolar geometry of a pair of stereo cameras is indepen-dent of a homography transformation in 3D space, [7].

C₁= (I|0), C₂= (R|t) (2.15)

where

R= R2RT1, t= RT1(t2−t1). (2.16)

A result of this gives that the epipolar geometry of two normalized cameras only depends on the rigid transform between them, the transformation from the 3D coordinate system for camera 1 to the 3D coordinate system for camera 2, (R|t). The relative pose of the cameras, the rigid transform between them, is referred to as thestereo base.

2.4.2 Calibration of a stereo camera rig

Calibration of a stereo camera rig consists of finding the optimal internal pa-rameters for the two cameras that constitutes the stereo pair, but also the rigid transform between them. The flowchart for a calibration of a stereo camera rig is seen in Fig. 2.3.

Figure 2.3: Flowchart of a calibration algorithm for a stereo rig, where the three first steps are described in Section 2.3.2 and performed individually for each camera, and the two last steps are performed for the stereo pair.

(27)

2.4 Stereo Cameras 13

Initial estimate of stereo rig

Assume 2xl images taken from l different positions of the stereo rig relative the calibration board, resulting in 2xl rotations and translations of the cameras. For given K matrices for the two cameras, the coordinate system for each stereo rig can be aligned with one of the cameras of the rig as described in Section 2.4.1. This gives l rigid transformations from the origin of the board relative each mas-ter camera, and l rigid transformations from the masmas-ter camera to the slave camera. The master camera is chosen as one of the cameras in the rig, placed in the ori-gin of themaster camera centered coordinate system, and the slave camera is the

other camera in the rig. Therig pose is the position of the stereo camera rig and is

defined as the pose of the master camera relative the origin found in the board. The stereo base (R|t) is assumed to be constant for all views, and is determined by a refinement step described in the section. An initial solution to the refinement can be chosen as any of the stereo bases, which are obtained from C2in Eq. (2.15). See Fig. 2.4 for visualization of coordinate system.

Figure 2.4:Camera centered coordinate system, where (Rb|rb) is the rotation and translation of slave camera relative master camera and constitutes the stereo rig, and (Rri|rri) is the rotation and translation of master camera i, relative origin of the calibration board.

The internal camera matrices K1 and K2 obtained in the calibration of the two single cameras described in Section 2.3.2 are used as initial values in the stereo calibration, together with the lens distortion parameters.

Refinement of parameters

From the initial estimate of the camera parameters and stereo rig a non-linear op-timization is applied to minimize the re-projection error [9]. It is minimized over

k individual rig poses, (Rri|tri), i = 1, ..., k and the common stereo base (Rb|tb),

(28)

in the optimization, in Eq. (2.17) shortened as Ω, are optimized over each 3D point xj. stereo = l X j=1 k X i=1 k_y_ij −_y(_ˆ Ω, x_j_)k2_. _(2.17)

From this optimization the refined stereo base and internal camera matrices for the stereo rig are obtained.

2.5 Geometry estimation of calibration board

The described calibration method assumes full knowledge of the geometry of the calibration board. It is assumed to be planar, but given the inaccuracy in the man-ufacturing of calibration board this is not exactly true. In [10] a novel approach to improve the final accuracy with respect to standard method is presented. The method suggested to improve the accuracy is by removing the geometric constraints on the calibration board and iteratively run the three step algorithm described below, as a final step of the camera calibration [11]. The algorithm

uses Bundle Adjustment (ba), a method to perform a non-linear optimization to

simultaneous refine several parameters to minimize the re-projection error. The parameters are the 3D coordinates, the internal and external camera parameters and the relative poses of the cameras.

1. Optimization of camera calibration parameters while the 3D points on the board are assumed accurate and kept fixed.

2. The ba technique is used to refine the camera poses, 3D points and stereo base while calibration parameters are assumed to be accurate and kept fixed.

3. Re-scaling of the 3D points of the board from known scale of the calibration board. This since ba affects the 3D points and the known scale of the board can be modified. To avoid this a re-scaling is applied. The new geometry of the board is used as input for step one.

The algorithm is iterated until a threshold of the re-projection error is reached or a maximum of number of iterations are made. In addition to the calibration parameters a 3D model of the calibration board is obtained from the calibration. The flowchart for the camera calibration with geometry estimation of calibration board is seen in Fig 2.5.

(29)

2.6 3D reconstruction 15

Figure 2.5:Flowchart of calibration algorithm for of stereo camera. All steps except the rightmost are the same as in Fig. 2.2.

2.6 3D reconstruction

3D reconstruction is a method to recover the 3D information of an object from the detected 2D points in the images. When the corresponding points in different images are known triangulation can be used.

2.6.1 Triangulation

For each corresponding point pair in the image plane, y1and y2, there is a projec-tion line L1and L2that intersects the corresponding image point and the camera center, n1 and n2[7]. Due to definition of corresponding points, these two lines will intersect at the 3D point x. Fig. 2.6 shows the geometry interpretation of triangulation.

Figure 2.6:Geometric interpretation of triangulation.

Due to noise in the image coordinates the two lines tend to not intersect exactly. There are several methods to solve this problem and themidpoint method is one of

(30)

them. It finds a 3D point even if the two lines do not intersect. Parametrization of the projection lines can be written as

x₁(s) = sx1+ (1 − s)n1, x2(t) = tx2+ (1 − t)n2 (2.18) where n1, n2are the two camera centers and x1, x2are two additional points that lie on the projection lines L1respectively L2. These points are obtained from the pseudo inverse of the camera matrices x1 = C+1y1 and x2 = C+2y2. The midpoint method finds the points that minimizes the following error function:

MP = kx1(s) − xk2+ kx2(t) − xk2. (2.19)

The two points found in the optimization of the error function are the two points in each projection line that are as close as possible to each other. The 3D point x is identified as the point that lies in the middle of the line that connects the two projected points, see Fig. 2.7.

Figure 2.7: Geometric interpretation of triangulation using the midpoint method.

(31)

3

Method

This chapter presents the method used for the project, which includes both the practical approach and implementation. It is of interest to investigate if and how the internal camera parameters are affected by temperature changes to see if it is possible to modulate the changes. It is also of interest to investigate the behavior of the stereo base due to thermal impact.

3.1 Hardware

The hardware used for collection of the data were two different types of lenses and cameras. The two cameras were the Basler piA2400-17gm GigE CCD sensor camera with 2456x2058 pixels and a pixel size of 3.45 µm, referred to as camera 1, and the Basler acA1300-60gc GigE CMOS sensor camera with 1280x1024 pixels and a pixel size of 5.3 µm, referred to as camera 2. Two cameras of each model were used to create the stereo rig. The latter camera type has a color sensor but in this master thesis all images were collected in monochrome format. The major difference between the cameras is the size of the camera house, where camera 1 is considerably larger than camera 2. This is believed to affect the behavior of the camera under thermal impact.

The two lenses were a Schneider Cinegon 1.4/8, referred to as lens 1, and a Cos-micar Pentax CCTV lens, referred to as lens 2, both with a focal length of 12.5 mm. They have the same properties but different construction due to different manufacturers, which might result in different behavior for temperature changes. In Table 3.1 are the different parameters of the lenses and cameras presented.

(32)

Table 3.1:Camera/lens specification

Camera/Lens Pixel size

[µm] Sensor size [pixels] Focal length [pixels] Focal length [mm] Camera 1/Lens 1 3.45 2456x2058 3623 12.5 Camera 2/Lens 1 3.45 1250x1024 2358 12.5 Camera 2/Lens 2 5.3 1250x1024 2358 12.5

3.2 Collection of data

The data needed for the investigation of the thermal impact of the stereo cam-eras was collected as a start of the thesis. It was collected by placing the camera in a temperature chamber while the calibration object was placed outside and images were taken in different temperatures. The calibration object was placed outside the camber to avoid modification due to thermal impact. The door was open when an image was taken. Two different methods were used for the data collection. In the first a fixed 3D calibration object was used and in the second a single calibration board which was moved between images was used. A more detailed explanation of the experimental setup and data collection is described in the following section.

3.2.1 Experimental setup

The placement of the two cameras which constitutes the stereo base, i.e. the relative pose between the cameras, was constant during the data collection, while the camera and lens used and the placement of the calibration object were varied. The cameras were placed in the temperature chamber and were affected by the temperature changes while the calibration object remained placed outside the chamber at a constant temperature. The calibration object was placed at such distance that the entire camerafield of view, fov, was covered and points could

be detected in the entire image.

The two cameras were attached to brackets fixed to an aluminum beam, which composes the stereo rig. The rig was attached to a plastic tripod so it could easily be moved. The tripod was placed in the temperature chamber, facing down with approximately 45◦angle relative to the ground. When switching between the cameras the brackets were left untouched.

(33)

3.2 Collection of data 19

3.2.2 Construction of 3D calibration object

The 3D calibration object used was constructed from three identical, square cali-bration boards. Each board had the same pattern which was generated according to Albarelli [5], and were placed mutually orthogonal. This calibration object was then placed on the ground for stability.

3.2.3 Data Collection

3D calibration object

The 3D calibration object was placed in front of the camera rig, outside of the tem-perature chamber. It was placed such that the entire fov of the two cameras was covered. The cameras were turned on and left to reach thermal equilibrium in room temperature. Once the cameras had reached the equilibrium, the chamber temperature was set to the minimum temperature of the interval of interest. An image was taken after approximately 20 minutes, giving the components of the cameras time to reach thermal equilibrium in the new surrounding temperature. Three different masks were applied on the image taken, resulting in three differ-ent images. The calibration object was never moved during the data collection and three images were used at each temperature. Each one of the images con-tains one of the boards, placed in different angles as seen in Fig. 3.1. From these three masked images a calibration was performed. This procedure was repeated at increasing temperature interval of 5◦C until the maximum was reached.

(a) (b) (c) (d)

Figure 3.1:Masked images with the original image to the far left.

Calibration board

In the second method for data collection, only one calibration board was used. The board was placed in positions equivalent to the boards in the 3D object, but since only one board was used there was no need for masking the images. Instead, two additional views were added for this method and in total five images were used for each temperature. In both views the board was placed right in front of the camera rig, but in one view the board was rotated 90◦. Similar to before the cameras were left to reach equilibrium between changes in the temperature.

(34)

3.2.4 Geometric estimate of calibration board

As a part of the calibration of the datasets collected with one calibration board, a geometric estimate of the board was acquired and used as a ground truth, gt, as described in Section 2.5. The camera parameters are optimized by minimizing the re-projection error of the 3D points, and for the best result the 3D points should be as accurate as possible. The ground truth is obtained as a calibration of the stereo cameras with geometric estimate of the calibration board. When the images of the calibration board were collected it was ensured that each corner of the calibration pattern of the board was detected several times in both cameras, and that the entire fov of both cameras was covered.

3.3 Datasets

Table 3.2 shows the different datasets used in the master thesis. The temperature interval differs in the different datasets, which is due to practical issues. The step size in the temperature change between each set of images was the same for all datasets, set to 5◦

C.

Table 3.2:Datasets collected and analyzed Dataset 3D calibration object/ Calibra-tion board Camera 1/ Camera 2 Lens 1/ Lens 2 Temp. in-terval◦C A Object 1 1 [-20, 30] B Board 1 1 [0, 50] C Board 2 1 [-5, 40] D Board 2 2 [-5, 40]

3.4 Implementation

3.4.1 Processing of data

The calibration was performed differently for Dataset A and Dataset B, C, D. For the latter three a geometric estimate of the calibration board was used for the calibration, while it was not for Dataset A.

Calibration without geometric estimate of the calibration board

This calibration does not take a geometric estimate of a calibration as an input, but returns one as an output, as described in Section 2.5. The output is applied in Dataset B, C and D. The calibration method described in this section is applied

(35)

3.5 Evaluation of results 21 for Dataset A and for the geometric estimation of the calibration board. The re-sults from this calibration are the internal camera parameters with the distortion coefficients κ1, κ2and the relative pose of the stereo rig together with the 3D co-ordinates of the modeled calibration board. The 3D coco-ordinates are expressed as the deviation from a perfect board in the coordinate system of the plane, de-scribed in Section 2.3.1, in millimeters.

Calibration with a geometric estimate of the calibration board

The calibration with a geometric estimate of the calibration board is performed for Dataset B, C, and D as described in Section 2.4.2. In the initial estimate of the homographies, the estimated coordinates of the calibration board are used as 3D coordinates, but with Z = 0. This since the homographies are estimated between two plane surfaces, according to Eq. (2.7). From these, the initial estimate for Eq. (2.12) can be found. In the non-linear optimization in Fig. 2.3 the re-projection error of the 3D points is minimized, as described in Eq. (2.17), the coordinates of the geometric estimated board are used, with Z , 0.

3.5 Evaluation of results

The results of the data collection are investigated as described in the following section.

3.5.1 Evaluation scenarios

The results are investigated for three different scenarios, described below. Full calibration refers to a calibration which returns K, R, t, κ1, κ2 for Dataset B, C and D. For Dataset A the 3D positions of the board are returned as well for the full calibration.

• Scenario 1: Full calibration with images taken with the cameras in a tem-perature interval. This scenario gives the variation of the calibration param-eters as a function of temperature.

• Scenario 2: Full calibration of a set of images taken at a reference tempera-ture, where the obtained calibration parameters are used in a temperature varying dataset. This gives the thermal impact of the re-projection error and the triangulation error and can be seen as if the sawmill application was calibrated at one temperature and then used at different temperatures. • Scenario 3: Calibration of external camera parameters from Scenario 2. In this scenario a re-calibration of the set-up from Scenario 2 is performed, but with the internal camera parameters and distortion coefficients kept fixed.

(36)

The result of this is updated coordinates of the stereo base. This investi-gates if a re-calibration of the external camera parameters and the relative pose of the stereo rig is sufficient to obtain a triangulation and re-projection error similar to Scenario 1. See below for description of re-projection and triangulation error.

3.5.2 Visible trends

Plots of the camera parameters as functions of temperature and time are used to evaluate if any visible trends can be found. The parameters plotted as functions of time are from the datasets taken at a constant room temperature, and are used as reference measurements of noise levels in the parameters.

3.5.3 Re-projection error

The re-projection error is used to evaluate the accuracy of the calibrated cam-era matrix. This measure, described in Eq. 2.17, is evaluated for three different scenarios, described in Section 3.5.1.

3.5.4 Triangulation

Triangulation is performed with the obtained camera matrices C1, C2, using the method described in Section 2.6.1. The accuracy in the triangulation is evalu-ated for each of the scenarios described in Section 3.5.1. This by calculating the differences between the coordinates of the triangulated board and the estimated board. To reduce the data the standard deviation, std, of the differences between triangulated and estimated coordinates separately in x-, y- and z-direction is cal-culated for each temperature.

The boards are expressed in different coordinate system, and need to be aligned first. For this purpose Procrustes analysis is used which adjusts the rotation and translation of the triangulated board [12].

(37)

4

Results

In this chapter the results of the thesis are presented for each dataset.

4.1 Outline

Three results are presented for the datasets, except for Dataset A where only one result is presented. More on this is found in Section 5.1.1. In Dataset A camera 1, lens 1 and the calibration object were used, and in Dataset B the object was exchanged to a single calibration board. In Dataset C camera 2, lens 1 and a calibration board were used and in Dataset D lens 1 was changed to lens 2, see Table 3.2. In the following section the different results are described.

4.1.1 Full calibration

In the section calledFull calibration, the parameters from a full calibration in

Scenario 1 is presented for the master and slave camera. See Section 2.4.2 for a definition of master and slave camera. The parameters are the focal length, the principal point and the distortion parameters. The first is plotted as a function of the different temperatures and by sample in the reference data with constant temperature of 23◦

C. The principal point is plotted as the movement over the image plane with the origin placed in the middle of the image. The temperature varying data is connected with a line between two subsequent temperatures. The distortion parameters κ1 and κ2 are not presented explicitly, instead the corre-sponding distortion function D(rc) = 1 + κ1rc2+ κ2rc4, described in Section 2.5 is

plotted for the resulting values of κ1and κ2. Here rcis the radius from distortion

center, in this case the principal point, to the coordinate for which the distortion

(38)

is calculated. The maximum value of rcis calculated as:

rmax =

w

2 + cx

f (4.1)

where w is the width of the sensor in pixels and cx is the pixel position of the

principal point in x-direction. The longest distance from the principal point to an edge is found in at the right or left edge of the sensor, by moving in the x-direction since the sensor is rectangular and wider in this x-direction. This is the reason for using the sensor width and principal point in the x-direction.

The lenses in this project produce a small radial distortion, and for a better under-standing of the impact of the variations in the distortion function the variation of the maximum and minimum distortion is presented in pixels and millimeters. The variation Dv is calculated as the difference in the distortion at rmax for the

two temperatures which produces the most and least distortion, here called T1 and T2. Using the value of rmax gives the distortion of the pixels at the edge of

the image. These pixels are the most distorted, except for the pixels in the cor-ners which can be seen as an extreme value. The variation in the distortion is calculated as:

Dv= (D(rmax, T1) − D(rmax, T2)) ∗ f ∗ rmax (4.2)

Here rmaxis the maximum value the x-coordinate can have, measured in mm and f is the focal length in pixels. This gives the result in pixels, and to translate it to

millimeter the result is multiplied with the pixel size, found in Section 3.1. The stereo base, i.e. the rotation and translation of the slave camera relative the master camera, is also presented as a function of temperature or samples. The coordinates are expressed in a master camera centered coordinate system, according to Fig. 4.1. Here the x-axis is the depth in front of the camera, the y-axis is pointing to the right and the z-axis is pointing down.

(39)

4.2 Dataset A 25

(a)Coordinate system. (b)Euler angles.

Figure 4.1:Master camera centered coordinate system.

4.1.2 Partial calibration

The parameters for the stereo base are plotted in the section Partial calibration

which are obtained from Scenario 3, the external calibration, as described in Sec-tion 3.5.1. The re-projecSec-tion errors for the three scenarios are demonstrated as an evaluation of the success of the calibration.

4.1.3 Triangulation

A triangulation is performed to evaluate the camera matrices from Scenario 1 and 3. The triangulated board points are compared with the 3D points of the gt cali-bration board as described in Section 3.5.4 and the std of the difference between coordinates is plotted.

4.2 Dataset A

In Dataset A camera 1 and lens 1 were used, and the data was collected in the temperature interval [−20, 30]◦C, with the 3D calibration object.

4.2.1 Full calibration

The calibrated parameters from Scenario 1 are presented, divided in internal pa-rameters and stereo base. The focal length in pixels for Dataset A is 3623 pixels.

(40)

Internal camera parameters

For Dataset A the focal length in Fig 4.2 has an increasing trend for the slave camera and a decreasing trend for the master camera, both of approximately two pixels. In the constant temperature, here used as a reference, the focal lengths are relatively constant but approximately 15 pixels longer than for the varying temperature. The variation of the principal point in Fig. 4.5 is bigger in the x-direction than in y-direction for the temperature varying data, where the vari-ation is five pixels in the x-direction and less than three pixels in the y direction. The temperatures are connected through lines and the movement appears as ran-dom. The principal points in Fig. 4.5 are clustered at different locations, where the centroids for the reference data are found at [37, −17] and [34, −4] and the temperature varying are found at [46, −25] and [32, −12]. The distortion func-tions for the master and the slave camera in Fig. 4.3 and 4.4 are similar to each other but with a higher noise level in the temperature varying data. The dis-tortion variance Dv, described in Section 4.1.1, is 1.52 pixels which is equal to

0.0052 mm for the master camera.

-20 -10 0 10 20 30 Degrees Celsius 3625 3630 3635 3640 3645 3650 Pixels Focal length Master camera Slave camera

(a)Varying temperature.

0 0.2 0.4 0.6 0.8 1 Samples 3625 3630 3635 3640 3645 3650 Pixels Focal length Master camera Slave camera

(b) Constant room temperature,

23◦C.

Figure 4.2: Focal length for the master and the slave cameras, in (a) as a function of temperature and in (b) for different samples. The trend lines are obtained from Matlab robust line fitting function.

(41)

4.2 Dataset A 27 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 r 0.985 0.99 0.995 1 D(r) -18.8 -14.3 -9.5 -4.7 0.9 5.0 10.0 14.9 19.9 24.6

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 r 0.985 0.99 0.995 1 D(r) 1 2 3 4 5 6 7 8 9 10

23◦C.

Figure 4.3:Distortion parameters k1and k2for the master camera plotted as

D(r) = 1 + k1r2c + k2rc4. 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 r 0.985 0.99 0.995 1 D(r) -18.8 -14.3 -9.5 -4.7 0.9 5.0 10.0 14.9 19.9 24.6

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 r 0.985 0.99 0.995 1 D(r) 1 2 3 4 5 6 7 8 9 10

23◦C.

Figure 4.4:Distortion parameters k1and k2for the slave camera plotted as a

(42)

30 35 40 45 50 Pixels -30 -25 -20 -15 -10 -5 0 Pixels Principal Point Master camera Slave camera

23◦C.

Figure 4.5: Principal point positions for the master and slave camera, ex-pressed in pixels and plotted in relation to the origin. For the temperature varying data the increasing temperatures are connected through lines.

Stereo base

The x-coordinates for the stereo base has a constant value with a noise level less than 1 mm, except for three outliers as seen in Fig. 4.6. The y-coordinates has a constant value and a noise level less than 0.3 mm for both data, but approx-imately 1.4 mm apart. For the z-coordinate the noise level is twice as high at 0.12 mm for the temperature varying data than for the reference data, and no temperature dependence can be seen in the temperature varying data.

For the roll and pitch angle in Fig. 4.7 the noise levels for the temperature vary-ing data are in the same range as for the reference and no behavior related to temperature changes is seen. The yaw angle is wider and with a higher noise level for the temperature varying data.

(43)

4.2 Dataset A 29 -20 -10 0 10 20 30 Degrees Celcius 31.5 32 32.5 33 33.5 34 34.5 35 mm x position (a)x position 0 0.2 0.4 0.6 0.8 1 Samples 31.5 32 32.5 33 33.5 34 34.5 35 mm x position (b)x position -5 0 5 10 15 20 25 30 35 Degrees Celcius 328.2 328.4 328.6 328.8 329 329.2 329.4 329.6 329.8 mm y position (c)y position 0 0.2 0.4 0.6 0.8 1 Samples 328.2 328.4 328.6 328.8 329 329.2 329.4 329.6 329.8 mm y position (d)y position -20 -10 0 10 20 30 Degrees Celcius -1.08 -1.06 -1.04 -1.02 -1 -0.98 -0.96 -0.94 -0.92 mm z position (e)z position 0 0.2 0.4 0.6 0.8 1 Samples -1.08 -1.06 -1.04 -1.02 -1 -0.98 -0.96 -0.94 -0.92 mm z position (f)z position

Figure 4.6:Position of the slave camera relative the master camera, in camera centered coordinate system. Temperature varying to the left in a,c,e and constant to the right in b,d,f.

(44)

-20 -10 0 10 20 30 Degrees Celcius -10 -9.5 -9 -8.5 -8 -7.5 Degrees ×10-4 Roll (a)Roll 0 0.2 0.4 0.6 0.8 1 Samples -1.065 -1.06 -1.055 -1.05 -1.045 Degrees ×10-3 Roll (b)Roll -20 -10 0 10 20 30 Degrees Celcius 6.158 6.1585 6.159 6.1595 6.16 6.1605 6.161 Degrees Yaw (c)Yaw 0 0.2 0.4 0.6 0.8 1 Samples 6.158 6.1585 6.159 6.1595 6.16 6.1605 6.161 Degrees Yaw (d)Yaw -20 -10 0 10 20 30 Degrees Celcius 2.4 2.5 2.6 2.7 2.8 2.9 3 3.1 Degrees ×10-3 Pitch (e)Pitch 0 0.2 0.4 0.6 0.8 1 Samples 2.4 2.5 2.6 2.7 2.8 2.9 3 3.1 Degrees ×10-3 Pitch (f)Pitch

Figure 4.7:Rotation of the slave camera relative the master camera. Temper-ature varying to the left in a,c,e and constant to the right in b,d,f.

(45)

4.3 Dataset B 31

4.3 Dataset B

In Dataset B camera 1 and lens 1 were used to collect data in the temperature in-terval [0, 50]◦C with one calibration board that has been geometrically estimated before usage.

4.3.1 Full calibration

This section presents the camera parameters from a full calibration. They are divided in internal parameters and stereo base. The focal length in pixels for Dataset B is 3623 pixels.

A weak decreasing trend is seen for the focal lengths in both the temperature varying data and its reference, seen in Fig. 4.8. The noise levels are higher for the temperature varying data with a variation of eight pixels while it is three pixels for the reference data. The variation of the principal point in Fig. 4.11 is five pixels in both y- and x-direction for both the master and slave camera, for both the datasets. The movement between different temperatures is random, as the lines connecting the temperature samples does not follow any pattern. The distortion in the image, seen in Fig. 4.9 and 4.10 appears relatively unaffected by the temperature variation. The distortion variance Dv, described in Section 4.1.1,

is 0.7080 pixels which translates to 0.0024 mm for the master camera.

0 10 20 30 40 50 Degrees Celsius 3658 3660 3662 3664 3666 3668 3670 3672 3674 Pixels Focal length Master camera Slave camera

0 0.2 0.4 0.6 0.8 1 Samples 3658 3660 3662 3664 3666 3668 3670 3672 3674 Pixels Focal length Master camera Slave camera

23◦C.

Figure 4.8: Focal length for the master and the slave camera, in (a) as a function of temperature and in (b) of samples. The trend lines are obtained from Matlab robust line fitting function.

(46)

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 r 0.985 0.99 0.995 1 D(r) 1.3 5.9 10.9 15.9 20.6 25.8 30.5 35.5 40.3 45.2 50.5

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 r 0.985 0.99 0.995 1 D(r) 1 2 3 4 5 6 7 8 9 10

23◦C.

Figure 4.9:Distortion parameters k1and k2for the master camera plotted as

D(r) = 1 + k1rc2+ k2r4c. 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 r 0.985 0.99 0.995 1 D(r) 1.3 5.9 10.9 15.9 20.6 25.8 30.5 35.5 40.3 45.2 50.5

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 r 0.985 0.99 0.995 1 D(r) 1 2 3 4 5 6 7 8 9 10

23◦C.

Figure 4.10:Distortion parameters k1and k2for the slave camera plotted as

(47)

4.3 Dataset B 33 0 5 10 15 20 25 30 35 Pixels -25 -20 -15 -10 -5 0 Pixels Principal Point Master camera Slave camera

0 5 10 15 20 25 30 35 Pixels -25 -20 -15 -10 -5 0 Pixels Principal Point Master camera Slave camera

23◦C.

Stereo base

The coordinates of the stereo base in in Fig. 4.12 has approximately twice as a high noise level for the temperature varying data than for the reference. A weak increasing trend is seen for the y-coordinate. The behavior of the angles in Fig. 4.13 are the same for the temperature varying data as for its reference with equivalent noise level. No dependence of temperature variation is found.

(48)

0 10 20 30 40 50 Degrees Celcius 33.5 34 34.5 35 mm x position (a)x position 0 0.2 0.4 0.6 0.8 1 Samples 33.5 34 34.5 35 mm x position (b)x position 0 10 20 30 40 50 Degrees Celcius 331.6 331.7 331.8 331.9 332 332.1 332.2 mm y position (c)y position 0 0.2 0.4 0.6 0.8 1 Samples 331.6 331.7 331.8 331.9 332 332.1 332.2 mm y position (d)y position 0 10 20 30 40 50 Degrees Celcius -0.7 -0.65 -0.6 -0.55 -0.5 -0.45 -0.4 -0.35 mm z position (e)z position 0 0.2 0.4 0.6 0.8 1 Samples -0.7 -0.65 -0.6 -0.55 -0.5 -0.45 -0.4 -0.35 mm z position (f)z position

Figure 4.12: Position of the slave camera relative to the master camera, in camera centered coordinate system. Temperature varying to the left in a,c,e and constant to the right in b,d,f.

(49)

4.3 Dataset B 35 0 10 20 30 40 50 Degrees Celcius -10.5 -10 -9.5 -9 -8.5 -8 -7.5 -7 Degrees ×10-4 Roll (a)Roll 0 0.2 0.4 0.6 0.8 1 Samples -10.5 -10 -9.5 -9 -8.5 -8 -7.5 -7 Degrees ×10-4 Roll (b)Roll 0 10 20 30 40 50 Degrees Celcius 6.1552 6.1553 6.1554 6.1555 6.1556 6.1557 6.1558 6.1559 6.156 6.1561 6.1562 Degrees Yaw (c)Yaw 0 0.2 0.4 0.6 0.8 1 Samples 6.1552 6.1553 6.1554 6.1555 6.1556 6.1557 6.1558 6.1559 6.156 6.1561 6.1562 Degrees Yaw (d)Yaw 0 10 20 30 40 50 Degrees Celcius 2.4 2.5 2.6 2.7 2.8 2.9 3 Degrees ×10-3 Pitch (e)Pitch 0 0.2 0.4 0.6 0.8 1 Samples 2.4 2.5 2.6 2.7 2.8 2.9 3 Degrees ×10-3 Pitch (f)Pitch

Figure 4.13:Rotation of the slave camera relative to the master camera. Tem-perature varying to the left and constant to the right.

(50)

4.3.2 Partial calibration

This section presents the results from Scenario 3 with the parameters from Sce-nario 1 as a reference where the sceSce-narios are described in Section 3.5.1.

Fig. 4.14 shows the coordinates for the stereo base of Scenario 3, which follows the same pattern as for Scenario 1, but with an offset that varies from 0.2-0.8 mm. The angles do not have this behavior, as can be seen in Fig. 4.15. An increasing trend as a function of the temperature can be distinguished for the roll and yaw angle in (a) and (b), and a decreasing trend for the pitch angle.

The re-projection error is 10 times greater for Scenario 2 than for Scenario 1 and 3, shown in Fig. 4.16. 0 10 20 30 40 50 60 Degrees Celcius 33.2 33.4 33.6 33.8 34 34.2 34.4 34.6 34.8 35 mm x position Scenario 1 Scenario 3 (a)x position 0 10 20 30 40 50 60 Degrees Celcius 331.6 331.7 331.8 331.9 332 332.1 332.2 mm y position Scenario 1 Scenario 3 (b)y position 0 10 20 30 40 50 60 Degrees Celcius -0.75 -0.7 -0.65 -0.6 -0.55 -0.5 -0.45 -0.4 -0.35 mm z position Scenario 1 Scenario 3 (c)z position

Figure 4.14: The stereo base, i.e. the slave camera position relative to the master camera for original coordinates, scenario 1 and re-calibrated, Sce-nario 3. They are described in master camera centered coordinate system.

(51)

4.3 Dataset B 37 0 10 20 30 40 50 60 Degrees Celcius -11 -10.5 -10 -9.5 -9 -8.5 -8 -7.5 -7 Degrees ×10-4 Roll Scenario 1 Scenario 3 (a)Roll 0 10 20 30 40 50 60 Degrees Celcius 6.1552 6.1553 6.1554 6.1555 6.1556 6.1557 6.1558 6.1559 6.156 Degrees Yaw Scenario 1 Scenario 3 (b)Yaw 0 10 20 30 40 50 60 Degrees Celcius 2.4 2.5 2.6 2.7 2.8 2.9 3 Degrees ×10-3 Pitch Scenario 1 Scenario 3 (c)Pitch

Figure 4.15: Rotation of the slave camera relative the master camera with original coordinates, Scenario 1 and re-calibrated, Scenario 3.

0 10 20 30 40 50 60 Degrees celsius 0 0.5 1 1.5 2 2.5 3 Pixels Scenario 1 Scenario 2 Scenario 3

(a)Re-projection error for all

scenar-ios. 0 10 20 30 40 50 Degrees celsius 0.1 0.11 0.12 0.13 0.14 0.15 0.16 0.17 0.18 0.19 0.2 Pixels Scenario 1 Scenario 3

(b)Re-projection error zoomed in for

Scenario 1 and 3.

(52)

4.3.3 Triangulation

The std of the differences between the triangulated coordinates and gt coordi-nates of a board is shown in Fig. 4.17. They are presented for each temperature and scenario. The difference in depth, i.e. the x-coordinate, between the scenarios are close to constant for each temperature, but the coordinates has a high std of 0.15-0.3 mm. For y- and z-coordinates the difference in std are small at 0.01 mm for Scenario 1 and 3, while it is up to 0.3 mm larger for Scenario 2. In scenario 2 it is higher for the extreme temperatures.

0 10 20 30 40 50 Degrees Celsius 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 mm Scenario 1 Scenario 2 Scenario 2

(a)Difference in x-direction.

0 10 20 30 40 50 Degrees Celsius 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 mm Scenario 1 Scenario 2 Scenario 2 (b)Difference in y-direction. 0 10 20 30 40 50 Degrees Celsius 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 mm Scenario 1 Scenario 2 Scenario 2 (c)Difference in z-direction.

Figure 4.17:The std of the difference between the triangulated board coor-dinates and the modeled reference board coorcoor-dinates in x,y,z direction. Tri-angulation is performed with the camera matrices obtained from the three different scenarios.

(53)

4.4 Dataset C 39

4.4 Dataset C

In Dataset C camera 2 with lens 1 were used and the data was collected in the temperature interval [−5, 40]◦C using one calibration board.

4.4.1 Full calibration

This section presents the camera parameters from a full calibration, divided in internal parameters and stereo base. The focal length in pixels for Dataset C is 2358 pixels.

Seen in Fig. 4.18 is the focal lengths for Dataset C. The robust fitted line is in-creasing for the temperature varying data, while it is constant for the reference. The visible trends for the focal lengths are both within the noise level, where the increase is approximately two pixels over 45◦C. The variation of the principal point is approximately five pixels for the x-coordinates and two pixels for the y-coordinates, Fig. 4.21, for both the master and slave camera. The movement of the principal point appears random as the lines connection the samples do not follow any pattern. The distortion function appears to be the same for the varying data and its reference with a minimal visible variations. The distortion variance Dvis found at 1.23 pixels which is equivalent to 0.0065 mm.

0 10 20 30 40 Degrees Celsius 2380 2382 2384 2386 2388 2390 2392 2394 2396 Pixels Focal length Master camera Slave camera

0 0.2 0.4 0.6 0.8 1 Samples 2380 2382 2384 2386 2388 2390 2392 2394 2396 Pixels Focal length Master camera Slave camera

23◦C.

Figure 4.18:Focal length for the master and slave camera, in (a) as a function of temperature and in (b) for different samples. The trend lines are obtained from Matlab robust line fitting function.

(54)

0 0.05 0.1 0.15 0.2 0.25 0.3 r 0.99 0.991 0.992 0.993 0.994 0.995 0.996 0.997 0.998 0.999 1 D(r) -5 0.1 5.0 10.0 15.1 20.4 25.4 30.5 35.8 40.9

0 0.05 0.1 0.15 0.2 0.25 0.3 r 0.99 0.991 0.992 0.993 0.994 0.995 0.996 0.997 0.998 0.999 1 D(r) 1 2 3 4 5 6 7 8 9 10

23◦C.

Figure 4.19: Distortion parameters k1and k2for the master camera plotted as D(r) = 1 + k1rc2+ k2rc4. 0 0.05 0.1 0.15 0.2 0.25 0.3 r 0.99 0.991 0.992 0.993 0.994 0.995 0.996 0.997 0.998 0.999 1 D(r) -5 0.1 5.0 10.0 15.1 20.4 25.4 30.5 35.8 40.9

0 0.05 0.1 0.15 0.2 0.25 0.3 r 0.99 0.991 0.992 0.993 0.994 0.995 0.996 0.997 0.998 0.999 1 D(r) 1 2 3 4 5 6 7 8 9 10

23◦C.

Figure 4.20:Distortion parameters k1and k2for the slave camera plotted as

(55)

4.4 Dataset C 41 0 5 10 15 20 Pixels -30 -20 -10 0 10 20 30 40 50 Pixels Principal Point Master camera Slave camera

0 5 10 15 20 Pixels -30 -20 -10 0 10 20 30 40 50 Pixels Principal Point Master camera Slave camera

23◦C.

Stereo base

The coordinates for the stereo rig are presented in Fig. 4.22. The noise level of the x-and z-coordinates of the stereo base is twice as great in the temperature varying data as the reference. For the y-coordinates a slight increasing trend can be seen. In Fig. 4.23 the angles for the stereo base in visualized. The three angles has the same behavior as their respective reference data, with similar noise level and constant behavior and appears to be unaffected of the temperature changes.

(56)

0 10 20 30 40 Degrees Celcius 31 31.2 31.4 31.6 31.8 32 32.2 32.4 32.6 32.8 33 mm x position (a)x position 0 0.2 0.4 0.6 0.8 1 Samples 31 31.2 31.4 31.6 31.8 32 32.2 32.4 32.6 32.8 33 mm x position (b)x position 0 10 20 30 40 Degrees Celcius 333.3 333.4 333.5 333.6 333.7 333.8 333.9 334 mm y position (c)y position 0 0.2 0.4 0.6 0.8 1 Samples 333.3 333.4 333.5 333.6 333.7 333.8 333.9 334 mm y position (d)y position 0 10 20 30 40 Degrees Celcius -1.35 -1.3 -1.25 -1.2 -1.15 -1.1 -1.05 -1 -0.95 -0.9 mm z position (e)z position 0 0.2 0.4 0.6 0.8 1 Samples -1.35 -1.3 -1.25 -1.2 -1.15 -1.1 -1.05 -1 -0.95 -0.9 mm z position (f)z position

Figure 4.22:Position of the slave camera relative the master camera, in cam-era centered coordinate system. Tempcam-erature varying to the left in a,c,e and constant to the right in b,d,f.

(57)

4.4 Dataset C 43 0 10 20 30 40 Degrees Celcius -9 -8 -7 -6 -5 -4 -3 -2 Degrees ×10-4 Roll (a)Roll 0 0.2 0.4 0.6 0.8 1 Samples -9 -8 -7 -6 -5 -4 -3 -2 Degrees ×10-4 Roll (b)Roll 0 10 20 30 40 Degrees Celcius 6.1506 6.1508 6.151 6.1512 6.1514 6.1516 6.1518 Degrees Yaw (c)Yaw 0 0.2 0.4 0.6 0.8 1 Samples 6.1506 6.1508 6.151 6.1512 6.1514 6.1516 6.1518 Degrees Yaw (d)Yaw 0 10 20 30 40 Degrees Celcius 8 8.5 9 9.5 10 10.5 Degrees ×10-3 Pitch (e)Pitch 0 0.2 0.4 0.6 0.8 1 Samples 8 8.5 9 9.5 10 10.5 Degrees ×10-3 Pitch (f)Pitch

Figure 4.23:Rotation of the slave camera relative the master camera. Tem-perature varying to the left and constant to the right.

Thermal Impact of a Calibrated Stereo Camera Rig

Institutionen för systemteknik

Department of Electrical Engineering

Examensarbete

Thermal Impact of a Calibrated Stereo Camera Rig

Thermal Impact of a Calibrated Stereo Camera Rig

Examensarbete utfört i Datorseende

vid Tekniska högskolan vid Linköpings universitet

av

Sammanfattning

Abstract

Acknowledgments

Contents

1

Introduction

1.1

Motivation

1.2

Background

1.3

Purpose

1.4

Problem Description

1.5

Restrictions

1.6

Outline

2

Theory

2.1

Related work

2.2

Camera Model

2.2.1

Internal camera parameters

2.2.2

External camera parameters

2.2.3

Lens distortion model

2.3

Camera Calibration

2.3.1

Homography between two planes

2.3.2

Camera Calibration

2.4

Stereo Cameras

2.4.1

Calibrated epipolar geometry

2.4.2

Calibration of a stereo camera rig

2.5

Geometry estimation of calibration board

2.6

3D reconstruction

2.6.1

Triangulation

3

Method

3.1

Hardware

3.2

Collection of data

3.2.1

Experimental setup

3.2.2

Construction of 3D calibration object

3.2.3

Data Collection

3.2.4

Geometric estimate of calibration board

3.3

Datasets

3.4

Implementation

3.4.1

Processing of data

3.5

Evaluation of results

3.5.1