Fabrication and tracking of

IN

DEGREE PROJECT ELECTRICAL ENGINEERING, SECOND CYCLE, 30 CREDITS

STOCKHOLM SWEDEN 2016 ,

Fabrication and tracking of

Artificial Bacterial Flagella using stereo holographic diffraction

Supervisor: Prof. Bradley J. Nelson Examiner: Prof. Kristinn B. Gylfason SUMIT MOHANTY

KTH ROYAL INSTITUTE OF TECHNOLOGY

SCHOOL OF ELECTRICAL ENGINEERING

Acknowledgements ii

Acknowledgements

I’d like to express my gratitude to Prof. Dr. Bradley J. Nelson for giving me this opportunity for doing my master thesis at MSRL and explore the field of microrobotics. On the administrative end, I’d like to thank Brigitte Geissmann (MSRL), Malin Hedberg (KTH), Prof. Dr. Kristinn B. Gylfason (KTH) and MST department at KTH for allowing me to pursue this project.

I’d like to thank my supervisor Dr. Andrew Petruska for introducing me to the mathematical realms of computer vision and constantly pushing me to work on this project. I’d be grateful to him for all his knowledge that inspired me to hone my skills in image processing and CV. I’d like to thank my supervisor Ayoung Hong for accepting me as her student, helping me get across every labyrinthine hurdle I faced and relentlessly giving her time in pursuit of this project. I’d like to thank Carlos Alcantara for accepting me as a thesis worker, and supervising me on all the requisite fabrication tools. Special thanks to Samuel Charreyron for accepting my application, helping me decide the right topic along with Carlos and constantly guiding me on vision related problems. I’d like to thank Erdem Siringil for helping me make the setup, Burak Zeydan for helping in programming the user interface for acquisition, Franziska Ullrich for teaching me the ANOVA method and Xiaopu Wang for introducing me to the ABF setup. I’d like to thank Dr. Daniel and Naveen for the great time in and outside the lab and all the members of MSRL who have enriched my experience during this tenure.

Further, I’m grateful to my friend Kajsa Djupfeldt for helping me translate the abstract in Swedish.

Financial support for my thesis period was provided by Zeno Karl Schindler foundation, Geneva. I’d always be grateful to the foundation for providing me sufficient financial means to survive in this tough economy.

On a personal note, I’d like to thank my parents for their constant support and

encouragement during tedious times. I’m extremely grateful to them for bearing

with my long absence at home and holding up their determination for my progress

even when I gave up. I’d like to dedicate every ounce of my hard work to my

friends Navneet Kumar and Uday Dua.

Abstract iii

Abstract

The emerging field of mobile microrobotics has been suggestive of a plethora of applications, from intelligent micromanipulation to diversifying the prospects of minimally invasive surgeries. Several designs of microrobots have been proposed and are being notoriously studied to enhance the knowledge of their behavior in different environments to cater such applications. At present, control and surveillance of these microrobots have been aided by the convergence of vari- ous technologies like magnetic actuation, microscopy and computer vision. The quintessential knowledge of their maneuverability could provide interesting im- plications of these microrobots in their targeted applications.

In context of widening this understanding of their behavior, several dexterous methods have been proposed to study and control these microrobots, employ- ing microscopy in stereo vision for visual surveillance. The intuitive drawback of microscopy, jeopardizing its focusing accuracy against field of imaging, has limited these studies to smaller observable volumes. Addressing limitations of conventional microscopy, holography has been explored as a potential candidate for imaging and spatial tracking of these microrobots by means of reconstructing 2-D information in the images. However, the resolution of depth estimation and processing cost incurred in reconstruction posed drawbacks on its applicability.

The novel method proposed in this report, employed the holographic imaging in stereo vision, overcoming the limitations of both conventional microscopy and holographic reconstruction. The cost of processing this image information at a much lower processing speeds further benchmarked its candidature for spatial tracking of microrobots. The inventive setup design proposed in the report was inspired by conventional stereo vision. It also entails the tracking procedure of a class of flagellated microrobots called Artificial Bacterial Flagella (ABF) based on retrieval of its two diffraction based holograms, and estimating its 3-D posi- tion based on lateral positions in the two stereo projections. It further suggests feasible design metrics for fabrication of these ABFs based on Fresnel diffraction.

The central idea behind this 3-D estimation proposed here is based on processing

diffraction pattern produced by the ABFs using an image segmentation algorithm

and further projecting the lateral coordinates so obtained to real space, achieving

much more finesse over the depth resolution in either direction.

Sammanfattning iv

Sammanfattning

Den framv¨ axande omr˚ adet mobila mikrorobotik ses kunna anv¨ andas f¨ or en m¨ angd olika till¨ ampningar, fr˚ an intelligent mikromanipulering till att diversifiera m¨ ojligheterna f¨ or minimalt invasiva operationer. Flera utformningar av mikrorobotar har f¨ oreslagits och studeras notoriskt f¨ or att ¨ oka kunskapen om deras beteende i olika milj¨ oer f¨ or att se hur dessa kan f¨ orb¨ attras och anv¨ andas f¨ or dessa ¨ andam˚ al. I dagsl¨ aget kontrolleras och ¨ overvakas mikrorobotara med hj¨ alp av en konvergens av olika tekniker som; magnetisk aktivering, mikroskopi och data igenk¨ anning. Kun- skapen om hur mikrorobotarna agerar i olika milj¨ oer g¨ or att vi kan utveckla dessa mikrorobotar i sina kommande till¨ ampningar.

I samband med att bredda denna f¨ orst˚ aelse f¨ or deras beteende har flera metoder f¨ oreslagits f¨ or att studera och kontrollera dessa mikrorobotar, som t.ex att ut- nyttja dessa f¨ or mikroskopi f¨ or visuell ¨ overvakning fr˚ an tv˚ a h˚ all. Den uppenbara nackdelen med mikroskopi ¨ ar att den ¨ aventyrar noggrannheten genom att endast fokusera p˚ a vissa delar, vilket begr¨ ansar dessa studier till att inte observera hel- heten. P˚ a grund av dessa begr¨ ansningar av konventionell mikroskopi har holografi unders¨ okts som ett alternativ f¨ or avbildning och spatial sp˚ arning av dessa mikro- robotar genom att rekonstruera 2-D informationen som ges i bilderna. Uppskat- tning av till vilket djup uppl¨ osning h˚ aller samt tid˚ atg˚ angen f¨ or ˚ ateruppbyggnaden av bilderna ¨ ar dock tv˚ a nackdelar f¨ or holografins anv¨ andbarhet. Det nya f¨ orfarandet som f¨ oresl˚ as i denna rapport, har en utg˚ angspunkt i holografisk avbildning i syfte f¨ or denna funktion i dubbelseende och f¨ or att ¨ overvinna de begr¨ ansningar som finns hos b˚ ade konventionell mikroskopi och holografisk rekonstruktion. Tids- besparingarna f¨ or att behandla denna bildinformation p˚ a en l¨ agre bearbetning- shastighet ¨ ar en ytterliggare faktor till varf¨ or en holografisk metod ¨ ar v¨ asentlig f¨ or spatial sp˚ arning av mikrorobotar. Den nyskapande installationsutformnin- gen som f¨ oresl˚ as i denna rapport ¨ ar inspirerad av konventionellt dubbelseende.

Det inneb¨ ar ocks˚ a att sp˚ arningsprocessen av en klass av flagellated mikrorob- otar kallas Artificiell Bakterieflagella (ABF) baserat p˚ a h¨ amtning av sina tv˚ a diffraktionsbaserade hologram, och uppskatta dess 3-D-l¨ age baserat p˚ a sido po- sitioner i de tv˚ a bildprognoserna. Den f¨ oresl˚ ar vidare genomf¨ orbara konstruk- tionsm˚ att f¨ or tillverkning av dessa ABFs baserade p˚ a Fresnel diffraktionerna.

Den centrala id´ en bakom denna 3-D uppskattning som f¨ oresl˚ as h¨ ar bygger p˚ a

Sammanfattning v

en bearbetning av diffraktionsm¨ onsterna som produceras av ABFs med hj¨ alp av

en bildsegment-algoritm och ytterligare utskjutande sidokoordinater erh˚ allna till

verkligt utrymme, och p˚ a s˚ a vis uppn˚ ar en mer konkret djupuppl¨ osningen i endera

riktningen.

Contents vi

Contents

Acknowledgements ii

Abstract iii

Sammanfattning iv

List of Figures viii

1 Introduction 1

1.1 Review on microrobotics . . . . 1

1.1.1 Design and control of magnetic microrobots . . . . 2

1.1.2 Artificial Bacterial Flagella (ABFs) . . . . 3

1.2 State of the art in tracking microscopic entities . . . . 4

1.2.1 Trade-offs with conventional microscopy . . . . 7

1.3 Holography . . . . 8

1.3.1 Principle . . . . 8

1.3.2 Holographic diffraction . . . . 10

1.3.3 Holographic reconstruction . . . . 10

1.4 Stereo-holography . . . . 12

1.4.1 Holography employed in stereo vision . . . . 12

1.4.2 Implementation and scope of the tracking method . . . . . 13

2 Fabrication of ABFs 15 2.1 Overview of fabrication approaches . . . . 15

2.2 Direct Laser Writing . . . . 16

2.2.1 Design metrics . . . . 18

2.2.2 Different writing strategies . . . . 19

2.3 Metallization and Characterization . . . . 21

3 Setup Design 24 3.1 Helmholtz coil system . . . . 24

3.1.1 Interfacing the coil setup . . . . 26

3.1.2 Calibration of the coil setup . . . . 26

Contents vii

3.2 Optical components and setup interface . . . . 27

3.2.1 Assembly of optical components in the coil setup . . . . . 27

3.3 Camera calibration procedure . . . . 29

3.3.1 Newton-Raphson Inverse Jacobian method . . . . 30

4 Tracking algorithm 35 4.1 2D image segmentation . . . . 36

4.1.1 Gradient based image segmentation . . . . 37

4.1.2 Voting . . . . 38

4.1.3 Local neighborhood suppression . . . . 39

4.1.3.1 Extending to multiple tracking . . . . 39

4.1.4 Non maximal suppression . . . . 40

4.1.5 Least squares solution in ROI . . . . 42

4.2 Background subtraction methods . . . . 43

5 Experiments and Discussions 45 5.1 Implementing the tracker with a user interface . . . . 45

5.2 3-D position estimation . . . . 46

5.3 Tracking experiments with ABFs . . . . 48

6 Conclusion and future perspective 53 References 55 A Appendix 59 A.1 MATLAB code for continuous writing method . . . . 59

A.2 DeScribe script to convert gwl files to job (Continuous writing) . 61 A.3 DeScribe script to convert gwl files to job (Pulsated writing) . . . 62

A.4 Setup components and assembly . . . . 64

A.5 Tracking algorithm . . . . 66

List of Figures viii

List of Figures

1 Applications of microrobotics . . . . 1

2 Review of magnetic microrobot designs . . . . 2

3 ABF propulsion mechanism . . . . 4

4 Magnetic coil setups at MSRL . . . . 5

5 Overwiew of tracking methods . . . . 6

6 Aspects of conventional microscopy . . . . 7

7 Principle of Holography . . . . 8

8 Holographic reconstruction . . . . 11

9 Stereo vision versus holography as a motivation for stereo holography 12 10 Overview of ABF fabrication methods . . . . 15

11 Direct Laser Writing . . . . 17

12 Rendered frame showing the two writing strategies . . . . 20

13 SEM results of continuous writing strategy . . . . 22

14 SEM results of pulsed writing strategy . . . . 22

15 Helmholtz coil setup, interface and calibration curve . . . . 25

16 Schematic of the setup interface with optical components . . . . . 27

17 Illustration of the setup showing Helmholtz coils with optical com- ponents . . . . 28

18 Schematic for camera calibration . . . . 30

19 Flow chart of Newton-Raphson Inverse Jacobian method . . . . . 31

20 Camera calibration data points with corrected fit . . . . 32

21 Flowchart of tracking algorithm . . . . 35

22 2-D image segmentation . . . . 37

23 Gradient based image segmentation . . . . 38

24 Voting and bitwise processing for local neighborhood suppression 39 25 Suppressing local neighborhood and non maximal suppression for multiple detection . . . . 41

26 Least squares optimization . . . . 42

27 Background subtraction methods . . . . 43

28 Schematic of the setup interface with optical components . . . . . 45

29 Triangulation of height pixel coordinates . . . . 47

30 Time lapse images of an ABF from the two cameras . . . . 49

Notation ix

31 Tracking experiments with ABFs . . . . 51

32 Multiple tracking experiments with ABFs . . . . 52

1 Introduction 1

1 Introduction

1.1 Review on microrobotics

Bridging science fiction and reality, microrobots have converged a plethora of technologies to the brink of invasive surgeries. Recent developments in robotics have offered intelligent means for control and manipulation at micro-nano scale, opening gateways for biomedical applications from surgeries to targeted drug de- livery [1][2]. One of the novel designs intended towards retinal surgeries, could be controlled in a wireless fashion guided by magnetic fields [3]. The ability to control and maneuver these robots to inaccessible parts of the human body has been looked upon with great interest [1]. This wireless mobility has also been exploited in numerous other applications demanding micromanipulation such as protein crystal harvesting [4]. However, the manipulation of these robots neces- sitated a better understanding of their motion to be able to control and survey these mobile microrobots in order to reliably manipulate them for the desired ap- plication [5][6]. This motivation has stimulated a plethora of studies conducted on design and understanding of microscopic robots and swimmers under different working conditions [7].

(a) (b)

Figure 1: Illustration of microrobots employed in different application showing

(a) a magnetic microrobot used for retinal surgeries [3], and (b) a novel design

called RodBot used for micromanipulation [4]

1.1 Review on microrobotics 2

1.1.1 Design and control of magnetic microrobots

Among different approaches, the ability to control and maneuver magnetic mi- crorobots in presence of spatially varying magnetic fields has shown promising efficacy in achieving several degrees of freedom [6][8]. Several intelligent mi- crorobot designs have been proposed demonstrating efficient locomotion when actuated magnetically [6][7]. Many of these designs are inspired from nature, bio- mimicking microbes and other microscopic entities [9][10]. Fig. 2a shows sperm inspired microrobot that could navigate using weak magnetic fields reciprocating the motion of a spermatozoa [9]. Another inventive design has been reported that could be controlled in a wireless fashion based on magnetic resonance shown in fig. 2b [11].

(a)

(b) (c)

Figure 2: Review of different designs of magnetically driven microrobots showing (a) a sperm inspired microrobot called Magnetosperm [9], (b) resonant magnetic microrobot design called Magmite [11], and (c) flagellated Magnetotactic bacteria for magnetotaxis [12]

Contrary to the above mentioned designs, a strategy of using and integrating

magnetostatic bacteria with microscpic components has been attempted at bio-

actuation under MRI control, shown in fig. 2c [12]. In a nutshell, designs inspired

1.1 Review on microrobotics 3

from nature or mimicking microbes found in nature have been explored to study propulsion and swimming behavior under the influence of magnetic field.

1.1.2 Artificial Bacterial Flagella (ABFs)

At micrometer scale, the dominance of surface effects over bulk properties, changed the paradigm of modeling the locomotion of these small robots. The traditional methods of generating translational forces aren’t sufficient to overcome the sur- face drag acting on these objects thereby suppressing their motion in fluids. An intelligent design has been proposed, exploiting spatial magnetic gradients and rotating fields, to dexterously imitate a bacterium termed as Artificial Bacte- rial Flagella, or h-ABF, as shown in fig 3[13]. Citing the classical example from Purcell’s lecture on ”Life at low Reynolds number”, the corkscrew motion of E.

Coli bacteria in nature has been held as an inspirational model for realizing the motion of these flagella [10][14]. The central idea behind such a design was to use reciprocal motion of the flagellum of these bacteria to be able achieve forward motion in fluids. This back-and-forth motion, gave an impression of a rotational motion when viewed along the cross section as shown in fig 3b. This formed the foundation for the swimming behavior of these ABFs. Digressing from the biological aspect of such a motor design, a rather more convenient and crafty method of developing such a flagellum was by means of developing a helical fil- ament and imparting magnetic characteristics to it [10]. This helical filament, when made or coated with a magnetic material, instantaneously magnetized in response to changing magnetic fields [10]. Therefore, magnetic field could be ap- plied in a rotatory fashion, allowing these filaments to catch up and synchronize with these instantaneous field vectors thereby generating a torque about its own axis as shown in fig 3a. A comprehensive description of the fabrication process would be explained in the second chapter of the report.

The flagellar hydrodynamics in a low Reynolds number regime, could be mod- eled as a one dimensional propulsion matrix [16], relating the torque τ , force F acting on the body with the linear and rotational velocities u an ω respectively as,

"

F τ

#

=

"

a b c d

# "

u ω

#

As discussed earlier, since the influence of inertial forces was negligible in this

1.2 State of the art in tracking microscopic entities 4

(a) (b)

Figure 3: Illustration of the propulsion mechanism of an artificial bacterial flagella (ABF) showing (a) magnetic field driven torque [15], and (b) the classical cork screw motion of E. Coli bacteria’s forward translation [a], reciprocated by the ABF [b] [14]

regime, the coefficients of linear velocity and rotational velocity become linearly proportional [10][16]. Hence, the forward motion of this flagellum is linearly related to the rotational component, which is attributed by the toque compo- nent. Moreover, this frequency component here could be directly related to the frequency of rotation of magnetic field [16]. Hence, it was possible to achieve control over motion of these ABFs by controlling the magnitude and frequency of magnetic field. A detailed description of the magnetic control setup would be described in the third chapter of the report. In conclusion, these helical struc- tures showed locomotion when actuated under a rotating magnetic field, much akin to a rotary motor, offering great propulsion in fluids where surface drag effects strongly attenuated their translational motion [14].

1.2 State of the art in tracking microscopic entities

Various magnetic coil setups have been proposed to steer and control the mo- tion of such magnetic entities in presence of uniform fields or gradients obeying the traditional Maxwell principle [8][17]. Fig 4a illustrates an 8 electromagnetic coil setup providing spatially varying magnetic fields for guiding magnetic micro- robots in a designated work space [8]. Another such implementation of magnetic coil setup is shown in fig 4b, with three pairs of electromagnetic coils, designed in a fashion to provide uniform magnetic magnetic field at its center [13]. This setup could be used to provide rotating magnetic field of constant magnitude in a designated work space at its center.

An equally important consideration is the visual servoing of these micro-scale

1.2 State of the art in tracking microscopic entities 5

(a) (b)

Figure 4: Magnetic coil setup for control and tracking of microrobots featuring (a) Octomag, an 8-electromagnetic coil system for providing field and spatial gradients for magnetic control [8], and (b) a three pair Helmholtz coil system for providing rotating magnetic fields to actuate ABFs [13]

robots for efficient tracking of their motion. The surveillance of these microrobots has been ubiquitously performed by using stereo vision. For instance, the 8 coil magnetic setup mentioned above accommodate two such cameras in adjunct with microscopic objectives for localized tracking of microrobots, thereby giving a top and side profile of its positions as shown in fig. 5a. To aid visual servoing of these robots, these coil setups have been developed with a feedback loop for providing position of the robots in real time providing further control of these robots in their work space [8][18]. The 3-D closed loop control of microrobots in the setups shown in fig. 3a and 5c, have been aided by cameras positioned in orthogonal positions w.r.t. each other for visual surveillance. In both these setups, the idea of tracking microrobots is based on extracting its lateral projections from the cameras employed in stereo mode to triangulate the 3-D positions in work space [8][18].

Another variant of tracking mechanism shown in fig 5b employed diffraction

based phase contrast microscopy [19]. In this method, the diffraction pattern

of E. Coli bacteria in the given work space was recorded and compared with

a pre-calibrated library of patterns to estimate the depth information in the

work space. This depth information coupled with the already tracked 2-D lateral

position thereby gave the overall 3-D position [19]. However, the knowledge

of diffraction patterns in a working volume was a pre-requisite for gaining the

leverage over this depth information. Contrary to the case of stereo vision based

1.2 State of the art in tracking microscopic entities 6

(a) (b)

(c)

Figure 5: Overview of tracking mechanism featuring microscopy based stereo vi- sion showing (a) Side and Top View of tracking with the Octomag setup and the respective trajectory [8], and (b) phase contrast imaging based tracking diffrac- tion patterns of E. Coli and the corresponding trajectory obtained [19], (c) stereo vision based tracking of Janus micromotors (with the side and top view corre- sponding to a trajectory) along with the setup for magnetic control [18]

.

microscopy, this diffraction information coming from a lateral projection could

impart the height information by means of contrast matching, though it in order

to do so accurately, the focus of the objective had to be precisely adjusted. Hence,

there was no direct method for computation of depth simply based on the depth

information using conventional optical methods. In a nutshell, for the given cases

of tracking methods, there was a high reliability on the lateral information for 3-D

position estimation, with microscopy based stereo vision being the commonplace

method.

1.2 State of the art in tracking microscopic entities 7

1.2.1 Trade-offs with conventional microscopy

(a) (b) (c)

Figure 6: Limitations of conventional microscopy. (a) Two microscopes employed in stereo vision showing top and side view of an array of ABFs and the limited working volume for imaging, and the classical trade off between (b) field of view and (c) depth of focus

.

Delineating into the conventional microscopy when employed in stereo vision, it could be clearly seen that this technique offered a limitation on the working volume as shown in fig 6a. It shows a pair of orthogonal objectives being used to image the long axis (top) and the cross section (side) of an array of ABFs.

Intuitively, microscopy has a high field of imaging when imaging an array of ABFs

as shown in fig 6b while it lost this areal resolution when focusing any one the

individual ABFs at a certain depth as shown in fig 6c, thereby attributing this

limitation of having a small working volume for 3-D tracking. Hence, despite

having a larger field of view, these stereo vision based imaging techniques lose

finesse over the depth of focus. Conclusively, in order to track extremely tiny

robots, the field of view has to be jeopardized for achieving accuracy in depth of

focus.

1.3 Holography 8

1.3 Holography

Digital holograms, have been ubiquitously known to contain graphical informa- tion of objects when viewed from different angles. The most commonplace per- ception of holography is the science and practice of illuminating and recording images of objects in a way to represent the 3-D information contained in the object on a special photographic plate without any aid of intermediate optics for visualization. This method of recording the light field opened gateways for recording geometrical information of an object into an image overcoming limita- tions of traditional microscopy [20].

(a)

Figure 7: Schematic showing a setup for holographic imaging [21]

1.3.1 Principle

Holography could be explained by an interplay of optical principles of interfer-

ence, coherence and diffraction. Traditionally, wave front of light from a coherent

source like laser, is known to interact with each other giving interference pat-

terns. The nature of interference pattern is said to be constructive or destructive

1.3 Holography 9

based on the path difference or the phase difference between the two wave fronts.

In holography, this interference pattern is formed by the light illuminating the object in its path (known as object or signal wave) and the original light unper- turbed by any obstacle, (known as reference beam) coming from the same source as shown in fig. 7a [20][21]. The underlying principle here is, two light beam travel the exact same path before interfering on the screen, which meant that the only path or phase difference that could be held accountable for variation in the interference pattern must owe to the geometry of the object or the obstacle that the signal arm encounters on its way, which thus gets encoded into the in- terference pattern. The intermediate optics involved in splitting the light wave into the reference and signal arm shown in the figure ensure that light doesn’t encounter any additional path difference, thereby making it spatially coherent.

Further, having a reliable source of light like laser ensure that the light is also

temporally coherent. Hence, having a perfectly coherent light ensures that path

difference encountered by light is solely due to the object [22]. Another important

consideration for imaging is the pixel resolution of the detecting screen, which

necessitates the pixel size to be of the order of wavelength of light [21]. From a

photographic perspective, given a fine grain size of the photographic plate, the

holograms could be viewed by merely illuminating it [22]. This is enabled by

diffraction of light by the fine grain particles on the plate, making the 3-D geom-

etry visible to the observer merely by illumination [22]. Furthermore, holograms

could also be stored digitally by means of digital CMOS camera with appropriate

pixel resolution and further reconstructed. In this regard, digital holography has

been explored as a potential candidate for imaging due to its ability of storing 3-

D volumetric information at different depths, which could be accessed by further

post processing [23][24]. Mathematically, this 3-D information could be accessed

by reconstructing the geometry of the entity contained in it at different planes of

focus [25]. When imaging microscopic objects, the interference pattern is formed

by the light diffracted by these objects, which could be digitally analyzed and

reconstructed. The following segment elaborates how the holographic diffraction

varies from the classical definition of diffraction.

1.3 Holography 10

1.3.2 Holographic diffraction

Objects at microscopic scale are known to scatter light and to form diffraction patterns corresponding to the specific wavelength of the light source when kept at a certain distance. This diffracted light, upon interference with the light coming directly from the source (termed earlier as reference beam), results into a holographic diffraction pattern [20][25]. Mathematically, if the distribution of scattered object wave in a plane is denoted by O(x,y), which could describe either its electric or magnetic field component, and the reference wave is denoted by R(x,y), then the overall intensity at the image plane I(x,y) could be described as,

| I(x, y) | ² =| O(x, y) + R(x, y) | ² =| O(x, y) | ² +

| R(x, y) | ² +O(x, y) ^∗ R(x, y) + O(x, y)R(x, y) ^∗

(1.1)

where, |O(x,y)| ² represents the amplitude information of the object wave and

|R(x,y)| ² represents that of the reference wave, collectively known as the DC term [25]. The former of the last two terms represents the real image, where the object wave is complex conjugated i.e. O(x,y) ^∗ while the second term represents the virtual image, collectively known as twin images [25]. These twin images essentially form the interference pattern and contain the phase information of the image, which could be extracted to obtain the depth of object scattering the light [25]. Contrary to the setup shown in fig 7a, this method of imaging could be implemented without the need for splitting the reference beam of light and therefore, both the diffracted and reference beams could be aligned to each other in the same axis as shown in fig 8a. This method of taking digital holograms along the same principal axis is called digital in-line holography [23][24][25].

1.3.3 Holographic reconstruction

Several reports have demonstrated holographic reconstruction methods for track-

ing microscopic entities [24][26]. One such method developed at MSRL, described

a closed loop control of microrobots with real time tracking based on digital in-line

holographic reconstruction [26]. The depth reconstruction of the object from the

digital hologram requires back-propagation of the hologram to the object plane

using Fresnel Kirchoff approximation [25]. The setup shown in fig. 8b illustrates

the microrobot in a working volume being imaged, where the corresponding holo-

1.3 Holography 11

(a) (b)

Figure 8: (a) Schematic of in-line holography showing the reference (incident) and object (scattered) wave [25], and (b) setup (left) and reconstruction method (right) described by a real time holographic tracking method [26]

.

gram was recorded on CMOS camera sensor. It employed a collimating lens for imaging the work space that posed no limitation on the size of work space im- plying a much larger observable volume compared to traditional microscopy. Fig 8c shows the reconstruction process with the hologram h _o recorded at the holo- gram plane (ξ, η) containing the 3-D information of the object. This could be reconstructed to an image plane (u,v) located at a distance d from the hologram plane to give the corresponding coordinate information. Based on the above mentioned approximation for planar wave fronts, the complex amplitude image h(u,v,d) could be given by the integral,

h(u, v, d) = i λ

Z Z

h _o R exp(−i ^2π _λ ρ) ρ

 1 2 + 1

2 cosθ 

dξdη (1.2)

where R represents the reference wave and ρ and θ represent the polar vector com- ponents between the planes shown in 8c [26]. The tracking process comprised of lateral isolation of the hologram and applying this integral formula in a defined volume around a previously known depth in an iterative fashion to obtain the h _o , and thus the 3-D position of the object [26]. However, extracting the object’s 3-D information from the hologram is a computationally cumbersome procedure.

This process of acquiring hologram information from the image, followed by its back-propagation could be performed at 40Hz at GPU processing frequency [26].

Further, such post-processing of an image might offer great difficulty for tracking

multiple objects in a wider field of view since it required simultaneous retrieval

of all the objects in real time. As a result, multiple computational steps for ex-

1.4 Stereo-holography 12

traction of multiple objects in tandem might levy on the hardware requirements.

Secondly, the depth accuracy in the reconstruction method described above is about 125µm [26].

1.4 Stereo-holography

(a) (b) (c)

Figure 9: A comparative summary of the methods of imaging described in the text showing schematics for (a) microscopy employed in stereo vision, (b) holographic reconstruction and (c) holography employed in stereo vision showing the Airy ring diffraction pattern formed on bottom and side view

.

1.4.1 Holography employed in stereo vision

As discussed earlier, digital in-line holography has been used to preserve the

height information of the objects in image plane on their respective holograms

[24]. Off focus images have been used in the past to retrieve both lateral and

depth information of the object without reconstruction by merely normalizing

the pixel intensity values of the diffraction pattern, thereby giving the location

coordinates [27][23]. Henceforth, combining in line holography and stereo vision,

it could be possible to extract lateral information of an object in two orthogo-

nal image planes, one acting as the depth coordinate for the other. Thus, the

holographic setup described earlier (9b) could be used with another replica of

1.4 Stereo-holography 13

the same, engaged in orthogonal manner w.r.t each other and having two CMOS cameras functioning similar to that in conventional stereo vision as shown in fig 9c. Fig 9 a-c shows a comparison of the three methods discussed so far. The motivation behind using holography in stereo vision could be seen w.r.t. stereo vision as giving a significantly higher observable volume owing to the similar in- termediate optics as the conventional holography. On the other hand, the ability to track without reconstruction could make the technique much faster compared to traditional holography. Moreover, the two depth directions being coupled to each other via orthogonal cameras could improve the depth accuracy as well.

1.4.2 Implementation and scope of the tracking method

The two cameras in this setup retrieved holographic diffraction patterns, referred to as Airy ring patterns shown in 9b-c, similar to that in conventional holography whose centers were instead used to detect the lateral position of the object being imaged. The two lateral projections obtained this way could be used to triangu- late the 3-D position of the object just like conventional stereo vision. Therefore, the most essential aspect of this tracking method was reliable detection of the centers of these Airy ring patterns as corresponding pixel coordinates in the re- spective cameras.

Lateral detection of these pixel coordinates from two different stereo projection images could enable exact mapping of the microrobot in a 3-D workspace. This could be done by hackneyed image processing techniques like the Hough trans- forms, similarity mapping or cross-correlating the pixel intensities in the image with an existing template. However, these methods have been either known to give false alarms, or based on template matching methods that were dependent on prerequisite information [28]. The principle proposed in this method is based on mapping gradient vectors across the whole span of the image [28]. The in- tuitive idea of having a gradation of intensities, either towards or away from a central spot, could provide a more reliable way of isolating the airy ring pattern.

Furthermore, these vectors could be extrapolated to give this central intersection coordinate thereby saving tremendous computational time. Triangulating this information from two different image planes could determine the object’s exact location in a 3-D workspace.

This method might not reproduce the 3-D volume information of an object, but

1.4 Stereo-holography 14

could effectively provide the object location in significantly less number of com- putational steps. This would suffice for multiple tracking of small microrobots such as ABFs in a wider field of view. Therefore, the trade off between the field of view in conventional stereo vision and depth accuracy in holographic diffraction is addressed by the novel setup design proposed in the report.

The report is organized into different section, entailing the details of fabrica-

tion aspects of ABFs, its magnetic control, implementation of centroid tracking

algorithm and subsequent interfacing of the overall system.

2 Fabrication of ABFs 15

2 Fabrication of ABFs

2.1 Overview of fabrication approaches

A plethora of fabrication methods have been explored in making the artificial bac- terial flagella (ABFs), which from a design perspective, ultimately boiled down to different helix writing strategies. The helical microrobots could be later met- allized to impart the desired magnetic properties. A pedantic doctoral disserta- tion report at MSRL, describes a comprehensive overview of different fabrication methods explored so far in making ABFs or similar helical structures, entailing the four primary approaches as rolled up method, direct laser writing, glancing angle deposition and template assisted method [15]. One of the foremost at- tempts on fabricating these helical structures was based on the rolled up method, where the overall design was composed of a magnetic head and a non magnetic tail [13]. In this method, the helical tail was fabricated using conventional thin film deposition based on several layers of InGaAs [10]. The internal stresses de- veloped in the bi/tri-layer of the material resulted into a self-scrolled structure as shown in fig. 10a.

(a) (b)

Figure 10: Showing (a) the rolled up method with magnetic head and helical tail

(scale bar is 4µm) [10], (b) [a-e] showing laser writing using Nanoscribe ^{T M} on

a positive tone resist with the helical body comprising of polypyrrole deposited

using electroplating [29]

2.2 Direct Laser Writing 16

As mentioned before, ABFs could translate using rotatory motion of the heli- cal body in a cork screw motion. In the rolled up structure, it could be achieved by means of a magnetic head made of Nickel that could guide the non magnetic body in a reciprocal motion [10].

Another approach of designing these helices could be by using the helical body itself made of magnetic material. This could be very conveniently done by di- rect laser writing, a kind of 3-D laser lithography technique, where the desired structures could be directly written on a polymeric resist as shown in fig 10b, which could later be developed and metallized [29]. A dexterous equipment man- ufactured by Nanoscribe GmbH provided with a platform of DLW based on two photon polymerization (2PP) process as shown in fig. 11a and b. In this method, a photoresist material was deposited on a glass substrate, that could be conve- niently handled by piezoelectric stage, and subjected to two laser beams causing localized polymerization of the resist at the focal points of the beams as shown in fig 11b [30].

Following traditional MEMS fabrication process, these structures could be made by either choice of photoresist i.e. both using a positive tone and a negative tone resist. Using a positive tone resist, these structures could be written on a resist, creating a helical template after developing thereby acting as a hollow cast for the following step. This could be followed by subsequent electroplating of CoNi as a metallic head, much like rolled up design. Similarly, the helical body could electroplated with poly-pyrole using the substrate itself as one of the electrodes [29]. The resist later on could be dissolved as shown in the fig. 10b. On the contrary, using a negative tone resist, these structures could rather be made of the resist material itself, while evaporating metal on its outer body. This way helical structure itself could be made magnetic without necessitating a magnetic head for guidance [15]. With larger design prototypes, it could be considered more pragmatic to have the negative photoresist leveraging on judicious material consumption and ease of magnetic control. In this report, DLW technique using a negative tone resist was preferred based on the design suitability.

2.2 Direct Laser Writing

As mentioned earlier, DLW is based on the principle of two photon polymer-

ization (TPP), which uses localized chemical reaction on the resist material by

2.2 Direct Laser Writing 17

(a)

(b) (c)

Figure 11: Showing (a) a schematic of DLW tool Photonic Professional GT (Pic- ture borrowed from www.nanoscribe.de), (b) an illustration of two photon poly- merization process (2PP) [30], (c) DLW of helices based on 2PP [31]

focusing two laser beams as shown in fig 11b. An equipment manufactured by

Nanoscribe GmbH called Photonics professional GT was used for DLW, that uti-

lizes femtosecond laser beams with a center wavelength of 780nm, to localize this

polymerization on the photoresist with a resolution of 100nm [15]. An appara-

tus of this equipment shown in fig 11a describes the optical components and a

piezoelectric scanning stage, further magnified in fig 11b. The motion of scanning

2.2 Direct Laser Writing 18

stage and the laser beam determined the speed and resolution of writing. A con- tinuous mode of scanning involved a beam moving along the trajectories defined by coordinate input of the design to ensure higher accuracy due to closely spaced points. On the contrary, a pulsed mode of scanning involved a rather impulsive impact of the laser beam on collective set of points defining the outer contours and boundaries of the design. The writing process could be visualized in real time due to focused beam selectively influencing the optical properties of the resist in contrast to the unpolymerized resist. A schematic of the DLW process for the helices could be seen in fig 11c.

2.2.1 Design metrics

As mentioned earlier, the underlying principle of holography in this method was diffraction. However, based on the setup specifications it could be possible to operate in two regimes of diffraction i.e. Fresnel and Fraunhofer diffraction. Ac- cording to the classical definition of diffraction, the distance between the aperture, the source of light and the screen on which the pattern was obtained, determined which regime of diffraction is being operated, given a fixed aperture size and wavelength. In the case of a holographic setup for tracking, the distance of the workspace from camera would make the aperture-screen distance equivalent. As described in the introductory section, in line holography for imaging microscopic objects involved a collimating source of light. However, given the practical con- siderations in optics, it might be difficult to consider an ideal collimator beam across longer distances. Since any tolerance for a converging or diverging beam might add to the loss of information in tracking, it was considered pragmatic to operate in a work space at close distances to the camera. Furthermore, upon increasing the work space-camera distance, the diffraction pattern so obtained was relatively bleak. Hence, keeping these considerations into account, it was considered more heuristic to operate in the Fresnel diffraction regime. Moreover, the knowledge of this diffraction regime coupled with a range estimate of the setup specifications, gave a ballpark estimate of desired ABF design metrics to be fabricated.

A parameter called Fresnel number, related the wavelength of light λ, distance

from the screen (camera) L and the aperture size (dimensions of ABF) a. Ac-

2.2 Direct Laser Writing 19

cording to this parameter, Fresnel diffraction necessitated, F = a

Lλ << 1 (2.1)

Given the red wavelength of light (780nm), and distance of the camera from the work space at about 10cm, the size required by the ABF turned out to be in the range of few 100s of µm in order to give a good diffraction pattern. However, given the stability in swimming behavior of these structures and ease of maneuverability under magnetic control, it was necessary for these bigger ABFs to have enough cross section and thickness. Furthermore, a vertical limit of laser writing with Nanoscribe posed another consideration of having stable aspect ratio of design.

Therefore, a preliminary metrics of design were formulated at:

ˆ Length = 100-250µm

ˆ Pitch = 30-50µm

ˆ Radius = 20-30µm

ˆ Thickness = 5-10µm

2.2.2 Different writing strategies

The general flow of writing process involves converting an existing design proto-

type into actual coordinates to be written, by two different approaches. Firstly,

the design could be modeled as an equation on MATLAB and the respective

coordinates of the structure could be directly written onto a general writing file

for example, in this case a general equation of helix could be modeled with given

specification of helix angle, length, radius etc to generate ABF coordinates. Al-

ternatively, the designs could be modeled as CAD designs which could be further

sliced to obtain closely stacked cross section planes within finely spaced contours

internal to these planes. This sliced information could as well be fed in form

of coordinates. In general, the designs are pre-written on the resist to get an

estimate of important writing parameters, namely laser power of the beam and

its scanning speed, to ensure stable structures.

2.2 Direct Laser Writing 20

(a)

(b)

Figure 12: Rendered frame on Nanoscribe graphic interface illustrating (a) the

continuous writing strategy designed on MATLAB, (b) and pulsed writing strat-

egy designed on CAD

2.3 Metallization and Characterization 21

As mentioned earlier, the two primary techniques of laser writing being used were the continuous writing and pulsed writing mode. For writing bigger and thicker structures, the continuous mode was used to overwrite helices on top of each along the long axis of the ABF structure as shown in fig 12a. These closely placed structure designs were programmed on MATLAB by having a uniform distribution of points around a helical trajectory thereby making the overall helix thicker. Typically, in case of a single helix, laser power is indicative of the possible helical thickness on a given resist. Therefore, in order to have finely stitched structure along the long axis, it was ensured not have a very high laser beam power to ensure uniformity of the structure.

Another strategy utilized the pulsated writing mode where the cross sectional slices of a helix of pre-defined thickness, were carefully written along the short axis of the helix as shown in fig 12b. This involved finely hashing the ABF structure into thin slices, defining appropriate contours and tessellates such that in the writing process, these planar cross sections polymerized and thus stitched to each other. The critical parameter to be tested in the pre-writing phase here was the laser scanning speed, since a lower scan speed with a relatively higher laser power could allow these sliced structures to have enough time for polymerization.

The structure produced as a result of these two writing approaches were further metallized and characterized.

2.3 Metallization and Characterization

The ABFs so obtained were further advanced for metal evaporation (PVD) with Ni deposition of about 100nm and passivated with Au as a protective measure for corrosion. The deposition process was performed at a certain tilting angle to ensure conformal coating of the metal minimizing shadowing effect. Owing to the metallic coating around the polymer substrate, these ABFs acted like a soft magnet when subjected to magnetic fields of relatively lower magnitude (upto 5mT) and thus this instantaneous magnetization occurring in response to con- tinuously varying fields helped them to propel [15].

Fig 13a shows the SEM images of the ABFs fabricated with the continuous

writing strategy. These were horizontally lying ABFs (length 150µm, pitch 30µm,

diameter 15µm and thickness 5µm) made up of a IPL-780P resist as the body

2.3 Metallization and Characterization 22

(a) (b)

Figure 13: SEM results of continuous writing approach showing (a) two ABFs (length 150µm, pitch 30µm, diameter 15µm and thickness 5µm)(scale bar is 50µm), (b) and an enlarged view of ABF showing finer features of overwriting (scale bar is 10µm)

(a) (b)

Figure 14: SEM results of pulsed writing approach showing (a) an array of ABFs

(length 250µm, pitch 30µm, diameter 15µm and thickness 5µm) (scale bar is

500µm), (b) and an enlarged view of ABF showing finer features due to closely

sliced and stitched structure (scale bar is 40µm)

2.3 Metallization and Characterization 23

substrate coated with Ni. Taking a closer look at the the fig 13b, the structure appears as closely amalgamated strands of several ABFs. It could be clearly seen that with this approach, the structure doesn’t share uniform thickness along the long axis. ABFs of thickness of 10µm fabricated with similar approach turned out to be even more disproportionately thick along the long axis. This eventually resulted into non-homogeneous magnetization of the ABF along its body thereby adversely affecting its swimming behavior.

On the contrary, the ABFs obtained with pulsed writing approach turned out

to be uniformly thick as shown in fig 14a as expected. An enlarged view of a

250µm long ABF (pitch 30µm, diameter 15µm and thickness 5µm) shown in fig

14b shows the fine contouring on the outer body as a result of closely hashed

and stitched structure along the short axis. This ABF being uniformly thick in

the design phase itself, posed no limitation in designing thicker structures. These

structure could be metal deposited with much more uniformity.

3 Setup Design 24

3 Setup Design

3.1 Helmholtz coil system

As described in the introduction, ABFs could be made to propel by creating rotating magnetic fields generating torques resulting into the pitch cork motion given by the propulsion matrix. This magnetic torque could be modeled as a triple product given by traditional Maxwell’s equation as,

τ = υ. ~ M x ~ B (3.1)

where, υ represented the magnetic volume of the ABF, M as the magnetization and B as the applied magnetic field. The magnetization of the ABF was depen- dent on the magnitude of field and it geometry, although with thin coating of 100nm Ni, the ABF acted as a soft magnetic material at lower magnitudes of field [15]. Further, due to the nature of e-beam deposition, the coating of finely deposited nanoparticles attributed a magnetization direction to be perpendicu- lar to the short axis [15]. Hence, upon applying magnetic field to the ABF, a restoring torque acted on it aligning it to the magnetic field. This principle was used for actuation of ABFs by providing a magnetic field rotating about a cer- tain expected direction of propulsion to propel the ABF in that direction. In this context, Helmholtz coils were used an actuation setup due to their capability of providing uniform magnetic field [13][15].

A Helmholtz coil comprised of a pair of co-axially arranged coils placed sym- metrically along their common axis, with direction of current passing through each coil parallel to each other [13]. According to the classical Biot-Savart law, this arrangement could minimize the first order gradient of the magnetic field at its center along the common axis, given the separation distance of the coils was same as their radius,

B _o =  4 5

 3/2 µ _o I

2R (3.2)

where B _o being the field, I being the current and µ _o being the constant of perme-

ability [13]. This region of uniform magnetic field around the center of the coils

was used to provide uniform magnetic field. Hence, such an arrangement was

considered suitable for providing rotating fields of constant magnitude ensuring

3.1 Helmholtz coil system 25

(a) (b)

(c) (d)

Figure 15: Helmholtz coil setup (b) showing an assembly of three coils with a container at the center as a work space with uniform magnetic field, (b) schematic depicting rotating field generation, (c) user interface designed to control the setup and (d) the calibration curve obtained for the three coil pairs to determine field to current coefficients

uniform magnetic torque acting on the ABF in a working region around the cen-

ter [13]. Further, assembling three such pairs of coils, appropriately varying in

radius and number of turns of magnetic material, it could be possible to have a

3-D region with field uniformity. This volumetric region would comprise as an

ideal work space for propulsion of ABFs. Fig. 33a shows such a work space at

the center of three orthogonal pairs of coils.

3.1 Helmholtz coil system 26

3.1.1 Interfacing the coil setup

Superimposing the magnetic field components of the three coils described above, could be used to provide rotating DC magnetic field about any arbitrary axis of rotation in 3-D. Fig.33b illustrates how the rotating magnetic field was generated.

The angle shows in the figure, α and β could be used to determine the values of currents required to generate rotating magnetic field vectors B1 and B2 spanning the plane of rotation at any instant of time, thereby generating a propulsive force in a direction perpendicular to this plane [13]. Therefore, the coil could be made to function with a user interface providing the desired direction of propulsion in terms of pitch and yaw parameters for direction, and along with the magnitude and frequency of rotating field. This could be further decomposed as the current components, corresponding to requisite magnetic field components, that could be fed to each of the coils superimposing to generate the desired propulsive force.

Fig. 33c shows the user interface designed to take these parameters as input and generate desired rotating field. The coil setup was interfaced to a computer via a current amplifier with a voltage range of -10V to 10V to provide the necessary amplification for generating such high values of DC currents. Therefore, in order to operate this setup, the information of field to current ratio was required for each of the coils. These field to current coefficients were determined by the calibration procedure described in the following section and were used to tune the user interface.

3.1.2 Calibration of the coil setup

In order to calculate the field to current coefficients, calibration experiment was

performed with a gaussmeter manufactured by Metrolab for measuring magnetic

field in the work space. Prior to calibration, the outer coil direction was chosen

as X, middle as Y and innermost as Z as reference. Firstly, for a constant DC

current applied to each of these coils individually, and the variation of magnetic

field was observed using the gaussmeter in a range of about a cm both along the

common axis and away from the common axis. With the knowledge of this field

uniformity in this region, experiment was performed by varying the magnitude of

DC current supply and the corresponding magnitude of the field was measured at

the center of the setup. It was found that the field varied linearly with increments

of current in the range of -12A to 12A though at different rates for each of the

3.2 Optical components and setup interface 27

coil pairs as shown in fig. 33d. The slopes of these curves were used to determine the field to current coefficients as 0.526mT/A, -0.807mT/A and 1.136mt/A for coils designated as directional to X, Y and Z respectively. Since, the X direction coil being the least powerful in terms of current conversion, put a limitation on the output range of 6mT as the magnitude of rotating fields, considering safer operation of DC currents in the coils. This setup was later assembled with the optical components to forming the overall tracking apparatus.

3.2 Optical components and setup interface

3.2.1 Assembly of optical components in the coil setup

Figure 16: Helmholtz setup with optical components and a designated working volume, interfaced independently for magnetic actuation and tracking as an open loop control system

The Helmholtz coils were assembled with the optical components, namely

laser source, camera and lens as the stereo holography setup. The laser source

used here was a diode laser at central wavelength of 635nm red light, which in

turn was collimated with a plano convex lens of 75mm as focal length. Two such

laser sources were used in adjunct with two CMOS cameras placed orthogonal

3.2 Optical components and setup interface 28

to each other as shown in fig. 16. The first camera (shown to left in the figure) had a screen resolution of 1920x1200 pixels with a pixel size of 4.8µm per pixel while that of the second camera (shown to the right) were 2048x2048 and 5.5µm per pixel respectively. A double sided polished cuvette manufactured by Thor- Labs was used as the work space providing a working volume of about a cubic centimeter when illuminated by the two lasers.

The coil setup was interfaced with a computer via a current amplifier, while the two cameras were interfaced with another computer by means of a user interface providing requisite API libraries as shown in the fig. 16, thus making the overall setup an open loop control system. An illustration of the setup described above is shown in fig. 17.

The pixel size of each camera gave the intrinsic camera matrix for the respective camera, which was further used in calibrating the extrinsic camera matrix for the two cameras as a coupled system as described in the following section.

Figure 17: Illustration of the Helmholtz coil setup attached with optical compo-

nents

3.3 Camera calibration procedure 29

3.3 Camera calibration procedure

The most ubiquitous means of camera calibration in stereo mode was mapping and correlating a uniform 2-D pattern like chessboard to obtain intrinsic or ex- trinsic parameters of the camera. Though a CMOS camera, as mentioned in the setup description, could be easily modeled intrinsically unlike a pinhole cam- era, an intelligent calibration procedure was required to determine the extrinsic camera parameters for the given arrangement of cameras. This was necessary to provide a basis for triangulating the 2-D information obtained from the two views in corresponding cameras into a global 3-D coordinate.

However, given a working volume for monitoring microscopic objects, it was cum- bersome to design and employ a stencil like chessboard but on a micrometer scale.

Therefore, in order to overcome this limitation an experiment was performed by recording precise movements of a microbead and taking multiple such points in a defined volumetric space within the working volume. For this experiment, a precise micro-positioning tool called smarAct was used to traverse a spheri- cal magnetic bead of sub-mm order dimension, in a defined cube of size 4mm ³ , at 1mm step size thereby making 125 closely spaced and uniformly distributed points. The smarAct positions were recorded as reference positions in 3-D space which acted as the world coordinate frame. Images were captured at each of these positions by the two cameras arranged w.r.t. world coordinate frame as shown in fig 18a. These images were post processed by the tracking algorithm for centroid detection of the microbead in the images, thereby giving the corresponding pixel data in the camera frame of reference. Hence, the mapping between the smarAct positions and corresponding camera pixel positions in the two cameras comprised of the experimental data for extrinsic calibration.

Given the world coordinate frame as [X,Y,Z ] ^T , the pixel coordinates for the first camera as [U ₁ ,V ₁ ] ^T , for the second camera [U ₂ ,V ₂ ] ^T and their respective intrinsic matrices as S 1 , S 2 respectively, the overall equation of the camera matrix in augmented form could be written as,

U =











 U ₁ V 1

U ₂ V ₂













=

"

S 1 0 0 S ₂

#

∗

"

R 1

R ₂

#

∗





 X Y Z





 +

"

S 1 0 0 S ₂

#

∗

"

T 1

T ₂

#

3.3 Camera calibration procedure 30

(a) (b)

Figure 18: (a) showing the world coordinate frame and the two orthogonal camera with their respective axes aligned to X and Y axis, (c) showing the Helmholtz setup with two cameras and a smarAct along side for calibration experiment

where the intrinsic matrix S for each camera could be defined as,

S _i =

"

p _x 0 0 0 p _y 0

#

where p x , p y being the respective pixel sizes of the camera, and [R 1 ,R 2 ] ^T and [T ₁ ,T ₂ ] ^T represent the rotational and translational matrices for the combined camera system. The two latter pairs of matrices are the overall camera transfor- mation matrix and form the extrinsic parameters of the camera. The calibration process was employed to determine the exact values of these camera transforma- tion matrices to used for triangulation of data points attained in tracking process thereby projecting them back to the real space.

3.3.1 Newton-Raphson Inverse Jacobian method

The extrinsic camera calibration was performed by using the data points ob-

tained from calibration experiments with smarAct (as shown in fig. 20a) and

using them to find the camera transformation. A dexterous method of convex

optimization like Newton Raphson inverse Jacobian could be applied to calculate

this transformation matrix, given a rough estimate of the setup parameters. In

this process, the experimental data mentioned above was used to numerically

compute first order partial derivatives i.e. Jacobians, of the camera pixel coor-

3.3 Camera calibration procedure 31

dinates as variables w.r.t. transformation parameters. Fig 19 describes the work flow of algorithm used to compute camera extrinsic parameters. Firstly, an ini- tial guess of parameters to be computed were fed as seed to algorithm. Secondly, the camera matrix equation was re-evaluated to generate calculated pixels values (U = [U ₁ V ₁ U ₂ V ₂ ] ^T ). Therefore, the difference of calculated and calibrated pixel values were used to evaluate an error function, which was further combined with inverse of Jacobians described earlier, to produce values of the respective extrinsic parameters. Moreover, the difference of these pixel values was also used to estimate a cost function that described the convergence of the method. This process was performed in an iterative fashion until a good convergence indicated by the cost function. The solution thus obtained as a result of convergence gave a rather more accurate values of the extrinsic parameters.

Figure 19: Flow chart of Newton-Raphson inverse Jacobian algorithm for extrin- sic camera calibration

Initial guess parameters A makeshift measure of the camera transfor- mation matrix, comprising of translation and rotation components, could be ap- proximately estimated with the setup information. Given the the two cameras were employed in orthogonal directions, the relative transformation could be well assumed to the 90 ^o to the assumed world coordinate system in the calibration experiments. Further, based on the angle axis representation of SO3 rotation groups, this rotation matrix could be condensed into a three element vector as shown below,

R _i = exp(Skew h

Ω _x Ω _y Ω _z i

)

3.3 Camera calibration procedure 32

The rotation vector in angle-axis form could be used to obtain the 3x3 R matrix using the traditional Rodrigues formula. It used a truncated exponential series expansion of the skew-symmetric matrix formed by the elements of rotation vec- tor. Moreover, given the center to center alignment of the work space with respect to the camera, the translation parameters could be roughly approximated as half the total camera resolution in the respective directions. However, the depth translation of individual camera could be taken as a zero thereby diminishing the overall translation matrix to a two element vector.

T _i = h

T _x T _y T _z i

(a) (b)

Figure 20: (a) showing data points taken in a volumetric space of 4mm ³ using smarAct suspected magnetic bead, (b) showing the corrected fit obtained after Newton-Raphson inverse Jacobian method

Thus, combining the guess parameters for the two cameras, the overall camera equation could be re-evaluated as shown below,











 U 1

V 1

U ₂ V ₂













=

"

S 1 0 0 S ₂

#

∗

"

R 1

R ₂

#

∗





 X Y Z





 +

"

S 1 0 0 S ₂

#

∗

"

T 1

T ₂

#

Based on the calculated pixel values (U ) and the experimental data (U ), an