Noise modelling for high-throughput super-resolution microscopy

IN

DEGREE PROJECT ELECTRICAL ENGINEERING, SECOND CYCLE, 30 CREDITS

STOCKHOLM SWEDEN 2018 ,

Noise modelling for high-

throughput super-resolution microscopy

XAVIER CASAS MORENO

KTH ROYAL INSTITUTE OF TECHNOLOGY

Sammanfattning

Super-uppl¨ osta fluorescensmikroskop ¨ ar ett framv¨ axande f¨ alt inom av- bildningsteknik som syftar till att ¨ overvinna di↵raktionsgr¨ ansen hos ljus med hj¨ alp av tillst˚ ands¨ overg˚ angar hos fluorescenta molekyler. Det finns idag stora utmaningar inom f¨ altet som just nu ofta begr¨ ansas av sm˚ asynf¨ alt, l˚ angsamma avbildningstider och l˚ ag bildkvalitet.

Gruppen f¨ or avancerad bio-imaging vid Science for Life Laboratory i Stockholm har nyligen utvecklat ett mikroskop kallat MoNaLISA (Molecu- lar Nanoscale Live Imaging with Sectioning Ability). Detta ¨ ar ett mikroskop som n˚ ar en h¨ og spatiell uppl¨ osning (45-65 nm) med l˚ aga ljusintensiteter (kW cm

), l˚ anga inspelningar (40-50 bilder) och ett stort synf¨ alt (50x50 mu m

) utan att kompromissa avbildningshastigheten. En ny version av mikroskopet ¨ ar under utveckling och syftar till att uppn˚ ah¨ ogre genomfl¨ ode (dvs st¨ orre synf¨ alt).

MoNaLISA, likt m˚ anga andra superuppl¨ osta mikroskop, anv¨ ander en s˚ akallad sCMOS (scientific Complementary Metal-Oxide Semiconductor) kamera vilket ger h¨ og kvante↵ektivitet och snabb utl¨ asningstid. Kameran introducerar emellertid flera k¨ allor till deterministiskt och stokastiskt pix- elberoende brus p˚ agrund av kamerans elektroniska struktur.

Under det experimentella arbetet med det nya mikroskopet noterades en d˚ aligt dokumenterad typ av brus till f¨ oljd av de belysningspulser som anv¨ andes. Vi valde att kalla brustypen f¨ or Trapped Charge Noise (TCN).

I kombination med andra klassiska typer av brus som producerats av sCMOS-kameran, minskade kvaliteten p˚ abilderna som togs med MoNaL- ISA avsev¨ art.

Projektet syftar till att demonstrera de fysikaliska grundprinciperna f¨ or superuppl¨ ost fluoroscencemikroskopi och framf¨ or allt konceptet bakom MoNaLISA, samt att utf¨ ora experimentellt arbete med den h¨ oga genom fl¨ odesversionen av installationen. F¨ or att f¨ orb¨ attra bildkvaliteten hos mikroskopet kr¨ aves ocks˚ aatt vi modellerade de olika brustyperna och utvecklade en algoritm f¨ or att minimera den destruktiva inverkan av dessa.

Resultaten avseende experimentellt arbete inneh˚ aller bilder p˚ a MoN- aLISA mikroskopet samt rekonstruerade bilder av cellul¨ ara strukturer m¨ arkta med det v¨ axelvis fluoroscenta proteined rsEGFP (reversibly switch- able Enhanced Green Fluorescent Protein).

F¨ or att modellera TCN formuleras en serie fr˚ agest¨ allningar, f¨ oljt av en pixel- och belysningsberoende funktionsanpassning av brusets medelv¨ arde som en summa av exponentiella funktioner. Anpassningsparametrarna lagras och anv¨ ands i algoritmen, som har mikroskopets r˚ adata som indata.

Algoritmen tar ocks˚ ah¨ ansyn till kamerans andra brustyper s˚ asom Fixed Pattern Noise (FPN).

Bildkvaliteten hos b˚ ade r˚ adatan och rekonstruerade data f¨ or MoNaL-

ISA f¨ orb¨ attrats v¨ asentligt efter till¨ ampningen av algoritmen.

Abstract

Super-resolution fluorescence microscopy is an emerging imaging field that aims at breaking the di↵raction barrier of light based on state transi- tion in fluorescent molecules. Current challenges in the existing approach are to achieve large field of views, fast recordings and increasing the image quality.

The Advanced Bio-Imaging group at the Science for Life Laboratory in Stockholm invented the Molecular Nanoscale Live Imaging with Section- ing Ability (MoNaLISA), a microscope that reaches high spatial resolution (45-65 nm) with low light intensities (kW cm

), prolonged (40-50 frames) recordings, and a large field of view (50x50 µ m

) without compromising the recording speed. A new version of the microscope is under develop- ment, aiming at achieving high-throughput (i.e., larger field of view).

MoNaLISA, as well as most of the super-resolution techniques, incor- porates a scientific complementary metal-oxide semiconductor (sCMOS) camera in the detection path, which provides high quantum efficiency and fast readout time. However, it introduces several sources of deterministic and stochastic pixel-dependent noise due to the electronic structure of the camera.

During the experimental work, a rarely documented type of noise was encountered due to the characteristics of the illumination scheme and the detection characteristics, which we called the Trapped-Charge Noise (TCN). In conjunction with other classical types of noise produced by the sCMOS camera, it considerably decreased the quality of the images taken with MoNaLISA.

This project aims at demonstrating the physical foundations of super- resolution imaging and the MoNaLISA setup, as well as performing exper- imental work with the high-throughput version of the setup, at modelling the noise and creating a signal processing algorithm for noise reduction directly applied to the raw data acquired from the microscope.

The results regarding experimental work contain images of the MoN- aLISA setup alignment as well as reconstructed data of cellular struc- tures tagged with the reversibly photoswitchable green fluorescent protein (rsEGFP).

In order to model the TCN, a series of hypothesis are formulated, followed by a pixel- and illumination-dependent fitting of the noise mean as a sum of exponential functions. The fitting parameters are stored and utilized in the algorithm, which has the microscope raw-data as input.

The algorithm takes into consideration the fixed-pattern noise (FPN) of the camera as well.

The image quality of both the raw and reconstructed data of MoNaL-

ISA improved substantially after the application of the algorithm.

Acknowledgements

During this period there are many people I am thankful for. For the ones that were already there and for the fantastic people I have met, for the academic knowledge I have gained and for the things I have learned.

I am very lucky and grateful for your patience and support. All of you have always believed in me and I am thankful for that.

I would like to thank my supervisor Ilaria Testa for welcoming me to the Science for Life Laboratory and guiding me along the way, her passion for science and research is fascinating and has motivated me to find my career path. Thank you for your patience, your knowledge, your advice and your support.

Thank you Francesca Pennacchietti, Andreas Boden and Federico Bar- baras, without you this work would not have been possible.

Thanks as well to every single one of my colleagues that have helped me enormously academically and personally. Thanks for being like a small family Francesco, Martina, Jonatan, Elham, Giovanna, and all the oth- ers. I would like to thank Luciano as well for encouraging me find this opportunity.

I would like to thank Joakim Jald´en for his patience and understand- ing, his professional advice, expertise and experience.

Thanks to all my family and friends, your support has and will always

guide me along the way.

Acronyms

CCD Charge-coupled Devices.

CDS Correlated Double Sampling.

FPN Fixed-Pattern Noise.

GPU General Processor Unit.

MLA Microlens Array.

MoNaLISA Molecular Nanoscale Live Imaging with Section Ability.

PBS Polarizing Beam Splitter.

PSF Point Spread Function.

RESOLFT Reversible Saturable Optical Linear Fluorescence Transitions.

ROI Region of Interest.

rsEGFP Reversibly Switchable Enhanced Green Fluorescent Protein.

rSFP Reversibly Switchable Fluorescent Proteins.

sCMOS Scientific Complementary Metal-Oxide Semiconductor.

SNR Signal-to-Noise Ratio.

STED Stimulated Emission Depletion.

TCN Trapped-Charge Noise.

TRN Telegraph Noise.

WF Widefield.

List of Figures

1 Fluorescent molecule: state transitions . . . . 9

2 STED illumination . . . . 10

3 RESOLFT illumination scheme . . . . 11

4 MoNaLISA illumination patterns . . . . 12

5 MoNaLISA reconstruction . . . . 13

6 CCD structure . . . . 14

7 CMOS structure . . . . 14

8 MoNaLISA high-throughput optical scheme . . . . 17

9 MoNaLISA high-throughput setup . . . . 18

10 MoNaLISA high-throughput patterns . . . . 18

11 Di↵raction grid . . . . 19

12 Micro Lens Array (MLA) . . . . 19

13 Detection camera pixels . . . . 20

14 Deactivation pattern field of view 68.7x68.7 µm ² . . . . 20

15 Activation pattern field of view 68.7x68.7 µm ² . . . . 20

16 Excitation pattern field of view 68.7x68.7 µm ² . . . . 21

17 Overlap activation and deactivation patterns . . . . 21

18 Overlap excitation and deactivation patterns . . . . 22

19 Raw data in MoNaLISA . . . . 22

20 Reconstructed data in MoNaLISA . . . . 23

21 Fluorescence bar . . . . 25

22 Experiment A . . . . 26

23 Experiment B . . . . 26

24 Experiment scheme C, where there is no fluorescence previous to exposure. The laser pulse is applied only during the exposure of the camera. . . . 26

25 Exp. A, raw data . . . . 27

26 Exp. A, image histogram . . . . 27

27 Exp. A, variance and mean . . . . 28

28 Exp. A, saturation curve . . . . 28

29 Exp. A, fitting curve . . . . 29

30 Exp. A, fitting parameters . . . . 29

31 Experiment B (left) and C (right), saturation curves . . . . 30

32 Experiment A, fittings . . . . 30

33 Histograms of all experiments . . . . 31

34 Standard deviation as a function of k . . . . 32

35 Diagram for noise removal algorithm . . . . 33

36 Noise correction in raw data . . . . 34

37 Noise correction in raw data . . . . 34

38 Zoomed noise correction in raw data . . . . 35

39 O↵set correction . . . . 35

40 Noise correction in reconstructed image . . . . 36

41 Noise correction in reconstructed image . . . . 36

42 Noise correction in reconstructed image . . . . 37

Contents

1 Introduction 7

2 Background 9

2.1 Super-resolution fluorescence microscopy . . . . 9

2.2 RESOLFT . . . . 10

2.3 MoNaLISA . . . . 11

2.3.1 Image reconstruction . . . . 12

2.4 Scientific cameras . . . . 13

2.4.1 Noise in CMOS cameras . . . . 15

3 Results 17 3.1 Experimental Setup . . . . 17

3.1.1 Deactivation pattern . . . . 18

3.1.2 Activation and excitation patterns . . . . 19

3.1.3 Detection . . . . 19

3.1.4 Alignment . . . . 19

3.1.5 Image data . . . . 21

3.2 Trapped-Charge Noise (TCN) . . . . 23

3.2.1 Signal model . . . . 24

3.2.2 Hypothesis . . . . 24

3.2.3 Characterization experiments . . . . 24

3.2.4 Experiment Results . . . . 26

3.2.5 Algorithm for noise removal . . . . 31

3.2.6 Results in raw data . . . . 33

3.2.7 Results in reconstructed data . . . . 33

4 Discussion 38

5 Conclusions and Outlooks 39

1 Introduction

The di↵raction of light limits the resolution of conventional light microscopes to about half the spatial wavelength, typically around 200 nanometers [1]. It was not until 1994 that Stefan Hell demonstrated that it is possible to overcome the di↵raction limit by taking advantage of state transitions in fluorescent molecules [2]. This concept introduced a new research field, often called nanoscopy or super-resolution fluorescence microscopy. To date, many imaging techniques have been investigated, giving rise to scientific discoveries [3].

In a fluorescence microscope, the specimen is illuminated with light at a certain wavelength which is absorbed by fluorophores bound to the sample, causing them to emit light of a higher wavelength. The fluorescence is captured by light detectors and an image of the specimen is extracted.

Super-resolution fluorescence microscopy exploits the photoswitching char- acteristics of fluorophores either in a stochastic or deterministic manner in order to achieve higher resolutions.

In particular, Simulated Emission Depletion microscopy (STED) [2]

is a deterministic technique that achieves super-resolution images by strategic fluorophore deactivation. In STED microscopy a focused beam of light excites the fluorophores located at the center of a focal spot while another beam is used for deactivating their surroundings. By scanning both beams across the sample, a super-resolved image is reconstructed. However, a very intense laser power is required in order to deactivate the molecules to the ground state by stimulated emission.

In order to utilize lower light doses, Reversible Saturable Optical Linear Fluorescence Transitions (RESOLFT) [4], [5] uses reversible switchable fluorescence probes, which photoswitch between a fluorescent and a dark state.

The lifetime of the states is much longer (µs-ms) compared to the STED states (ns). Thus, the molecules can be deactivated by applying much less intensity (W/cm ² ) than in STED (GW/cm ² ).

The imaging scheme in RESOLFT consists of three consecutive illumina- tions: first, activating the molecules with a di↵raction-limited spot in a specific location of the sample, followed by the deactivation of the periphery with an en- gineered light pattern similar to a ’doughnut’; finally another di↵raction-limited spot illuminates and excites the molecules that are still active.

The process of scanning a single beam over the sample is slow, especially in a large field of view where many pixels need to be recorded. Additionally, the increased lifetime of the RESOLFT state transition lead to extended pixel dwell time witch further slows down the recording. Parallelized RESOLFT techniques [6], [7] have considerably increased the speed of a point-scanning recording by introducing light patterns composed by thousands of focal spots instead of one.

Based on the principles of Parallelized RESOLFT, Testalab at the Science

for Life Laboratory in Stockholm developed the Molecular Nanoscale Live

Imaging with Sectioning Ability (MoNaLISA) [6]. This setup aims at

increasing the field of view without compromising the speed of recording. In

order to do so, the illumination area as well as the pixel size of the detection is

enlarged. As a consequence, new challenges regarding not only physical imaging

but also image reconstruction need to be overcome. In this project, I worked on

the setup and operation of the MoNaLISA microscopes. Furthermore, I focused

on the data analysis with a special focus on the noise and its influence on the quality of the image reconstruction.

Scientific Complementary Metal-Oxide Semiconductor (sCMOS) cameras are widely used in super-resolution microscopy as light detectors since they accelerate data acquisition and enlarge the field of view. However, they introduce pixel-dependent noise because of their electronic architecture. The paper in [8] provides a framework to overcome the noise challenges in single- molecule switching nanoscopy, a stochastic super-resolution technique, by mod- elling the o↵set, gain and variance of each pixel. To our knowledge, there is no similar study regarding STED or RESOLFT microscopy, and we identified open challenges that have not yet been covered by previous research.

More specifically, due to the MoNaLISA illumination scheme, a new type of noise was encountered, which we named Trapped-Charge Noise (TCN). To our knowledge, the TCN has only been noticed in [6] and [7], but no study has been performed in order to understand and model its behavior. Other types of noise such as the Fixed-Pattern Noise (FPN) and Telegraph Noise (TN) are also analyzed in this report.

During the experimental work, the image quality was a↵ected by the noise hampering the quantification of cellular fine structures. More specifically, the Fixed-Pattern Noise (FPN) and the Trapped-Charge Noise (TCN) deteriorated the raw data, thus creating square e↵ects in the reconstructed data.

The project has two main objectives:

• To deepen the understanding of the physical theory and image formation behind MoNaLISA super-resolution imaging, including the challenges re- garding the setting-up and operation of the experimental setup.

• To improve image quality, model and subtract the noise by applying a noise removal algorithm directly in the raw data. Both the quality of the raw and reconstructed data benefit from the application of the algorithm.

This project is carried out at the Advanced Optical Bio Imaging group at the Science for Life Laboratory and is centered in both the physical and image processing challenges in the MoNaLISA microscopy technique.

This report is structured as follows: Section. 2 describes the necessary back-

ground to understand the physical concepts behind STED, RESOLFT, and the

MoNaLISA implementation. The image reconstruction and the CMOS camera

noise is described. Section. 3 focuses on the experimental activity, followed

by the noise analysis and modelling. Furthermore, an algorithm for noise re-

duction has been developed and applied to the data, providing imaging with

higher quality. The Discussion of the results follows in Section. 4. Finally, the

Conclusions and Outlooks are discussed in Section. 5.

2 Background

2.1 Super-resolution fluorescence microscopy

Light microscopy allows us to see beyond what our eyes can observe, and many discoveries have taken place since its invention, such as the cells as basic units, bacteria and mitochondria [9]. With light microscopy, one can look inside a living cell and observe the di↵erent processes and interactions at a molecular level while being minimally invasive.

The idea behind fluorescence microscopy is that scientists can attach a flu- orescent molecule to the specimen of interest and then be able to observe it since it will produce light when excited. This is possible because of the inner characteristics of a fluorophore, which has at least two states: a ground state and an excited fluorescent state. Whenever a fluorophore absorbs a photon, the molecule raises from ground to excited state and releases a photon with higher wavelength. Since the wavelength is shifted, the excitation and fluorescence can be easily separated (see Fig. 1).

ON OFF

Figure 1: Ground (S 0 ) and excited (S 1 ) state. A fluorescent molecule is excited with illumination light (S 0 > S 1 ). After some nanoseconds, it goes back to S 0

and releases a photon.

The optical resolution of a microscope is defined as the measure of how close we can distinguish di↵erent features. For example, if a set of fluorescent molecules are closer than 200 nanometers, a traditional microscope will not be able to discern them. This is because, according to Abbe, the objective lens of a microscope will focus the light down to a di↵racted spot of light, that will depend on the spatial wavelength and the numerical aperture of the objective lens, typically around 200 nanometers wide and 500 nanometers along the optical axis.

In super-resolution microscopy, one can physically control the properties of the fluorescent molecules so that some are detectable while others are not.

In this way, not all the molecules within a di↵racted spot emit light at the same time and therefore they can be distinguishable. Since the fluorescence molecules have di↵erent states, one can take advantage of state transition in order to silent molecules in time to increase the resolution of the microscope.

This is possible because, for the molecules to emit fluorescence, they have to be in their excited state so when a fluorescent molecule is in a ground state it doesn’t produce light.

In STED microscopy, a beam of light activates molecules from the ground to

the excited state, thus obtaining a di↵racted limited spot of light. Consecutively,

another beam of red-shifted light induces stimulated emission, meaning that the molecule descends to the ground state by absorbing a photon. This molecule is then not detectable. The saturation intensity is defined as the threshold where the emitted fluorescence decreases by a factor of ¹ _e . The second beam used in STED has a ring or ”doughnut” shape, where the center intensity is below the saturation but not the surroundings. The more intensive the beam, the smaller the center. In this way, only the molecules from the center of the spot produce fluorescence, and by scanning both beams along the sample a super- resolution version of the specimen can be extracted. Fig. 2 [Hell & Wichmant, Optical Letters (1994)] illustrates the superimposition of the green- (activation) and red-shifted (deactivation) beams.

Figure 2: STED imposed green- (activation) and red-shifted (deactivation) beams. The yellow molecules are the ones emitting fluorescence.

2.2 RESOLFT

STED is a powerful super-resolution technique in which stimulated emission is exploited in a deterministic manner to switch ON/OFF molecules step by step, thus making structures discernible within the di↵racted spots. This is possible because the fluorescent molecules have at least two states: the excited (ON) and ground (OFF) state. However, once a molecule has transitioned to the excited state, it normally will come back to the ground in nanoseconds, thus the excited state is very short in terms of lifetime. Consequently, a very high intensity has to be applied in order to make the STED process efficient in deexciting molecules.

The use of high intensity during imaging can be harmful to the cells, and therefore another super-resolution microscopy technique aiming at reducing the light doses applied to the sample is needed.

Some molecules, such as the reversibly Switchable Fluorescent Proteins (rSFP), contain more states apart from the fundamental ground-excited states, called triplet or long-lived dark states. Instead of having a lifetime of nanoseconds, they switch slowly in the microseconds order.

The threshold intensity escalates inversely with the lifetime of the states.

Therefore, much lower intensities can be applied to deactivate the molecules.

This is the concept of reversible saturable optical linear fluorescence transitions (RESOLFT).

The imaging scheme in RESOLFT is slightly di↵erent to the one in STED.

First, a di↵racted illumination beam with the wavelength corresponding to the activation of the rSFP molecules is applied in order to send the molecules to the activation state (typically 405 nm). Then, a ring-shaped deactivation spot is applied in order to deactivate the surroundings (488 nm). Finally, another di↵raction limited spot is applied in order to excite the molecules in the center (488 nm). The reason that the deactivation and excitation wavelengths are the same is that, in order to deactivate the molecules, they will be excited for a certain time and power so that it can be ensured that they will eventually tran- sition to the dark state. Fig. 3 illustrates the illumination scheme in RESOLFT when using rSFPs.

Read-Out

ON OFF time

Fluorescence

Figure 3: Illumination scheme in RESOLFT, consisting in activation (ON), de- activation (OFF) and excitation (Read-Out) illumination pulses. The molecules in yellow are the ones a↵ected by the deactivation and the excitation, and the fluorescence over time is shown.

The process of scanning point by point the multiple beams along the sample becomes slower when lower intensities are applied due to the increased pixel dwell time.

Parallelized versions of RESOLFT aim at, instead of scanning the beam along the sample, di↵erent multi-point patterns are engineered in order to reduce the scanning steps and increase the speed of the recording. In [7], a living cell is recorded within two seconds with more than 100,000 intensity minima in parallel.

2.3 MoNaLISA

Di↵erent techniques within the rapidly evolving field of super-resolution fluo- rescence microscopy have achieved great results in terms of spatial resolution, speed and illumination light intensity reduction.

The Molecular Nanoscale Live Imaging with Sectioning Ability (MoNaLISA) is a rSFP-based parallelized RESOLFT microscope with high spatial resolution of about 45-65 nm and large field of view (50 x 50 µm ² ). Since minimal light doses are applied, prolonged recordings are enabled (40-50 frames) at 0.3-1.3 Hz.

[7] presents a widefield (WF) parallelized RESOLFT implementation that reaches high-resolution of large field of views and relatively fast recordings.

While the ”OFF” pattern consists of an array of intensity minima, the ”ON” and

read-out illuminations are uniform. Therefore, there is unnecessary switching of the molecules as well as out-of-focus light from other planes in the optical axis. MoNaLISA reduces the information from other planes by confining the activation and excitation beams using multi-foci patterns created by Micro Lens Arrays (MLAs).

The setup is based on three illumination steps and a scanning scheme, as shown in Fig. 4 (extracted from [6]). First, the molecules are switched ON, followed by the deactivation of the center surroundings and then the excitation step, which is synchronized with the camera read-out. This procedure is done in parallel by using multi-foci patterns (”ON” and excitation) and arrays of intensity minima (”OFF” switching).

Figure 4: MoNaLISA illumination patterns. In the left, the three 1-D patterns are displayed, as well as the detected signal. In the right, the 2-D pattern over a field of view exemplifies how a molecule is read out.

The illumination patterns employed in MoNaLISA are optimized in shape and periodicity in order to read out the fluorescence of the reversible switchable Fluorescence Proteins (rsFP). The On-switching (405 nm) and read-out (488 nm) multi-spot patterns are created by microlens arrays (MLA) and composed of N individual foci with periodicity p mf . The OFF-switching (488 nm) features standing waves with intensity minima of periodicity p sw . The three patterns are aligned in order to obtain super-resolution when the OFF minima are located at the same place as the multi-foci maxima.

In particular, the first version of MoNaLISA features N = 5436, with peri- odicity p mf = 750 nm and p s,w = 250 nm.

2.3.1 Image reconstruction

The data obtained in the image acquisition process of the MoNaLISA setup, also called raw data, consists of a stack of frames. Since the biological sample is scanned line by line, a frame is acquired every scan step.

Each frame contains a series of foci, corresponding to the readout pattern

for a determined sample shift. The reconstruction algorithm is a MATLAB

software that outputs a super-resolution image by quantifying the in-focus light

in each step and assigning it to the higher resolution final image.

Fig. 5 illustrates the reconstruction algorithm having a stack of raw data frames (upper part) as input and the super-resolution image (lower part) as output.

At the top of the figure, the stack of frames is represented. Since the sample is scanned in x and y, the first M images correspond to the shifts in the x direction of the sample at y 0 . Then, the sample is shifted in y one step size and the next M frames are stored. This process is repeated until the region of interest (ROI) is fully scanned.

The final image of higher resolution can be divided into di↵erent sub-squares, corresponding to the N di↵erent multi-foci. For each square i, j (illustrated in the lower-right part of the image), the quantified emission from the foci i, j of each of the raw-data frames is stored taking into consideration the shift between the frames.

Frame 00 Frame 01 Frame 0M Frame N0 Frame N 1 Frame NM

… … …

x

y

x scan step y scan step x scan step

…

…

… From 00 to 0M

From N0 to NM

Figure 5: MoNaLISA reconstruction scheme. The upper part shows the stack of frames corresponding to a line-by-line scan of the samples. In the lower part, the reconstructed image is drawn composed by a set of subsquares. Each subsquare consists of the contribution of a foci from each frame.

The complete reconstruction scheme is detailed in the Supplementary infor- mation of [6].

2.4 Scientific cameras

Scientific cameras are used as photodetectors in super-resolution microscopy in order to obtain an image of the specimen in 2D (and in some cases 3D).

The commonly used cameras are charge-coupled devices (CCDs) and scientific complementary metal-oxide semiconductors (sCMOS).

Fig. 6 shows the internal architecture of a CCD image sensor. First, as

described in [10], the incoming photons are converted to charge which is accu-

mulated by photodetectors, represented in the image as black squares, during

the exposure time of the camera. Then, the charge is placed first vertically in

the CCD columns, and then horizontally in a row that is then passed to an amplifier that sequentially converts the charge to voltage pixel by pixel.

Since the process is sequential and there is a shared amplifier circuit, the noise is pixel independent and therefore uniform over the camera chip, and the Signal to Noise Ratio (SNR) is high. However, the readout process is slow since the process is performed sequentially. Furthermore, CCD cameras consume high power.

… … … … … …

…

Figure 6: CCD structure

Just like in CCD image sensors, photons are converted to charge and stored in photodetectors for each camera pixel in CMOS cameras. However, each pixel contains an amplifier circuit as well (see Fig. 7) so the charge is converted to voltage independently and in parallel. Then, each column shares amplifier circuits to further process the voltage.

… … … …

Figure 7: CMOS structure

Since the process is performed in a parallel manner, the readout speed is much higher than in CCD sensors. Furthermore, the power required in CMOS cameras is also lower than in CCD image sensors. However, the independent processes corrupt the image with pixel dependent noise due to circuit di↵erences in every sensor.

sCMOS cameras are widely used in microscopy due to their accelerated data acquisition, increased quantum efficiency as well as the large field of views they record. Each pixel has a di↵erent gain, variance, and o↵set value, which makes it difficult to characterize the noise.

A recent paper [8] provides an extended overview of the noise that deteri-

orates the image quality in stochastic super-resolution fluorescence microscopy

using sCMOS cameras. They state that sCMOS cameras provide an efficiency of 73% at 600 nm, while imaging a large field of view in faster recordings. Dif- ferent algorithms are presented in the paper in order to estimate the o↵set, gain and variance of each pixel and include the characteristics in the reconstruc- tion process. The noise is modeled as a combination of Poisson shot noise and Gaussian readout noise. However, the algorithms are very time consuming and implemented in a highly parallelized graphical processor unit (GPU).

To our knowledge, there has not been a characterization of CMOS cameras in deterministic super-resolution microscopy (STED, RESOLFT), where the challenges in the reconstruction process are di↵erent due to the fact that the sample is scanned in order to produce a high-resolution image and the position of the molecules is already known.

2.4.1 Noise in CMOS cameras

The noise in CMOS cameras can be classified into two di↵erent types:

• Pattern noise, which produces some visible structures that are deter- ministic and nearly constant in time.

• Random noise, temporally random and can be reduced by averaging over frames.

Since each pixel in a CMOS camera contains one or more transistors, which provide gain and bu↵ering functions, fixed pattern noise (FPN) is one of the biggest challenges in CMOS image sensors.

The FPN is a deterministic noise and usually has two components: a pixel- dependent value, since each pixel has di↵erent circuits; plus another component that depends on the structure of the camera pixels, thus creating patterns typ- ically with stripes due to the shared amplification circuits.

Several techniques have been developed in order to reduce this type of noise on the chip of the camera. For example, Correlated Double Sampling (CDS) is a common technique where the signal is captured twice and in di↵erent conditions, and it reduces the e↵ects of the noise.

Temporal noise causes random variations in the signal that vary over time, and has several sources:

• Shot noise is a temporal random noise created by the characteristic ar- rival of photons in the sensors, which follows a Poisson distribution.

• Readout noise is a Gaussian-distributed noise originated in the sensor when the pixel value is extracted.

• Flicker noise or Telegraph Noise is caused by defects on the oxide that create traps for the accumulated charge. The result is that, under low-light conditions, a subset of the pixels oscillate between more than one center value.

There can be other sources of temporal noise such as the reset noise in the photodetectors and thermal noise from the electronic circuits.

The camera employed in the MoNaLISA setup has already built-in calibra-

tions for reducing all the cited sources of noise. However, the FPN is still a

concern in the background areas of an image. Furthermore, a new type of noise

has been encountered, the Trapped-Charge Noise (TCN), which will be

described in Section 3.

3 Results

3.1 Experimental Setup

The setup described in this section is a high-throughput version of MoNaLISA, which is under development, and aims at increasing the field of view, provid- ing a higher SNR and making the recording faster. In order to do so, some characteristics are di↵erent than in the first version of MoNaLISA:

• The illumination module features higher power and bigger beam size in order to increase the sample area to have a higher throughput.

• The camera pixel size is increased, therefore the area recorded by the camera is also larger.

The setup is shown in Fig. 8 and Fig. 9. There are three illumina- tion branches: deactivation (488 nm laser), excitation (473 nm) and activation (405 nm). The sample is placed in the objective, where the patterns are imaged, and the emitted fluorescence is captured by a CMOS camera (Hamamatsu Orca Flash 4.0 v3).

100 300

300 180

150 250

300 130

Figure 8: MoNaLISA high-throughput optical scheme

As explained in section 2.3, the activation and excitation illumination pat- terns are created using MLA. In this setup, the excitation and deactivation wavelengths are not exactly the same in order to optimize the power when the di↵erent lasers are combined and imaged to the sample.

The deactivation pattern is created with two perpendicular di↵raction grids merged by a polarizing beam splitter (PBS). A di↵raction grid is an optical component that di↵racts an incoming beam into several beams in di↵erent di- rections, and a PBS can either split or combine two beams of di↵erent polariza- tion. Therefore, these components create in the sample a 2D sinusoidal pattern with periodic intensity minima.

The lens right after each laser collimates the light. In other words, it makes

the light rays parallel to each other.

Figure 9: MoNaLISA high-throughput setup

The optical components Di03-R473, Di03-R442, and Di03-R488 are dichroic mirrors, which have di↵erent transmission and reflection properties depending on the wavelength. For example, the Di03-R473 only transmits light of wave- length higher than 473 nm.

The other lenses have the function of adjusting the periodicity of the pat- terns in order to have the minima of the deactivation pattern aligned with the activation and readout (see Fig. 10).

nm nm

nm

Figure 10: 1-Dimensional plot of the three illumination light patterns

3.1.1 Deactivation pattern

The deactivation pattern is created with a di↵raction grid of size 15 x 15 mm ² and a line period of 10 µm. An interference pattern is applied to the sample by using two perpendicular di↵raction grids and a PBS.

In order to modify the periodicity of the pattern so that it reaches the desired value in the sample plane, a telescope of two lenses (100 mm and 300 mm) is placed after the PBS.

The periodicity after the telescope formed by the lenses is: d 1 = 10µ m ³⁰⁰ ₁₀₀ ¹ ₂ = 15µ m. The factor of two comes from a mask placed in between the lenses.

After going through the objective, which has a magnification of 63x, the periodicity is demagnified, thus having the following value:

d OF F = 15µ m ₆₃ ¹ = 238 nm. Therefore, the deactivation pattern has a

periodicity of 238 nm in the sample plane.

10 μm

15 mm

Figure 11: Di↵raction grid

3.1.2 Activation and excitation patterns

The MLAs are composed by micro-lenses spaced uniformly, with a period of 100 µ m, over an area of 400 x 400 mm ² . They both create the excitation and ac- tivation patterns, which will have the following periodicity in the sample plane:

d ON = 100µ m ¹⁵⁰ _{250 63} ¹ = 952 nm = d Readout = 100µ m ¹⁸⁰ _{300 63} ¹

40 mm

100 μm

Figure 12: Micro Lens Array (MLA)

Therefore, these patterns have the same periodicity in the sample plane, which equals to four times the deactivation pattern.

3.1.3 Detection

In this setup, the pixel size is increased as much as possible, therefore we want that each intensity minima of the deactivation pattern corresponds to one pixel in the camera. In order to do so, two lenses are placed before the CMOS camera.

Since the camera has a pixel size of 6.5 µ m, the size of a pixel in the biological sample can be computed as:

d ppx = 6.5 µ m ³⁰⁰ _{130 63} ¹ = 238 nm 3.1.4 Alignment

The position of the multi-foci patterns have to coincide with the intensity min- ima of the deactivation pattern. Therefore, an alignment procedure is followed before imaging with MoNaLISA.

First, the rotation and shape of the patterns are optimized by manipulating

the optical components in the illumination paths. The activation, deactivation,

and excitation patterns are displayed in Fig. 14-16.

2000 ppx

238 nm

Figure 13: Detection camera pixels

Figure 14: Deactivation pattern field of view 68.7x68.7 µm ²

Figure 15: Activation pattern field of view 68.7x68.7 µm ²

Then, since the precision has to be in a nanometer scale, the three beams can be separately controlled by moving mechanically the mirrors in the illumination paths until the maxima of both activation and excitation patterns fall in the minima of the deactivation pattern.

In order to visualize the results of the alignment, a sample containing flu-

orescent beads of 200 nm is scanned in order to reconstruct the Point Spread

Function (PSF) of the microscope. In other words, by scanning a bead over a

Figure 16: Excitation pattern field of view 68.7x68.7 µm ²

certain region, while only activating one laser at a time, one can observe the di↵erent patterns in the sample plane.

The multi-foci activation and readout patterns alignment with the deactiva- tion pattern is displayed in Fig. 17 and Fig. 18.

Figure 17: Overlap activation and deactivation patterns

Furthermore, the optical components in the detection are also mechanically manipulated in order to synchronize the fluorescence with the detection scheme.

3.1.5 Image data

Once the setup is properly aligned, Vimentin samples are imaged, tagged by rsEGFP.

In order to compare the image quality and resolution, two images are taken.

First, only the activation and excitation illumination schemes are applied in order to take a confocal image of the biological sample. Second, the RESOLFT image is taken by introducing the deactivation pattern as well.

Fig. 19 displays the confocal (left) and RESOLFT (right) images. It is

clear that the RESOLFT image is highly corrupted by noise, which will be

characterized in Section. 3.2.

Figure 18: Overlap excitation and deactivation patterns

Figure 19: Raw data in MoNaLISA

The reconstruction images are shown in Fig. 20. The image in the right

(RESOLFT) contains a square e↵ect that deteriorates the image in both the

cell structure and background regions.

Figure 20: Reconstructed data in MoNaLISA

3.2 Trapped-Charge Noise (TCN)

The imaging process employed in MoNaLISA consists of three sequential illu- mination phases: activation, deactivation and excitation (see Section. 2.3). In order to record the fluorescence emitted by the photo-switchable proteins, the camera is exposed only during the excitation phase, also known as readout.

During the deactivation phase, the molecules are exposed to the illumination pattern, using certain power and duration until they switch to the OFF state.

As explained in Section. 2.3, the deactivation and excitation wavelengths are the same, so the molecules produce fluorescence before switching to the OFF state.

Therefore, the fluorescence light emitted during this phase reaches the camera before it is exposed, causing a specific type of noise from trapped-charge in the sensors, the Trapped-Charge Noise (TCN). The magnitude of the noise depends on the camera pixel characteristics and the amount of fluorescence emitted from the biological sample, according to the model presented in this report.

The TCN has been reported in [6] and [7] as ”hot pixels” or ”defect pixels”, but it is typically solved using post-processing techniques. To our knowledge, it has not yet been characterized in the field of super-resolution fluorescence microscopy.

We believe that this noise is not present in other camera types, and therefore comes intrinsically from the pixel-dependent characteristics of the sCMOS cam- eras and the specific experiment where there is emission of fluorescence before camera exposure.

Due to the electronic structure of sCMOS cameras, the images are corrupted by several sources of noise, some of them being pixel-dependent (see Section.

2.4). For example, the Telegraph Noise (TRN) is caused by electron traps in the pixel transistor due to defects in the oxide; the Fixed-Pattern Noise (FPN) is another typical noise in sCMOS cameras composed by structure- and pixel-dependent components, their sources being the amplification circuits of each pixel as well as the shared electronic circuits within the rows/columns.

The pixel-dependency assumption has been an important part of this project,

and di↵erent experiments have been performed in order to prove this spatial dependency.

The TCN shows a non-linear behavior with the illumination intensity previ- ous to camera exposure, with di↵erent characteristics for di↵erent pixels; while when illumination and exposure occur simultaneously the relationship between intensity and detected counts seem to be pixel-independent.

3.2.1 Signal model

The model in [12] was slightly modified and extended in order to include the above-mentioned noise (See Eq. 1).

V i,j [n] = g i,j Q i,j [n] + U i,j + Z i,j [n] + M i,j [n] (1) Where,

• V ^i,j [n] is the detected charge by the camera pixel i, j at time n.

• g ^i,j is the pixel-dependent gain and Q i,j , the accumulated charge during the exposure of the camera, which follows a Poisson distribution.

• U ^i,j is the pixel o↵set, which is constant over time.

• Z ^i,j [n] is the read-out noise, following a Gaussian distribution.

• M ^i,j [n] is the TCN, which is dependent on the camera pixel and the illu- mination previous to the camera exposure.

3.2.2 Hypothesis

In order to study and characterize the TCN as well as proving the model in Eq.

1, a series of hypothesis are formulated:

1. The TCN follows a Poisson distribution.

2. The TCN increases non-linearly with the illumination intensity, reaching a saturation region for high-intensity levels.

3. The camera has regions more susceptible to the TCN than others.

4. The noise behaves similarly for di↵erent camera exposure times.

5. The pixels do not discharge unless the camera is exposed.

3.2.3 Characterization experiments

In order to resolve the hypothesis and successfully characterize the noise, di↵er- ent experiments were performed using a widefield illumination at 488 nm and a fluorescence bar, which is shown in Fig. 21. A total of 400 frames were taken for each configuration and averaged before analyzing the data.

The experiments are divided in two phases: previous to camera exposure,

with I p being the fluorescence emitted by the sample during t p ; and camera

exposure during t exp , I exp being the fluorescence at that time.

Figure 21: Fluorescence bar

• Exp. A, whose scheme is represented in Fig. 22, emulates the situation where the fluorescence is emitted before the exposure of the camera. First, a pulse of duration t p and power P p is applied, followed by the exposure of the camera during t exp . The fluorescent bar will emit fluorescence with intensity I p .

We assumed that the fluorescence emission is linear with the excitation intensity i.e., I p and P p are related linearly.

In this case, I exp is equal to zero, since there is no illumination while the camera is exposed. The experiment is performed for di↵erent laser powers P p = {1, 5, 10, 15, 20, 30, 50, 80, 100} mW , t ^p = 5 ms and t exp = {2, 4} ms.

Furthermore, the time between the laser pulse and the camera exposure t w

is increased to 2 ms and 4 ms, in order to investigate if the TCN discharges over time. If the pixels containing high values of TCN do not discharge until the camera is exposed, then the observed noise would be independent of t w .

This experiment focuses on: (1) determining the relationship between the fluorescence previous to camera exposure and the TCN (H. 2); (2) observing the di↵erences between the pixels to further identify if the TCN is pixel-dependent (H. 3); (3) observing if there is any di↵erence when altering the exposure time of the camera (H. 4); and (4) investigating if the pixel discharges over time in order to evaluate H. 5.

• In Exp. B, shown in Fig. 23, a laser pulse of duration t ^p + t exp is applied.

The camera is exposed during the last time t exp of the illumination pulse.

In this case, the fluorescence during exposure time I exp will not be zero.

The same P p and t exp as in Exp. A are used for this experiment.

The experiment is designed to evaluate the relationship with I p in the case where I exp is not zero (H.2).

• In Exp. C (Fig. 24), illumination and camera exposure occur simul-

taneously in order to map between intensity and the detected counts by

Figure 22: Experiment scheme A. The arrow represents the time, where the highlighted section corresponds to when the camera is exposed. In this case, there is fluorescence I p previous to camera exposure, after that I exp = 0.

Figure 23: Experiment scheme B, where there is fluorescence both previous and during camera exposure.

the camera in the absence of noise, using the same configuration as the previous experiments except t exp is fixed to 2 ms.

The goal of the experiment is to observe if, in the absence of noise, the pixels behave similarly, to make sure that the pixel-dependency is only introduced in presence of the TCN (H. 3).

Figure 24: Experiment scheme C, where there is no fluorescence previous to exposure. The laser pulse is applied only during the exposure of the camera.

3.2.4 Experiment Results

For each experiment, a stack of 400 frames of the same camera pixels (640x640 pixels) is recorded using the Hamamatsu Orca Flash 4.0 v3 camera. The average over 400 frames of Exp. A is shown in Fig. 25. In this case, there is no illumination when the camera is exposed, therefore the bright pixels are coming from previous illumination.

Fig. 26 displays the histogram of the average image of 400 consecutive frames. The peak of the histogram around 100 counts is the o↵set value intro- duced by the camera. The values that have higher counts, which are the pixels a↵ected by TCN, are part of the elongated tail in the distribution.

Since we want to know what distribution these pixels follow, the mean and

average of each pixel is plotted in Fig. 27. Each of the scatter points is a pixel

i, j with mean x(i, j) and variance y(i, j). It can be noted that, for the pixels

Figure 25: Exp. A, raw data

110 120 130 140 150 160 170 180 190 200 Counts

0 2000 4000 6000 8000 10000 12000 14000

Occurrence

Figure 26: Exp. A, image histogram

with higher counts, as the mean increases the variance does as well. Therefore, there is a linear trend between mean and variance for the pixels corrupted with higher TCN. This kind of linear behavior typical for the Poisson distribution, where the mean and variance are equal.

The values that do not follow the linear trend, which have higher variance, correspond to pixels with other noise behavior. For example, the Telegraph Noise (TRN) has a very high variance due to the electron traps, which produce a temporal shift in the detected counts, but that averages out over time. The TRN only a↵ects a very small population of the camera pixels [11].

In order to observe the relationship between illumination intensity and noise

counts, a di↵erent stack of frames is taken for each intensity value following the

100 200 300 400 500 600 Mean

0 500 1000 1500 2000

Variance

Figure 27: Exp. A, variance and mean

same illumination and exposure schemes.

Fig. 28 shows the average counts for Exp. A as a function of the laser power, where each curve corresponds to a di↵erent pixel. Each pixel follows a di↵erent saturation point, reaching a di↵erent saturation level. Since each pixel reacts di↵erently to the illumination intensity previous to camera exposure, this result suggests that there is a pixel-dependent component in the TCN.

Figure 28: Exp. A, saturation curve

Eq. 2 is used to fit the saturation curves (see Fig. 29), and Fig. 30 dis- plays the fitting parameters a, b, c, d for all pixels, ordered in decreasing order with the average counts. The parameters showed the same results for di↵erent illumination intensities at di↵erent time points.

y(P p ) = a e ^{b P}

+ c e ^{d P}

(2)

0 10 20 30 40 50 60 70 80 90 100 P

[mW]

100 150 200 250 300 350

Pixel counts

Figure 29: Exp. A, fitting curve

0 150 300

a

-1 0 1

b

10

-500 -300 -100

c

0.5 1 Pixels 1.5 2

10

-0.6

-0.3 0

d

Figure 30: Exp. A, fitting parameters

Therefore, this experiment concludes that the TCN follows a non-linear trend with I p , and it fits well with a sum of exponential functions whose fitting factors depend on the camera pixel. The pixels containing TCN follow a linear trend between mean and variance, and therefore the TCN seems to behave as a Poisson distribution. Finally, some pixels are more sensitive than others to the TCN, and the sensitivity level can be expressed by the fitting parameters.

In Fig. 31, the curves for Exp. B and C are plotted. One can notice that Exp.

B doesn’t follow the same trend as before (Exp. A), but the plot still shows a

non-linear trend towards the higher intensity values. This is due to the fact that,

since in this experiment there is both illumination before and during camera

exposure, the detected counts are a sum of a linear contribution, corresponding

to the signal; and a non-linear contribution, which is the TCN. However, data in

Exp. C follows a linear behavior (absence of TCN). Furthermore, all the pixels

have the same slope (i.e., gain), showing that, in normal conditions, the detected

intensity is not pixel-dependent. Therefore, the only pixel-dependency we have

observed is introduced by the illumination previous to the camera exposure, which is the TCN. In other words, from the saturation curves of Exp. A, B and C, one can conclude that the TCN follows a non-linear relationship with I p

Figure 31: Experiment B (left) and C (right), saturation curves The fitting parameters are displayed in Fig. 32 for the di↵erent values of t w , for Exp. A. The four values coincide for both t w , while if the pixels discharged over time we would expect to have radically di↵erent parameters.

0 100 200 300 400 500 600 700 800 900 1000

200

a 300

t

= 2ms t

= 4ms

0 100 200 300 400 500 600 700 800 900 1000

1 1.5

b

10

0 100 200 300 400 500 600 700 800 900 1000

-200 -150 -100

c

0 100 200 300 400 500 600 700 800 900 1000

Pixels -0.08

-0.07 -0.06

d

Figure 32: Experiment A, fittings

Fig. 33 shows all histograms for the di↵erent experiments. Please note that the distribution of the data in Exp. C does not contain a long tail. Also, the exposure time in Exp. A has no influence on the average histogram image;

while in Exp. B it only shifts the histogram depending on the illumination intensity and exposure time of the camera, since the data obtained has a linear component from the conventional illumination but a non-linear relationship with the intensity applied before exposure.

In conclusion, while Exp. A shows saturation curves without a signal com-

ponent, Exp. B contains both signal and noise, and Exp. C shows a linear

relationship between illumination and detected counts. This suggests that the

mean TCN has a non-linear relationship with the fluorescence previous to ex- posure. From Exp. A fitting parameters one concludes that the pixels do not discharge in time unless the camera is exposed, thus the photons remind trapped in the sensors.

Figure 33: Histograms of all experiments

3.2.5 Algorithm for noise removal

An algorithm that accounts for the FPN and the TCN has been implemented to improve the image quality of the MoNaLISA reconstructed images. Fig. 35 displays the block diagram of the proposed scheme.

In order to reconstruct the super-resolution image of a specimen from the data captured by the CMOS detector with higher image quality, three images are required. First, the image produced by RESOLFT, y, which contains the super- resolution image corrupted by noise v; Second, a di↵raction-limited version of the sample taken from the same microscope under Widefield illumination, w;

Finally, a dark-frame n, under no illumination and isolating the camera from background light as much as possible.

The dark image n is used to estimate the FPN (o↵set) by computing the average over 400 frames. Since it may vary over time it is preferable that this image is taken as close in time as possible from the recording; w is used to estimate the fluorescence previous to exposure, assuming that it has a spatial distribution similar to the widefield image and also taking into account the pixel gain from Exp. C. Then, a noise estimator is calculated by using the fitting parameters from Exp. A and the estimated P p .

From the stack of dark frames, the average over the frames is computed and subtracted from all the images n, y and w. Then, the noise estimate is computed using the fitting parameters and multiplied by the constant k that provides the minimum standard deviation in ˆ z. Finally, both the o↵set and the RESOLFT image are reconstructed, and the output image is calculated as the di↵erence between them two in order to account for errors in the o↵set subtraction process.

The main idea behind the TCN subtraction block is to subtract a pixel-

dependent value from the raw data before reconstructing the high-resolution

image. In other words, from the fitting parameters a, b, c, d and ˆ I p (estimation of I p ), an image with an estimation of the TCN is computed in this block.

The Widefield image w is used to estimate the emitted fluorescence coming from the sample in the deactivation stage of the RESOLFT imaging process.

For each pixel, the detected counts in w are converted linearly to P p using the slope and constant term which results from fitting Exp. C with a linear function. The TCN mean value per pixel ˆ v is calculated using Eq. 2 and the fitting parameters from Exp. A (i.e., a, b, c, d).

Assuming that the Widefield counts are proportional to the fluorescence in the o↵ stage, the noise estimate is scaled using a constant factor k.

Finally, the scaled estimate of the noise k ˆ v is subtracted from the RESOLFT noise-corrupted image y.

By minimizing the Mean Squared Error (MSE) of the reconstructed image we obtain a optimal value for k.

M SE = E{(ˆz z) ² ) } = E{((y kˆ v) z) ² } =

E{y ² } + k ² E{ˆv ² } 2 k E{y ˆv} + E{z ² } 2 E{y z} + 2 k E{ˆv z}

dM SE

dk = 0 = 2 k E{ˆv ² } 2 E{y ˆv} + 2 E{ˆv z}

k = E{y ˆv} E{ˆv z}

E{ˆv

}

However, since we don’t have the value of z, we found k such that the standard deviation of the corrected image is minimized (see Fig. 34).

0 0.2 0.4 0.6 0.8 1

5.5 5.51 5.52 5.53 5.54 5.55 5.56

Figure 34: Standard deviation as a function of k

Figure 35: Diagram for noise removal algorithm

3.2.6 Results in raw data

In order to evaluate the correctness of the algorithm the images before y and after correction ˆ z are shown in this section for di↵erent samples. Two corrections are performed: subtracting the average of 400 dark frames in order to correct the o↵set producing the FPN as well as the matrix for the TCN using the Widefield illumination to estimate the fluorescence.

The raw images before and after correction are shown in Fig. 36 and Fig.

37. The quality of the image is improved substantially after the application of the algorithm.

The subtraction of an estimated value from the Widefield image produce some black pixels as seen in Fig. 38, which may be due to the fact that the Widefield laser has a clearer Gaussian shape than the deactivation laser, as well as possible incorrect estimations or errors in the fitting process. Therefore the values of ˆ P p can produce some minor errors, especially in the periphery of the image. Since the structure is usually in the center of the ROI, the errors are not as relevant as the TCN.

3.2.7 Results in reconstructed data

In order to evaluate the performance of the TCN reduction algorithm, Fig.

39 displays the image after correction without subtracting the reconstructed o↵set image (i.e., ˆ Z) and the image where only the o↵set is subtracted (i.e., RECON (y 2 ) or Y 2 ). Since both images are corrected for the o↵set, the square- e↵ect is similar in the background areas for both images. However, the square e↵ect in the cell structure (i.e., where there TCN is higher since there is more fluorescence) is much less when both corrections are performed before recon- structing the super-resolution image.

Fig. 40, Fig. 41, and Fig. 42 display the e↵ect of the reconstructed image

after applying all the corrections (i.e., Y and ˆ X). The square-e↵ect is signifi-

cantly reduced both in the background and in the cell structure. Furthermore,

the fitting errors shown in Fig. 39 are also corrected by employing the error

correction over the reconstructed data, which means that the o↵set image is

Figure 36: Noise correction in raw data

Figure 37: Noise correction in raw data

also reconstructed using the same software.

Figure 38: Zoomed noise correction in raw data

Figure 39: Reconstructed o↵set corrected image Y 2 (left) and after TCN cor-

rection ˆ Z (right) noise correction.

Figure 40: Noise correction in reconstructed image

Figure 41: Noise correction in reconstructed image

Figure 42: Noise correction in reconstructed image

4 Discussion

During the experimental activity with the MoNaLISA setup, an unknown source of noise was noticed. The image quality of the RESOLFT image was severely corrupted: in the raw data, some pixels appeared much brighter than others;

and the reconstructed data was corrupted with square e↵ects. However, the confocal image was free of this noise in the raw data. Based on this information we identified a source of noise originated from the deactivation illumination, in which some photons get trapped in the camera pixels and are not released until the exposure time of the camera. We named it the trapped-charge noise (TCN).

In order to investigate the source and model the noise, di↵erent hypothe- sis were formulated regarding the behavior with the previous illumination, the distribution and discharge characteristics. Three experiments were designed in order to prove or verify the hypothesis.

First, the assumption of Poisson distribution for the TCN was drawn from the linear relationship between mean and variance, as well as the characteristics of light.

On one hand, Exp. A aims at emulating the situation where there is il- lumination only previous to camera exposure. Since the camera is exposed in darkness, the collected image corresponds to only TCN. On the other hand, in Exp. B illumination and exposure occur simultaneously. The resulting image is thus free of noise.

For each pixel, the average number of counts over 400 frames is calculated for di↵erent values of the illumination, and a curve is drawn for every pixel. The curve in Exp. A follows a non-linear behavior with the illumination intensity.

In Exp. B, the relationship between illumination and detected counts is linear, and all the pixels have the same slope.

The fact that there is only non-linear behavior when there is illumination before the exposure of the camera, and since that is the exact the same source of the TCN, proves that the noise is the component that follows this non-linear relationship.

The noise model was created from fitting the saturation curves with a sum of two exponential functions and storing the parameters in a matrix, that is later used in the noise reduction algorithm.

Di↵erent experiments were performed using di↵erent exposure times and illumination intensities, and yet the pixels which had highest value of TCN were still the same in the camera chip, thus suggesting that the noise depends in the pixel location as well.

Furthermore, the fitting parameters and image distributions (histograms) coincided for di↵erent exposure times and waiting times, defined as the time between illumination and exposure. Since the pixel intensities did not drop when increasing the waiting time, we concluded that the trapped photons are only released when the camera is exposed.

An advanced algorithm that takes into account the TCN as well as the fixed- pattern noise (FPN) was implemented. Its application showed improvements in the raw data (i.e., the TCN was reduced substantially) and reconstructed data as well by minimizing the square e↵ect.

From the results, it is observed that the FPN correction reduces the squares

in the background areas while the TCN correction improves the visibility of

cell structures. This is because the FPN degrades the image under low signal

intensity, while the TCN appears in the regions where higher fluorescence is detected.

Therefore, the TCN is properly modeled and characterized, and the algo- rithm for noise reduction shows improvements in image quality in both the raw and reconstructed data.

5 Conclusions and Outlooks

After the elaboration of this project, I have learned the theory behind super- resolution imaging and worked on the experimental setting-up of the MoNaLISA microscopes, including the alignment procedure and imaging of cell structures.

An overview of the CMOS camera noise was provided, and the fixed-pattern noise (FPN) corrected by subtracting an average of 400 dark frames.

Furthermore, the trapped charge Noise (TCN) was detected in the raw data, and the model and characterization of the TCN was performed.

Once the model was well characterized, an algorithm for reducing it was implemented, based on mean estimation and background subtraction. After applying the algorithm in the data, both the raw- and reconstructed data show improvements in the image quality. The algorithm has been implemented in the MATLAB version of the reconstruction software.

We demonstrate that the TCN originates from the fluorescence previous to camera exposure, which gets trapped in some pixels due to the characteristics of the CMOS cameras. Since the experiments support the claim, and algo- rithm for model reduction is implemented providing good results in the raw and reconstruction data.

However, the square e↵ect in the reconstruction data is not fully removed.

Therefore, better estimation techniques and adaptive noise removal algorithms could be applied in order to further increase the reconstruction of the data.

Di↵erent techniques for reducing the noise corruption in the images could also be implemented directly in the camera chip.

Regarding the experimental activity, future featuring semi-automatized align-

ment as well as adjustable pixel size in the detection have the potential to further

increase the throughput and image quality.