Automated visual evaluation of an electrode with neural networks

(1)

INOM

EXAMENSARBETE ELEKTROTEKNIK, AVANCERAD NIVÅ, 30 HP

STOCKHOLM SVERIGE 2019,

Automated visual evaluation of an electrode with neural networks

LOVISA COLÉRUS

KTH

SKOLAN FÖR ELEKTROTEKNIK OCH DATAVETENSKAP

(2)

R ^OYAL I NSTITUTE OF T ^ECHNOLOGY

M

ASTER

T

HESIS

Automated visual evaluation of an electrode with neural networks

Author:

Lovisa COLÉRUS

Supervisor:

Magnus JANSSON

November 5, 2019

(3)

(4)

iii

ROYAL INSTITUTE OF TECHNOLOGY

Abstract

School of Electrical Engineering and Computer Science

Master of Electrical Engineering

Automated visual evaluation of an electrode with neural networks by Lovisa COLÉRUS

This project was done in cooperation with the company Scibase. Scibase produces a product called Nevisense, which is used to detect skin cancer in a non- invasive way. The measurements are made with electrical impulses and the electrode that is in contact with the patients’ skin is only used once. When these electrodes are produced, they must pass visual inspections for each step in their assembly. These visual inspections are done by operators using a traditional microscope.

This inspection is both time-consuming and uncomfortable for the operators, e.g., microscopes strain their eyes and the design of the microscopes are not ergonomic.

This project is about the automation of these visual inspections to increase the production of electrodes and to improve working conditions.

To automate this, two parts were needed: images of the electrodes and a way to classify them as pass or fail. The images were taken with a digital microscope and to be able to get several images at once, a programmable XY-table was used. The images were processed with OpenCV, a computer vision library. The classification of the images was done using a neural network and the accuracy that was achieved was 99.2%, which is a higher accuracy than the conformity that the operators have.

(5)

iv

Svensk sammanfattning

Den här examensarbetet är gjort i samarbete med företaget Scibase. Scibase tillverkar en produkt vid namn Nevisense, som används för att utvärdera födelsemärken och andra hudförändringar för att se om de är eller kommer utveckla sig till malignt melanom, den farligaste sortens hudcancer. Mätningarna görs med elektriska im- pulser. Frisk hud har en annan impedans än skadad hud och genom att mäta impedansen kan en diagnos ställas. Elektroden som används vid mätningarna används endast till en patient och behöver sedan kasseras för de är så känsliga att de går sönder under mätningarna. Under produktionen av elektroderna sker visuella kontroller av operatörerna tre gånger per elektrod med hjälp av ett traditionellt mikroskop. Denna process är både tidskrävande och obekväm för operatörerna. Det här projektet har haft som mål att automatisera den första visuella kontrollen under produktion för att minska produktionstiden och förbättra operatörernas arbetsförhållanden.

För att automatisera det här behövdes två delar: bilder av elektroderna och klas- sificering av bilderna. Bilderna togs av ett digitalt mikroskop och behandlades med hjälp av OpenCV, som är ett bibliotek för bildbehandling. För att kunna automatisera ytterligare och få med flera elektroder på samma gång användes ett XY-bord som flyttade elektroderna under mikroskopet. Klassificeringen av bilderna gjordes med hjälp av ett neuralt närverk och en nogrannhet av 99.2% uppmättes, vilket är högre än vad operatörerna överensstämmer med varandra.

(6)

v

Acknowledgements

I want to thank my supervisor at KTH, Magnus Jansson and my supervisor at Scibase, David Melin for supporting me during this thesis.

(7)

(8)

vii

List of Figures

1.1 Example of malign melanoma . . . 2

1.2 Nevisense . . . 2

1.3 Patient being examined with the Nevisense . . . 3

1.4 Here we can see how healthy and damaged tissue differs and how the electric impulses will differ in their way . . . 4

1.5 How the electrode sends its electric impulses . . . 4

2.1 Example of a one layer neural network. . . 7

2.2 Concept of max pooling . . . 10

2.3 Electrode mounted on Nevisense pen . . . 11

2.4 Electrode seen from above, not mounted on body . . . 11

2.5 The new type of electrode that is produced in a slightly different way has a much darker surface with more reflections. . . 12

2.6 Acceptable spike . . . 12

2.7 Damaged spike . . . 13

2.8 Three steps of preparation of the image . . . 15

2.9 Two more steps of preparation of the image . . . 16

2.10 Illustration of applied Gaussian blur and thresholding. . . 17

2.11 Illustration of eroding and using ConvexHull . . . 17

3.1 ”Darkroom” for testing the lighting and the set-up of the microscope.. 20

3.2 The black rectangle around the upper bar is the result of the image processing that finds the bars. The dots are corners of other found bars in the image, but the algorithm uses the tightest rectangle it can find. . . 22

3.3 Finding the countours of the spikes to crop the bar accordingly. The dots are upper left corner and lower right corner of how the image will be cropped . . . 22

3.4 Resized images in the data set. Some have changed colours from RGB to BRG to make the neural network less sensitive to light and make it focus on the shape of the pyramids. . . 23

3.5 The best performing network . . . 24

3.6 A similar network that did not give as good accuracy . . . 24

3.7 Aluminum plate for the electrodes . . . 25

3.8 Plastic plate that is used for the manual control. The holes are deeper to prevent electrodes from falling out and they are wider so that the tweezers are able to reach down and lift them up . . . 26

3.9 Plastic plate manufactured by Prototypcenter at KTH . . . 27

3.10 The aluminum plate fits perfectly in the carved out space . . . 28

4.1 Aluminum plate with the mounted electrodes after assembly. If this could be sent into an electric and visual test, even more time could be saved. . . 30

(11)

(12)

1

Chapter 1

Background

1.1 Introduction

Nearly half of all cancer cases in the world are skin cancer. There are several types of skin cancer and the types can be divided into non-melanoma (treatable and non- fatal) and malign melanoma (can be fatal). The most common cause of skin cancer is ultra violet light, which most people get exposed to by being in the sun. Since the rise of tanning salons and tanned skin being a popular fashion trend, the direct exposure of sunlight on skin has increased over the years. This has the unfortunate side effect of skin cancer becoming more common and the cases of skin cancer increases. Sunburn, when the skin becomes red instead of brown and has a burned texture, is caused by too much exposure on skin with low pigment. Sunburn is a high risk factor for developing melanoma.

Melanoma can also develop from moles without any exposure to sunlight, either those moles that came with birth or that developed later in life . About 25% of skin cancer cases come from moles.

Malign melanoma is only a small percentage of skin cancer cases, but it is the most dangerous and it can be deadly, since the cancer can spread to lymph nodes and/or other places in the body. The cause of the melanoma is melanocytes, which are changes in the skin cells, damage to the DNA that causes the cells to function in a different way. If the malign melanoma is discovered and treated early, the risk of spread is low and the survival rate is high.

Diagnosing the melanoma is done by a visual examination where the examiner check for irregularities in shape, color, texture, size of the mole and studies the patients family history to see if they are in a risk group. If the examiner decides it might be a malign melanoma, the mole is cut and sent to a lab for tests. This is an invasive procedure and there is a large uncertainty when only diagnosed by a visual inspection. Examples of malign melanoma can be seen in figure1.1. These moles does not look that much different from normal moles and could be glossed over in a visual examination. This leads us into the product that is relevant for this thesis, the Nevisense, a product developed and manufactured by Scibase. The Nevisense can be seen in1.2and how it is used on patients can be seen in figure1.3.

Scibase was founded in 1998 and its product Nevisense is the result of 20 years of academic research at Karolinska Institutet. The aim of the product is to ease the diagnosis of malign melanoma in a non-invasive way and uses Electrical Impedance Spectroscopy (EIS) to detect anomalies in the structure of the skin. Healthy skin has electric properties that differs from damaged skin, in figure1.4it is possible to see the tissue in a cross section to illustrate the differences.

“EIS is a measure of the overall resistance within the tissue at alternating cur- rents of various frequencies. It is measured by applying an unnoticeable alternating potential between the bars on the tip of the probe. In order to cover the lesion in

(13)

2 Chapter 1. Background

FIGURE1.1: Example of malign melanoma

FIGURE1.2: Nevisense

both width and depth, the measurement is performed in 10 permutations covering both shallow measurements between neighboring electrode bars as well as deeper measurements between more distant electrode bars.”

Figure1.5 illustrates how the electrode is placed and how it interacts with the skin.

The electrode is placed at the tip of the pen and have direct contact with the patients skin. These electrodes are vital to the electric measurement and it is important that they aren’t damaged, since they have to give as exact measurements as possible to be able to give an accurate assessment of the damage of the skin. Since they are so small and need exact electric measurements they become fragile and it is important that there aren’t too many imperfections at the surface. The electrodes are visually examined to discard the imperfect ones, with clear instructions for what is an acceptable electrode and what the limit of that acceptance is.

(14)

1.2. Thesis objective 3

FIGURE1.3: Patient being examined with the Nevisense

1.2 Thesis objective

Each new patient has to use a new electrode, since the surface of the electrodes are covered by fragile spikes and also to avoid potential contamination between patients. This means that the electrodes has to be produced at a high rate to meet cus- tomer demand. A bottle neck in production is the visual inspection of the electrodes, which is done three times for each chip - before montage, after gluing it to the body and after the electric test just before it is packaged and sent to the customers. The inspection after the gluing is done by the bare eye but the other two are done manu- ally in microscopes. The inspection with the microscopes takes too much time and is the cause of back pain and migraines for the operators. This is because it is both an unergonomical working position and is a strain on the eyes because of the flickering lights and the concentration it requires. The objective of this thesis is therefore to automate one visual inspection of the electrodes to improve working conditions for the workers and speed up the manufacturing process of the electrodes. The visual inspection of the chips before gluing them onto the body is the one that is the main focus of this thesis.

The objective of this thesis is to shorten the production time for an electrode and improve the working conditions in the production centre.

1.3 Delimitations

The microscope that is provided is a limitation of the image quality. The goal will be to improve the image quality as much as possible with the right lightning but the microscope is ultimately a digital microscope and there will be some pixels instead of smooth lines. One constriction will be to focus on a few optimizers and activation functions instead of testing all possible combinations of a network. There is limited

(15)

FIGURE1.4: Here we can see how healthy and damaged tissue differs and how the electric impulses will differ in their way

FIGURE1.5: How the electrode sends its electric impulses

time available for this project, which means that the focus will be on implementation and not extreme fine-tuning.

(16)

1.4. Choice of methodology 5

1.3.1 Limitations with the data set

Since there was is no given data set, it will have to be created before a network can be tested. There is a limitation in how accurate the results will be, if the images are sorted to a 95% accuracy it does not matter if the network can sort it to a 99% accuracy since it won’t be true. There are too many images to re-sort several times, the length of this project is not enough so the data set will probably include some errors.

It will probably not include more errors than what would have been sorted wrongly in a manual inspection of an electrode. Another limitation of the data set is the size, it would be possible to have a huge data set with over one million images in each category, there is no shortage of material, but the time-limitations will hinder this.

The aim is that the data set should include around fifty thousand images. The manufacturing process was slightly changed during the time of the project and changed the appearance of the electrodes in a way that might be significant. The amount of images of the new type of electrode is limited from how many electrodes Scibase are able to produce and discard in the time up until the end of this project.

1.4 Choice of methodology

The electrodes will be analyzed by using a digital microscope to take pictures of an electrode and process the image to get each pin in its own image. These images of the pins will be fed into a neural network that will determine if the pin is acceptable or not. To minimize human work there will be an aluminium plate that holds 25 electrodes, which after the inspection will be put in to a robot for assembly of the electrode to body. This plate will be put on an movable table that will automatically move the plate so that each electrode will be photographed and controlled one by one. In the end the result of which electrodes are approved to go to the next phase in the production chain will be presented visually.

1.4.1 Tools

Python (v3.7) is used as the programming language in this project, since it is an easy language to implement other libraries in, e.g. it has scikit-learn and easy ways to use Matlab tools without being restricted to Matlab. As a development environment JetBrains Pycharm was used since it comes with several helpful tools and scientific aids, and Anaconda comes with it, which mean that it is easy to start using Keras to train small neural networks. Keras was used in the beginning to design smaller networks and re-familiarize myself with designing a neural network and preparing the data. Later, when the data set was bigger, Peltarion was introduced to the project, which is an online platform for designing and training neural networks. Peltarion lets the network run on a GPU, which made the training time significantly shorter.

This came with some restrictions, since an easy-to-use platform can’t incorporate every detail, that would defeat the point. It was decided that there was more than enough design choices and the extra freedom would not be needed to achieve the goal.

To get from raw images to the data set OpenCV was used which is an open source library for image processing and image recognition. It was chosen since there are many tutorials and answered questions available on the internet, which makes it easy to use and at the same time it has a depth that is much more than what will be needed, without using too much computation power.

(17)

The microscope is from Inspectis and is a digital microscope with 4K HD resolu- tion. This is much easier to use than an analog microscope, since the image is shown at the computer screen instead of the user having to bend over and look through a small lense in an unergonomic position. This is why the microscope was bought, to avoid an unergonomic working position. The Inspectis can photograph and save directly into the computer, which is why it is possible to use it for this thesis’s purpose.

An XY-table from Standa is used to move the electrodes around and it has a high precision with a coding library that is easy to implement with Python. It was just chosen because of convenience, there was an X-table with accessories that was already owned by Scibase, which made it easy to get a XY-table that matched.

(18)

7

Chapter 2

Theory

2.1 Neural networks

Neural networks are a type of machine learning network that is often used in classification problems. They are easy to implement and often give good results with low effort from the designer. Neural networks try to simulate the human brain with its synapses and nerves. Figure2.1 shows a model of a simple fully connected neural network. It has an input layer, one hidden layer and one output layer.

FIGURE2.1: Example of a one layer neural network

(19)

8 Chapter 2. Theory

2.1.1 Supervised learning

In this case, supervised learning is used. Supervised learning is a collective name that means the algorithm is provided with a data set that is used for training the network. The data set consists of tuples where one part is a label indicating the "right"

answer, and the other is a vector with information. In this thesis the information is an image in color. The network has to work out a "rule" to determine what label the information corresponds to. Learned-Miller,February 2014; visited June 2019 2.1.2 Loss functions and decisions

When there are only two classes to classify, ”ok” and ”not ok”, the output from the neural network will be a value between 0 and 1, where 0 means one of them and 1 is the other. If unweighted, the classification will decide that everything over 0.5 is 1 and everything under is 0. The distance from the output to the true value will be used in a loss function. Depending on which loss function is used, the loss will have different values, but the aim of the network is always to minimize the loss. The loss will be minimized by using gradient descent to get closer to 0.

2.1.3 Gradient descent

To minimize the loss the weights of the neurons in the network have to be updated in the next run through. This is what the training of the network is, updating weights.

The gradient will tell us how to change the weights to lower the loss, since the gradient is a representation of how the function changes from the point we look at it in.

it is only a good representation in a very small interval, which is why it has to take a small step and then recalculate the gradient. The size of the step is called the learning rate, and it is a parameter that is tuneable while designing the network. Using a sufficiently small stepsize and updating the gradient for each step is very memory and computing intense, which means that there has to be a solution that approximates this but is reasonable in computing power. There are several algorithms that optimizes the loss, called optimizers. Some of them that are relevant to this project will be briefly described here.

Back propagation is the principle of feeding the end data from the network through the network again and changing the weights in the direction of the loss functions gradient to achieve a local minimum of loss. For a deeper understanding and further reading on the math behind back propagation, see Nielsen,visited June 2019.

Stochastic Gradient Descent (SGD)

SGD is a stochastic approximation of gradient descent. Instead of calculating the true gradient, it approximates the gradient by calculating the gradient of a stochasti- cally chosen sample or samples and performs an update according to that. it is then repeated until a local minimum is achieved. Combining SGD with backpropagation usually yields a good result in training a neural network.

Root Mean Square propagation (RMSprop)

RMSprop tunes the learning rate automatically by checking in each update. If the last two gradients had the same sign it increases the stepsize since that means we are moving in the correct direction. If the last two gradients had different signs, it decreases the stepsize since the step it took was too large and missed the local

(20)

2.1. Neural networks 9

minimum. To avoid that the stepsize gets out of hand, the moving average is kept for the squared gradients of each weight.Then the gradient is divided with the square root of the mean square of the moving average. The update rule is then decided from this instead of just updating with a random multiplier, this is to avoid problems that can come with large data sets and many gradients.

Adadelta, Adagrad

Adagrad adapts the learning rate according to the parameters, it does large updates for infrequent parameters and small updates for frequent.

Adadelta is a robust version of Adagrad which uses a fixed window of past gradients to use in its updates instead of all past gradients. This means it can still learn and update after many iterations, instead of converging to a stop like Adagrad. More reading can be found here: Zeiler,2012,Adagrad.

Adam etc

Adam, Adagrad, Nadagrad, Adadelta, there are many types of optimizers that can be tried in a network. The description of these optimizers are out of the scope of this thesis, but further reading can be found at e.g. Bushaev,October 2018, visited June 2019.

2.1.4 Feature learning

When the input is an image, there is a need for the network to learn features before a fully connected network can classify the information. This is done by convolution and pooling.

Convolution layer

An image is a huge amount of information, but not every pixel is important and many pixels represent the same thing, e.g. an edge is usually thicker than one pixel, but all the pixels represent "an edge". To simplify the image and extract features from the image, a convolutional layer is used. A convolutional layer works like convolution in math, it takes a matrix (called kernel, the size is determined by the designer) and sweeps it over the image-matrix to create a new matrix with less amount of information but higher quality. For an RGB-image this is done in three dimensions, one for each color matrix.

Pooling layer

The pooling layer takes a matrix (once more, a kernel that is designed by the user) and sweeps it over the image, but instead of convolution it just takes a value, either the average value in that matrix if average pooling is used, or the maximum value in the matrix if max pooling is used. Max pooling has the best performance since it also works as a noise suppressant and performs de-noising at the same time as dimensionality reduction. Figure2.2shows the concept of Max pooling.

(21)

10 Chapter 2. Theory

FIGURE2.2: Concept of max pooling

2.1.5 Activation layers

To each layer in a neural network an activation function is assigned, usually the sigmoid function,

f(z) = ¹

1+e⁻^z (2.1)

the hyperbolic tangent,

f(z) =tanh(z) (2.2)

softmax,

f(z_j) = ^e

zj

∑^Kk=₁e^z^k j=1, 2, ..., K (2.3) or the rectified linear function (relu),

f(z) =max(0, z) (2.4)

These functions fit all the output values of the layer into the constraints

[0, 1] (2.5)

[−_{1, 1}] _(2.6)

[_0,_∞] _(2.7)

Depending on what the next layer is, different activation functions should be used, e.g. if the next layer is the output layer, Sigmoid or softmax is used to be able to determine what the class should be.

2.1.6 Overtraining and dropout

Overtraining a network is a valid risk. If the network is overtrained, it focuses on such small details of the specific training data that it might miss the big picture and it will have a hard time generalizing. To hinder this, Dropout-layers are used. A Dropout-layer takes a value between 0 and 100% and removes that percent of the outputs from a layer, it ”drops” them. This will hinder the network from becoming too specialized.

(22)

2.2. The electrode 11

2.2 The electrode

FIGURE2.3: Electrode mounted on Nevisense pen

FIGURE2.4: Electrode seen from above, not mounted on body

The electrode is shaped like a square with five rectangles and a gutter between each rectangle. Each rectangle, or bar, has three rows of 15 spikes and in the middle

(23)

FIGURE2.5: The new type of electrode that is produced in a slightly different way has a much darker surface with more reflections.

row there is a drilled hole. The electrode is set on a plastic bottom that is sprayed with gold and the gutters are sawed through the gold to prevent conduction between the bars. In figure2.4 the electrode is seen from above, the spikes look like regular triangles from this direction, but the spikes have a pyramidal shape.

FIGURE2.6: Acceptable spike

The saw has been changed during the time that this project has been active and it uses higher water pressure when rinsing the electrodes, which give a different type of appearance, darker background with more reflections, but the new electrodes are

(24)

2.3. Open CV and image algorithms 13

cleaner, with less frizz from the saw on the sides and on the surface of the electrode.

The difference can be seen by looking at figure2.4, which is the older version, and then at figure2.5, which is an electrode which is produced with the new saw.

FIGURE2.7: Damaged spike

For a spike to be acceptable it has to have a sharp peak without any damage on the rest of the triangle. In2.6there is an example of an acceptable spike and in2.7 there is an example of a damaged spike. it is also possible that one or more spikes are missing, that will count as a damaged spike. An electrode is approved if there is no more than one damaged spike on each bar. If two damaged spikes are located on the same bar the electrode has to be discarded.

2.3 Open CV and image algorithms

Open CV is an image vision and machine learning software library. In this thesis it will be used for image preparation, to find the contours of the image and crop it to get images of only the spikes from a picture of the whole electrode. The data that should be fed into the network is a single spike in an image, but taking 225 images for each electrode is inefficient and therefore there is a need to automatically split the image. In figure2.8 the first three steps can be seen and in2.9 the last two are shown. In figure2.10and2.11, Gaussian blur, thresholding, erosion and ConvexHull is shown on just one balk.

2.3.1 Gaussian blur

Image blurring is done by 2D-convolution. The image is convolved with a kernel to smooth the image and remove noise. The Gaussian blur has a Gaussian kernel, which is a matrix with Gaussian coefficients. For further reading, see library,visited June 2019(c); library,visited June 2019(f)

2.3.2 Threshold

Thresholding is simply changing gray pixels to white or black, depending on which limit is set. Everything below that limit will be turned black and everything above it will become white. By using a threshold limit that sets the ”best” limit for the image

(25)

(for further reading about this, see library,visited June 2019(e)), or deciding manu- ally which is best, it is possible to get the relevant portions of the image in black and white, and all other noise removed. Other alternatives are Adaptive thresholding, which uses the same algorithm but on a smaller area and moves that area over the image. This is good for images that changes in brightness, and it makes it possible to find the shapes anyway. For further reading, see library,visited June 2019(g) 2.3.3 Erode

Erode is a morphological transformation. A morphological transformation is done on an (usually binary) image and it transforms the image, the type of transformation is decided by an operation kernel. Erosion uses a kernel filled with ones. The kernel is convolved with the image. A pixel is set to one if all the pixels under the kernel is one, otherwise it will be set to zero. This is done to remove white noise in the image and close black areas. It is also used to thin out contours and make them easier to read. For further reading, see library,visited June 2019(d).

2.3.4 Canny

The library,visited June 2019(a)is an algorithm for edge detection. It uses five steps from an normal image to an image with only edges in white and the rest in black.

The first step is to use Gaussian blur to remove noise and smooth the image.

The second is to find the intensity gradients in the image by using the Wikipedia, visited June 2019(a)which returns one gradient in x and one in y. From this the edge gradient and direction can be found.

The third step is non-maximum suppression and it is supposed to find the largest, most defined edge.

At the fourth step, double threshold is applied with one high limit and one low.

This is to prevent color variation and noise to cause edges.

The fifth and final step uses Wikipedia, visited June 2019(b), which suppresses all edges that aren’t strong or connected to a strong edge. This leaves strong, true edges in the image in white and the rest is black.

2.3.5 Finding contours

The library, visited June 2019(b) looks for closed curves in white on a black background. This is done by following the algorithm made by Suzuki and be,1985, but the theory behind the algorithm is outside of the scope of this thesis.

2.3.6 Convex hull

Mallick,visited August 2019is a function that takes a contour and finds the convex contour around it. It finds the extreme points of the contour and draws lines between them to create an encompassing shape. This is very useful with the spikes since the base of the spike is less apparent than the sides, and therefore the bottom edge might be drawn in a different way. With Mallick,visited August 2019, every spike will have almost the same outline and area. This makes the contours easy to use and Mallick, visited August 2019is a good complement to the library, visited June 2019(a) and library,visited June 2019(b).

(26)

FIGURE2.8: Three steps of preparation of the image

(27)

FIGURE2.9: Two more steps of preparation of the image

(28)

FIGURE2.10: Illustration of applied Gaussian blur and thresholding

FIGURE2.11: Illustration of eroding and using ConvexHull

(29)

(30)

19

Chapter 3

Implementation

3.1 Creating the data set

To create the data set, a python algorithm was used. The algorithm cropped each electrode into bars, and then cut the image into pictures with only one spike in each, and then it discards the holes with the copper stick. The pictures are taken with the Inspectis microscope that will be used for the practical implementation. The electrodes have the same background in the images as they will have in production.

By automating the image process it was possible to produce many images in a short amount of time. The time-consuming part was to sort the data set.

Since the first iterations of the data set had blurry images it could be hard to decide which spikes were acceptable and which were defect. If a person couldn’t decide how to classify the data set, how could the computer be taught to do it? By going to Uppsala to discuss the image quality with production it was decided that we needed higher-quality images. Because of this, the data set had to be recreated.

Many images could fortunately still be used. The ones where the defects were obvi- ous and the ones where it was a clear, non-grainy spike were saved, but to redo the data set consumed more time than initially was planned for the creation of the data set.

To get sharper images, the lightning had to be changed since it was the only part that could be changed. The microscope was already chosen and could not be switched out and it has the highest quality with reasonable time to photograph dig- itally that is available in that price range.

3.1.1 Lighting

Fantastic lighting came one Tuesday 8:15 am when the sun shone in through the window and made the spikes have perfect reflection that showed the outline perfectly.

Unfortunately, that is not possible to use, but it was a good example to understand what perfect conditions looked like. A frame was built around the the microscope and covered it in black plastic to ensure reproducibility that did not depend on what time, day or weather it was.

First the microscopes’ built-in lamp was used, but it was not enough, and a LED- cable with white LED’s was procured to be able to move the light freely, and get a type of ambient light to avoid shadows. Unfortunately this made the images very pixelated and not sharp enough. To get sharper images sharper lights were used, e.g. the flashlight from a cellphone. Sometimes this gave good lighting, but other times the images was just as grainy as with the LED. A cellphone is not a long-time solution and the flashlight is not good enough. The company that produces the Inspectis also sell several lights that belongs to the Inspectis. They came to the office to try out the best lightning for the project. They left several different lights for to be

(31)

20 Chapter 3. Implementation

FIGURE3.1: ”Darkroom” for testing the lighting and the set-up of the microscope.

tried, and a very good and sharp way to light it was found to be with an one-armed lamp diagonally behind the electrode. As can be seen in figure3.1, the test of the lighting and the set-up was an untidy thing with a lot of cables.

The new type of electrode, as seen in figure2.5, has much more reflections and a darker surface, which meant that the lighting had to be less intense to avoid disturb- ing reflections. This made it easier to process the images since there was a bigger difference in color of the spikes and the background.

(32)

3.2. Data set 21

3.1.2 Image cropping algorithm

Open CV was used for all image handling with Python. To cut out each triangle, first each bar was found in the electrode and they were rotated to be straight if the image was not parallel to the screen. To find that this Gaussian blur was used, then the image was thresholded to create a black and white image. Canny was used to find the edges and then a Gaussian blur again to close the edges. From this the contours were found and the area of each bar was known, which means that the right contours were chosen as the ones with similar area as the bar should have.

Usually the algorithm cannot find all five bars in a satisfactory way, e.g. if a piece of dust is bridging two bars together, or one is very broken it has a hard time getting the right contour. Because of this, the bar with the right width and lowest height of the ones that the algorithm found is used to find all of the bars. Since some of the electrodes that were used varied in brightness it had to work around that by running the algorithm again with different settings. This means that the algorithm will loop through four times at the most, changing the settings until one bar is found. If it cannot find any bar after this, it will be counted as a faulty electrode. In figure3.2 the found rectangle is shown. This is loop two of the algorithm. The dots are corners for the other rectangles that was found, but the drawn rectangle is the one that will be used.

The spikes can have different positions on the bar, since sawing is a manual process. Therefore, it was decided to find the spikes too and crop the bar by finding the rows of spikes. The quick route of finding the spikes directly wasn’t implemented since some spikes could be hard to detect, they can have the same color as the background and the contours are then vague. In figure3.3 it can be seen that some of the spikes aren’t found, but there is a very high chance to find at least one spike in each row. To find the contours of the spikes a Gaussian blur was used, followed by thresholding and then eroding. To close the contours convexHull was used. The spikes are in an interval of areas too, so it used only the contours in that interval of area and then the algorithm finds the extreme points of each row to crop there.

When the three rows with spikes are tightly cropped, it is only to split the image into parts, because the spikes are always at the same distance from each other in horizontal and vertical direction.

3.2 Data set

The data set consisted of included 43233 images in total, 29973 acceptable images and 13260 unacceptable images. Peltarion flipped all these images horizontally, shifted them horizontally and vertically and zoomed to create several versions of each image and make the network robust to changes in images. To solve the problem of the human error, wrongly sorted images, only images that was easily identified as OK or not OK was included. The persons working in production double-checked the image sets to ensure that they were rightly sorted.

To use the images in the data set, they have to be the same size. They were resized to the average height and width of the data set,(91, 123). In figure3.4, some of the data set can be seen.

(33)

FIGURE3.2: The black rectangle around the upper bar is the result of the image processing that finds the bars. The dots are corners of other found bars in the image, but the algorithm uses the tightest rectangle

it can find.

FIGURE3.3: Finding the countours of the spikes to crop the bar accordingly. The dots are upper left corner and lower right corner of

how the image will be cropped

3.3 Neural network

The network takes images of size (91, 123, 3) as input. The layers that were used were 2D Convolutional layers with 64 or 32 filters, Max pooling with (2,2) as width and height, Batch Normalization, Flatten and Dense with 128 or 1 nodes. See3.5 for the architecture of the network. Several network structures was tried and ran in parallel with the same data to see which performed the best, with different optimizers (and learning rates for the optimizers), batch sizes, etc. When a structure that worked well was found, changes were made to it until it performed even better.

When 99% accuracy was reached for a data set that was around 17k images, 11k approved images and 6k unapproved, the experiments were stopped, since improving beyond that point is unnecessary because of human error in sorting the data set.99%

(34)

3.4. Implementation, set-up, XY-table 23

FIGURE 3.4: Resized images in the data set. Some have changed colours from RGB to BRG to make the neural network less sensitive

to light and make it focus on the shape of the pyramids.

was also seen as an acceptable accurancy level. The optimizer that performed the best was Adadelta with ρ=0.93

Peltarion was the platform that was used to train and design the networks, it helped a lot that several experiments could be run in parallel.

A similar network that was tried can be seen in3.6, the difference is that the optimizer is RMSprop and there is only two layers of 2D Convolution and Maxpooling, but one more Batch normalization and no extra Dense layer after the Flatten. This reached 94.3 in accuracy. Other networks that were designed the same way as3.5, could fail miserably if the wrong optimizer was used, e.g. if RMSprop was used it just classified everything as unacceptable. This is in difference to 3.6, which uses RMSprop with a good accuracy. This shows how small changes can make a world of difference when designing a neural network.

3.4 Implementation, set-up, XY-table

In figure3.7the aluminum plate with electrodes can be seen. This is what the electrodes will be placed on and the plate will be set on the XY-table for the visual control and after this the plate will be moved from the XY-table to the robot that will assem- ble the electrodes onto the bodies. This will lower the risk of damaging the electrodes since there is no need to touch them directly. Before the automation, the electrodes were placed in plastic plates, as seen in figure3.8, where the manual inspection was done with tweezers and first moved into the plastic plate, which had deeper holes than the aluminum plate, and then it was inspected with the microscope by using the tweezers. After they were controlled they were moved to the aluminum plate.

The tweezers have a moderate risk of damaging the electrode and by minimizing human interaction with the electrode the risk of damage is also lowered. The plac- ing on the aluminium plate is much tighter than on the plastic plates, which hinders the electrode from moving, which also poses a risk of damaging the electrode.

(35)

FIGURE3.5: The best performing network

FIGURE3.6: A similar network that did not give as good accuracy

There were a lot of problems initially with the XY-table. An XY-table from Standa (8MTF - Motorized XY Scanning Stage) was used, and the programming package that came with it did not work with the computer that was used in this project. Turn- ing off the anti-virus and firewall resulted in the program being able to be installed

(36)

3.5. Results 25

FIGURE3.7: Aluminum plate for the electrodes

and after that it worked with the anti-virus turned on.

The XY-table had a large quadratic hole in the middle, which posed a challenge since there was a need to find something that covered it and stayed still, and that the aluminum plate the electrodes was placed on was in the correct position. It could have been solved by letting the user place it correctly from the start, but this is a source of error that could be built around, and if if possible, it should be. Prototyp- center at KTH was contacted to make an acrylic plate with holes for screws and a milled hole that fit the aluminum plate perfectly. It consists of two plates, with the exact dimensions of the moving part of the XY-table. The upper plate has a laser cut hole for the aluminum plate and the plates have holes for screws to fit it to the XY-table. This will let the user place the aluminum plate without having to think about it, it will always be in the right position. In figure3.9the plates can be seen on top of each other and in figure3.10it can be seen how the aluminum plate fits in the laser cut hole.

3.5 Results

The accuracy reached 98.9% 34424 images was used for training and 8809 images was used for validation, which is a division of 80/20 of the data set The same accuracy was achieved when the division of the data set was 60/40, but with a loss of 0.038 instead of 0.031 as the 80/20 division had. In Peltarion testing and validation

(37)

FIGURE 3.8: Plastic plate that is used for the manual control. The holes are deeper to prevent electrodes from falling out and they are wider so that the tweezers are able to reach down and lift them up

data is the same, but when the network was used with real time data, there was one or two images that were wrongly classified for every 200 images.

The preprocessing of the images works well when real time testing, but the XY- table takes a long time to connect, which makes the process slower to start than what was expected.

(38)

3.5. Results 27

FIGURE3.9: Plastic plate manufactured by Prototypcenter at KTH

(39)

FIGURE 3.10: The aluminum plate fits perfectly in the carved out space

(40)

29

Chapter 4

Concluding remarks

4.1 Discussion

It will be possible to use the project in production, the accuracy that was achieved can measure up to the accuracy of a human classifier. The human classifiers often disagree with each other and are sometimes inconsistent. This network will always be consistent and not drift towards some other rule of classification. Before the accuracy we would expect was known, there was an idea to do a test of the people in production to see how well they agree on a data set of images and to compare their results with the way that the neural network classified. Since we got such high accuracy that test won’t be necessary, but the network will be tested the same way a human would be tested before being accepted.

4.2 Recommendations

The recommendation I have is to go through the results and save more data when the program is being used in production. This will make the network even more ver- satile and since some faults come up because of a trend in production, the network will become stronger and more general when the data gets updated. The re-training of the data would be done by using Peltarion, the network structure is saved there at Scibase login and the only thing to do would be to upload the data set, push a train-button and then change two lines of code in the python-program, which are the keys to the deployed network.

4.3 Future work

The principle of this set up is possible to apply to a later visual control, which is wanted for a measuring station that controls the electric properties and the final visual inspection before sending it out to the customers. This inspection is going to be automated in the future, but the scope of this thesis does not involve that part since it requires a very high standard of control. If we have 99% there is still 1% of the electrodes that will be falsely classified, which is 2 spikes per electrode, where only one incorrectly classified spike could mean an acceptable electrode is classified as unacceptable. Is this a number that is acceptable to send out to the customers? A risk analysis needs to be done to ensure what level of correctness is acceptable, and what the level is today? How much would this incorrect percent cost in production hours and resources? Does the cost of automation cover these costs? These questions cannot be answered by me at this stage, but it would be a quick thing to apply this network and image preprocessing to that measuring station. The only part that needs changing is the image processing to find the contours since they look a bit

(41)

30 Chapter 4. Concluding remarks

different mounted on the bodies. The electrodes mounted on bodies can be seen in figure4.1.

FIGURE4.1: Aluminum plate with the mounted electrodes after assembly. If this could be sent into an electric and visual test, even more

time could be saved.

More data is always the answer for reaching better classification. The new electrodes have just started to be produced when this thesis is written, and new changes could be made to the production process that changes the way the electrodes look.

Hopefully this project is general enough to be able to be applied to those changes, but the image preprocessing had to be changed drastically between the old and the new electrodes, which could be an indication that it might have to be changed again.

That is most likely the part that has to be reworked and the neural network with the data about the spikes can be reused for other changes.

Another electrode is under production, which does not have three dimensional spikes but flat ” spikes”. How these might be damaged has to be researched to tune this project for implementation, if that is relevant. Since they are flat they might not be broken except for scratches in the gold, and in this project the focus has been on the sharpness of the tips of the spikes. When this network is used in production, it would be possible to save all the sorted data to go through and retrain the network with more data, to make the network more robust to all types of changes.

(42)

4.4. Conclusion 31

4.4 Conclusion

The important part of a machine learning project is the data and the processing of the data. This takes the most time and is vital for the usability of the network. The actual network is easy to design when the data set is clear and have a good structure. I will take this with me into the next project I will take on. This project was a theoretical success, but theory has to be modified when it is applied to reality. Therefore it cannot be said if the project was a practical success yet.

(43)

(44)

33

Bibliography

Aston Zhang Zack C. Lipton, Mu Li Alex J. Smola. Adagrad.URL:https://www.d2l.

ai/chapter_optimization/adagrad.html.

Bushaev, Vitaly (October 2018, visited June 2019). Adam—latest trends in deep learning optimization. URL:https : / / towardsdatascience . com / adam - latest - trends - in-deep-learning-optimization-6be9a291375c.

Learned-Miller (February 2014; visited June 2019). Introduction to Supervised Learning.

URL:https://people.cs.umass.edu/~elm/Teaching/Docs/supervised2014a.

pdf.

library, OpenCV (visited June 2019[a]). Canny Edge Detector. URL: https : / / docs . opencv.org/3.4/da/d5c/tutorial_canny_detector.html.

— (visited June 2019[b]). finding contours algorithm. URL: https : / / docs . opencv .

org/3.1.0/d3/dc0/group__imgproc__shape.html#ga17ed9f5d79ae97bd4c7cf18403e1689a.

— (visited June 2019[c]). Image Filtering, Gaussian Kernel.URL:https://docs.opencv.

org/3.1.0/d4/d86/group__imgproc__filter.html#gac05a120c1ae92a6060dd0db190a61afa.

— (visited June 2019[d]). Morphological Transformations.URL:https://docs.opencv.

org/3.0- beta/doc/py_tutorials/py_imgproc/py_morphological_ops/py_

morphological_ops.html.

— (visited June 2019[e]). Otsu’s Binarization.URL:https://docs.opencv.org/3.4.

3/d7/d4d/tutorial_py_thresholding.html.

— (visited June 2019[f]). Smoothing Images.URL:https://docs.opencv.org/3.1.0/

d4/d13/tutorial_py_filtering.html.

— (visited June 2019[g]). Thresholding.URL:https://docs.opencv.org/3.4.3/d7/

d4d/tutorial_py_thresholding.html.

Mallick, Satya (visited August 2019). convexHull(..)URL:https://www.learnopencv.

com/convex-hull-using-opencv-in-python-and-c/.

Nielsen, Michael (visited June 2019). How the backpropagation algorithm works.URL: http://neuralnetworksanddeeplearning.com/chap2.html.

Suzuki, Satoshi and KeiichiA be (1985). “Topological structural analysis of digitized binary images by border following”. In: Computer Vision, Graphics, and Image Pro- cessing 30.1, pp. 32 –46.ISSN: 0734-189X.DOI:https://doi.org/10.1016/0734- 189X(85)90016- 7. URL: http://www.sciencedirect.com/science/article/

pii/0734189X85900167.

Wikipedia (visited June 2019[a]). edge detection algorithm.URL:https://en.wikipedia.

org/wiki/Edge_detection.

— (visited June 2019[b]). hysteresis. URL: https : / / en . wikipedia . org / wiki / Hysteresis.

Zeiler, Matthew D. (2012). “ADADELTA: An Adaptive Learning Rate Method”. In:

CoRR abs/1212.5701. arXiv:1212.5701.URL:http://arxiv.org/abs/1212.5701.

(45)

www.kth.se

TRITA-EECS-EX-2019:776

Automated visual evaluation of an electrode with neural networks

Automated visual evaluation of an electrode with neural networks

LOVISA COLÉRUS

R OYAL I NSTITUTE OF T ECHNOLOGY

M

T

Automated visual evaluation of an electrode with neural networks

Abstract

Acknowledgements

Contents

List of Figures

Chapter 1

Background

1.1 Introduction

1.2 Thesis objective

1.3 Delimitations

1.4 Choice of methodology

Chapter 2

Theory

2.1 Neural networks

2.2 The electrode

2.3 Open CV and image algorithms

Chapter 3

Implementation

3.1 Creating the data set

3.2 Data set

3.3 Neural network

3.4 Implementation, set-up, XY-table

3.5 Results

Chapter 4

Concluding remarks

4.1 Discussion

4.2 Recommendations

4.3 Future work

4.4 Conclusion

Bibliography

R ^OYAL I NSTITUTE OF T ^ECHNOLOGY