Software development of visualization tool for 3D

Full text

(1)LiU-ITN-TEK-A--11/021--SE. Software development of visualization tool for 3D cameras Martin Stenmarck 2011-04-28. Department of Science and Technology Linköping University SE-601 74 Norrköping , Sw eden. Institutionen för teknik och naturvetenskap Linköpings universitet 601 74 Norrköping.

(2) LiU-ITN-TEK-A--11/021--SE. Software development of visualization tool for 3D cameras Examensarbete utfört i medieteknik vid Tekniska högskolan vid Linköpings universitet. Martin Stenmarck Examinator Stefan Gustavson Norrköping 2011-04-28.

(3) Upphovsrätt Detta dokument hålls tillgängligt på Internet – eller dess framtida ersättare – under en längre tid från publiceringsdatum under förutsättning att inga extraordinära omständigheter uppstår. Tillgång till dokumentet innebär tillstånd för var och en att läsa, ladda ner, skriva ut enstaka kopior för enskilt bruk och att använda det oförändrat för ickekommersiell forskning och för undervisning. Överföring av upphovsrätten vid en senare tidpunkt kan inte upphäva detta tillstånd. All annan användning av dokumentet kräver upphovsmannens medgivande. För att garantera äktheten, säkerheten och tillgängligheten finns det lösningar av teknisk och administrativ art. Upphovsmannens ideella rätt innefattar rätt att bli nämnd som upphovsman i den omfattning som god sed kräver vid användning av dokumentet på ovan beskrivna sätt samt skydd mot att dokumentet ändras eller presenteras i sådan form eller i sådant sammanhang som är kränkande för upphovsmannens litterära eller konstnärliga anseende eller egenart. För ytterligare information om Linköping University Electronic Press se förlagets hemsida http://www.ep.liu.se/ Copyright The publishers will keep this document online on the Internet - or its possible replacement - for a considerable time from the date of publication barring exceptional circumstances. The online availability of the document implies a permanent permission for anyone to read, to download, to print out single copies for your own use and to use it unchanged for any non-commercial research and educational purpose. Subsequent transfers of copyright cannot revoke this permission. All other uses of the document are conditional on the consent of the copyright owner. The publisher has taken technical and administrative measures to assure authenticity, security and accessibility. According to intellectual property law the author has the right to be mentioned when his/her work is accessed as described above and to be protected against infringement. For additional information about the Linköping University Electronic Press and its procedures for publication and for assurance of document integrity, please refer to its WWW home page: http://www.ep.liu.se/. © Martin Stenmarck.

(4) Abstract The visualization is an important and vital tool of describing complex data for an uninitialized user. SICK has developed an array of advanced machine vision cameras all capable of producing large data sets of information at a high rate. Although capable of fast data handling and the ability of controlling robots and other types of controls, the data used for these calculations can be showed for an easy and fast feedback to the users or in a sales situation. The visualization available today, although fairly fast, is limited to a gradient textured model with few tools. Based on OpenGL the current visualization requires a wrapper to handle correctly within .NET applications which is the preferred platform at SICK. This thesis studies the possibility and development of new visualization tool based on .NET capable of adding textures and interaction with the visualization. Since the data derived from the different cameras can occur in so many different ways a robust application ia needed. With the use of Windows presentation Foundation (WPF) framework the application is divided in three components. The first one is a library capable of reading the correct section of the file and then structuring the data. The second component is a user control; this allows the visualization to be constructed and displayed with ease within all .NET applications. The third component is a application which make use of the two other components to display and interact with the visualization. The developed visualization can display several different textures and has support for full navigation in the form of zoom, panning and rotation. The application is developed in .NET for easy integration with other applications and tools. WPF has a straightforward way of handling 3d content, however it is mostly used for small elements of 3D not for as large models as provided by the cameras. By subsampling the data set, the loading time can be reduced to an acceptable value. Although visualizing the data in a detailed and interactive way, the WPF-framework is on its knees when handling the larger files. There are several limitations in WPF which indicates the unsuitability as the main graphical engine.. Sammanfattning Visualisering är ett viktigt och nödvändigt verktyg för att beskriva komplexa data för en oinitierad användare. SICK har utvecklat en rad avancerade kameror som alla kan producera stora datamängder i hög takt. Även om kamerorna klarar av snabb datahantering och kan styra robotar och andra typer av kontroller, kan datan som används för dessa beräkningar även visas för en enkel och snabb v.

(5) vi återkoppling till användare eller i en säljsituation. Visualisering som finns idag är begränsad till en enkel modell med få verktyg. Denna avhandling studerar möjligheten och utvecklingen av ett nytt visualiseringsverktyg. Eftersom datan kommer från olika kameror behövs ett robust och mångsidigt system. Med hjälp av Windows Presentation Foundation (WPF) som ramverk är programvaran indelad i tre delar. Den första är ett bibliotek som kan läsa rätt del av filen och sedan strukturera data. Den andra komponenten är en kontroll, vilket gör att visualisering kan byggas och visas med lätthet inom alla .NET-applikationer. Den tredje komponenten är en applikation som använder sig av de två andra komponenterna för att visa och interagera med visualiseringen. Den utvecklade visualisering kan visa flera olika texturer och har stöd för full navigering i form av zoom, panorering och rotation. WPF har ett enkelt sätt att hantera 3D-objekt, men det används mestadels för mindre tillämpningar och inte för så stora modeller som tillhandahålls av kamerorna. Genom att sampla ner datamängden, så kan belastningen minskas. Även om kraven uppfylls och programvaran är användbar så finns det flera begränsningar i WPF som indikerar att det är en olämplig plattform för denna uppgift..

(6) Acknowledgments I want to thank my supervisor at SICK, Andreas Wrangsjö for his guidance and helpful tips along the way of this master thesis. I want to thank Thomas Berglund at SICK for helping me with troubleshooting the code. I would also like to thank everyone at SICK in Linköping, for the opportunity to work with this problem and for interesting insight and discussions. I would like to thank my examiner, Stefan Gustavsson at Linköpings University for providing good directions. I thank Jousef Gabro for reading and providing me with good feedback on the report. I would like to thank Emma Carlsson for supporting and helping me with the presentation of this master thesis.. vii.

(7)

(8) Contents 1 Introduction 1.1 Background . . . 1.2 Problem . . . . . 1.3 Purpose and goal 1.4 Thesis outline . . 1.5 Related work . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. 1 1 2 2 3 4. 2 Theory 2.1 Visualization . . . . . . . . . . . . . . 2.2 3D graphics . . . . . . . . . . . . . . . 2.2.1 An introduction . . . . . . . . 2.2.2 Navigation . . . . . . . . . . . 2.3 SICK machine vision . . . . . . . . . . 2.3.1 Machine vision product family 2.3.2 Data acqusition . . . . . . . . . 2.4 Windows Presentation Foundation . . 2.4.1 Extensible Application Markup 2.4.2 WPF3D . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Language . . . . . .. . . . . . . . . . .. . . . . . . . . . .. . . . . . . . . . .. . . . . . . . . . .. . . . . . . . . . .. . . . . . . . . . .. . . . . . . . . . .. . . . . . . . . . .. . . . . . . . . . .. . . . . . . . . . .. 5 5 6 6 8 10 10 12 15 17 17. 3 Implementation 3.1 Language choice . . . . . . 3.2 Software development . . . 3.2.1 XML reader . . . . . 3.2.2 3D models . . . . . . 3.2.3 Textures . . . . . . . 3.2.4 Tools . . . . . . . . 3.3 User Interface development. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. 21 21 21 21 23 25 26 27. 4 System overview 4.1 Class library . . 4.2 User control . . . 4.3 Application . . . 4.4 Use case scenario. . . . .. . . . .. . . . .. . . . .. . . . .. . . . .. . . . .. . . . .. . . . .. . . . .. . . . .. . . . .. . . . .. . . . .. . . . .. . . . .. . . . .. . . . .. . . . .. . . . .. . . . .. . . . .. 29 29 30 30 31. . . . . .. . . . .. . . . . .. . . . .. . . . . .. . . . .. . . . . .. . . . .. . . . . .. . . . .. . . . . .. . . . .. . . . . .. . . . . .. . . . . .. ix. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . ..

(9) x. Contents. 5 Results 5.1 The new visualizer . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 33 34 35. 6 Discussion. 37. 7 Concluding remarks and future work. 39. Bibliography. 41. A System Requirement Specification A.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . A.1.1 Purpose and goal . . . . . . . . . . . . . . . . A.1.2 Product usage . . . . . . . . . . . . . . . . . A.1.3 Product background . . . . . . . . . . . . . . A.2 General system idea . . . . . . . . . . . . . . . . . . A.2.1 Coarse description of the product . . . . . . . A.2.2 Product components . . . . . . . . . . . . . . A.2.3 Dependencies . . . . . . . . . . . . . . . . . . A.2.4 Design philosophy . . . . . . . . . . . . . . . A.2.5 General requirements on the complete system A.3 The class library . . . . . . . . . . . . . . . . . . . . A.3.1 Initial description of the class library . . . . . A.3.2 User interface . . . . . . . . . . . . . . . . . . A.3.3 Design requirements . . . . . . . . . . . . . . A.3.4 Functional requirements for the class library . A.4 The application . . . . . . . . . . . . . . . . . . . . . A.4.1 Initial description of the application . . . . . A.4.2 Design requirements . . . . . . . . . . . . . . A.4.3 Functional requirements for system 2 . . . . . A.4.4 User interface . . . . . . . . . . . . . . . . . . A.5 Performance requirements . . . . . . . . . . . . . . . A.6 Further requirements . . . . . . . . . . . . . . . . . . A.7 Documentation . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . .. 43 43 43 43 43 44 44 44 44 45 45 45 45 45 45 46 46 46 46 47 47 47 47 47. B System Design Specification B.1 Introduction . . . . . . . . . . . . . . . . B.1.1 Purpose of this document . . . . B.1.2 Scope of the development project B.1.3 Definitions . . . . . . . . . . . . B.1.4 References . . . . . . . . . . . . . B.1.5 Overview of document . . . . . . B.2 System Archtecture description . . . . . B.2.1 Overview of modules . . . . . . . B.2.2 Structure . . . . . . . . . . . . . B.2.3 User interface issues . . . . . . . B.3 Detailed description of components . . .. . . . . . . . . . . .. . . . . . . . . . . .. . . . . . . . . . . .. . . . . . . . . . . .. . . . . . . . . . . .. . . . . . . . . . . .. . . . . . . . . . . .. . . . . . . . . . . .. 48 48 48 48 48 48 48 49 49 49 50 50. . . . . . . . . . . .. . . . . . . . . . . .. . . . . . . . . . . .. . . . . . . . . . . .. . . . . . . . . . . .. . . . . . . . . . . .. . . . . . . . . . . ..

(10) Contents B.3.1 Component Overview . . . . . . . B.3.2 Class library . . . . . . . . . . . . B.3.3 Desktop Client . . . . . . . . . . . B.4 Design decisions and tradeoffs . . . . . . . B.4.1 Windows Presentation Foundation B.4.2 Triangulation . . . . . . . . . . . . B.4.3 Decimation . . . . . . . . . . . . .. xi . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. 50 50 50 51 51 52 52. C Usability test form. 53. D Screenshots. 54.

(11)

(12) Chapter 1. Introduction 1.1. Background. SICK IVP AB develops software, supports and markets a range of cameras, mainly machine vision used in the production industry. The hardware is manufactured in Germany but all software development, research and support is done in Linköping, Sweden. The cameras are all specialized within a specific area of inspection which ranges from barcode and text reading to volume calculation and surface inspection. Since practically all cameras are of a "smart" type (all calculations and image processing occur in the camera) the resulting values are often used as input for other machines to make the correct choice in the production link. In some cases the camera requires a good calibration object to learn or to base its settings on. The visualization of this object’s data is vital for the configuration of the program and camera parameters. As mentioned all cameras have a specialized area of operation and these areas are some of the cameras focused on in this thesis: • Vision Sensors (CVS): These are small devices specialized on a few tasks such as contour verification, text reading and color sorting. The sensors are easy to install and will produce the result without the help of an external computer. • Smart Cameras: This type of camera as a built-in image analysis unit and just as the vision sensors produces a result without the help of a PC. There are two sub categories within the smart camera title: – 2D Smart (IVC-2D): This is a stand alone vision system for 2D analysis which can detect edges and measure positions of details. – 3D Smart (IVC-3D): These cameras are used for inspecting the surface or measuring height and volume. The camera scans calibrated data and can without the help of a PC produce a result. • PC-based Systems: When large amounts of data needs to be processed and analyzed the cameras stated above are not the recommended. Instead, a PC-based system is preferred. 1.

(13) 2. Introduction – 3D Camera (Ruler): Calibrated 3D data is collected and is streamed to a PC for analysis. This is used for i.e. knot detection for logs. – MultiScan Camera (Ranger): The Ranger system is capable of performing several scans simultaneously generating, for example, 3D data, 2D color image and an intensity image all in one scan. This system requires a more advanced lighting setup.. All cameras have special software and therefore separate ways of interpreting and displaying data.. Figure 1.1: Picture of an IVC-3D Camera. The tilted glass window on the left contains the camera and the round glass on the right is a calibrated laser.. 1.2. Problem. There are many technical advanced cameras at SICK, and they all have their own formats and data representation. The cameras functionality is not affected by not having a straight forward way of displaying a visualization, but when presenting the camera system, or the data produced by the cameras, there is no simple way of displaying examples. Today when data from a camera shall be displayed for a customer or at an exhibition you would need a station with the right software and the right settings to see the result. This software is of a high technical level and not always suitable in marketing or display situations. The current visualization is not native to the .NET platform which leads to customization and implementation problems.. 1.3. Purpose and goal. The purpose of this thesis is to simplify the process of displaying different generated camera data. The main goal of this thesis is to develop a software capable of visualize all formats generated by the cameras at SICK. The goal is divided in three major parts:.

(14) 1.4 Thesis outline. 3. Figure 1.2: Example of visual result from the current visualizer.. • Viewer Application: The goal is to develop software with accompanied installer capable of running on a Windows XP based laptop. This software should be able to load all formats and display the suitable information. • User control: The goal is to develop a control based on the visualization library that easily can be included in other applications. • Visualization library: The goal is to develop a library of useful classes and functions designed to handle and process the file formats resulting in a visualization. By creating an application and a library that will be able to load every format, and displaying the data, it will make it easier for demonstrations. The application should be focused on the visualization and not require any deeper understanding of the technology behind the cameras. Furthermore, the application could make simpler customer support errands faster by showing the camera technicians and retailers the exact grab settings active when the file was saved, and thereby speeding up the troubleshooting.. 1.4. Thesis outline. In chapter 2, background on visualization and 3d graphics can be found followed by a presentation of the different formats of the SICK cameras and Microsoft Windows Presentation Foundation1 . Chapter 3 is a guide through the implementation of the new visualization and explanations of the design choices. Chapter 4 explains the system and all of its components. In chapter 4 information about the UI development can also be found. Chapter 5 presents the results which are 1 WPF..

(15) 4. Introduction. then discussed in chapter 6. Chapter 7 ends the report with final thoughts and suggestions of future work.. 1.5. Related work. The first visualization was developed for trade fair and sales purposes and has since then been implemented in several applications. Today SICK uses several programs to view the data from the different cameras. For the Ranger cameras the application Ranger Studio is used for examining and viewing the data, the IVC uses IVC Studio for all the viewing as well as for the programming of the cameras. The visualization in both these cases is based on QT2 3 and OpenGL and provides a fast and scalable visualization although it has some limitations. The largest limitation is the framework. Almost all applications developed today at SICK use Java or .NET, and since the current visualizer is based on C++, a wrapper is needed to use the visualization in other applications. Another limitation is the navigation, the OpenGL visualization only handles rotation, and lacks pan and zoom capability. On the other hand, it makes use of polygon optimization in the form of LOD3 and will only display a set number of polygons; this is useful for displaying very large data sets. If you want a higher level of detail of a large data set you can crop the data set and the visualization will re-optimize the cropped data set and display it in higher resolution.. 2 Application 3 Level. framework used for developing graphical applications. of detail..

(16) Chapter 2. Theory 2.1. Visualization. The importance of data visualization is rapidly increasing due to more advanced ways to collect data. In order to make sense of huge data sets a visual simplification can be very effective. In many occasions the data is interpreted and analyzed trough a computer and the simplification is never needed, but as soon as the data shall be presented to a developer or a customer, the opportunity to visualize the data is often a better way than showing them the raw values. Visualization has the advantage of displaying a context and several different data at the same time. By adding animation, large data sets can be easily understood and interpreted, for example the airflow in a room; the air is constantly moving and by analyzing the values from the sensors one would eventually find intake and exhaust of the air. Over time these values effect each other, and the circulation is hard to see. By using a visualization one can get a fast grasp on the movement and strength of the airflow by combining color, glyphs and animation as seen in figure 2.1. This technique is frequently used in the airplane and car industry to see the drag of the air against a prototype; this reduces the cost of building the prototype and to test it in a wind tunnel.. Figure 2.1: Picture of visualized air flow inside a room 5.

(17) 6. Theory. 2.2. 3D graphics. 2.2.1. An introduction. To describe a point in three dimensions you need the three axis; X, Y and Z. To describe an area in its smallest form you need three points (called vertex in this context), these will then describe the three corners of a triangle. This is called a polygon. A polygon can have many different appearances and several points, but the smallest version in 3D context is the triangle. So a triangle consists of three vertices, each with three coordinates. Several polygons together create a mesh, which can be described as a 3D grid outlining the model. To calculate how many polygons are used in the objects used in the scope of this thesis one can use equation 2.1. polygons = (width − 1) ∗ (height − 1) ∗ 2. (2.1). However, a triangle also need a normal, this is a vector describing which way the surface is visible from, the normal also effects the lighting of the polygon face. All vertices have a normal and at render time the surface will interpolate the resulting normal from the corner vertices as seen in figure 2.2. There are several ways of describing these normals, and in many cases it is necessary to tilt them, mostly for the manipulation of light. An easy way of describing a standard normal is by using your right hand and with the thumb pointing up, follow the points describing the area with your fingers. By following the vertexes in a counterclockwise fashion your thumb will point in the normals direction. So now we also know that the order of points is of value as well. The more mathematical way is by taking the cross product of the vector spanning between points 2-3 with the vector between points 2-1.. Figure 2.2: Interpolation between two normals along a surface vector By manipulating the normal of the polygon, the light can be directed. The more normals the model have, the more smooth will the surface be perceived. And since the normals are connected to the vertexes a smoother surface can be achieved by increasing the number of polygons..

(18) 2.2 3D graphics. 7. (a) Point order. (b) Resulting normal. Figure 2.3: Triangle and normals creation. When creating a cube there are some surfaces that share corner vertexes but will face in total different directions. This corner vertex will not be able to describe all directions with one normal. One way of solving this is by declaring each corner vertex three times (up to six depending on the triangle orientation) and specify a separate normal for each instance. This way the amount of data increases quite fast but the edges will be clearly defined. In figure 2.4c an example of a cube with one normal describing all three surfaces is shown and in figure 2.4d each corner vertex is declared three to six times with separate normals displaying each side with correct amount of shading. The third important lesson about 3D is the texturing of the model. After creating all polygons and their normals the object has a uniformly colored area, often a gray shade is used for the polygons. However by adding a texture to the model we can simulate details in the object without increasing the polygon count. A texture is in most cases a two-dimensional image containing color or grayscale information, (in this context the information is a photograph taken from the same angle as the height data) which is placed on the model. The way the texture is placed must be described with the help of texture coordinates. In the example with point-order above, a list with three points creates a polygon, in the case of textures, only two coordinates are required. By setting a X and a Y value we can tell what part of the texture image will be placed on the current point. There is a technique which combines both texture maps and the normal manipulation. The normal map is created by illuminating a high resolution 3d object with colored light, different color from above, below, left and right. All details will get a different hue depending on how much of their surface face one direction. After applying this map to the model it can simulate very fine detail on a flat surface. This is often used in the gaming industry to keep a high frame rate but with great detailed models..

(19) 8. Theory. (c) Shared corner vertex with one nor-(d) Separate corner vertexes with sepmal arate normals. Figure 2.4: Visual impact of normal direction. (a) Standard normal reflection. (b) Manipulated normal reflection. Figure 2.5: Normal manipulation effect on light. 2.2.2. Navigation. While displaying the model in three dimensions is a step forward in the visualization, the ability to navigate the visualization and show the same data from.

(20) 2.2 3D graphics. 9. (e) Untextured model. (f) Textured model. Figure 2.6: Untextured and textured models. different angles is a necessity. There are many applications on the market today displaying 3d content, and almost all have their own approach on how their visualization should be navigated. One of the most common and intuitive ways of interaction and navigation is by the use of the computer mouse. By clicking and dragging the mouse you can control two axes at the same time, and by use of other buttons other axes can be activated. The developed application uses a modified version of 3D Tools for the Windows Presentation Foundation developed by Microsoft, allowing basic navigation (pan, zoom, rotate). The navigation simulates a sphere around the object, placing the point at which the camera is focused on in the center. When clicking and dragging,.

(21) 10. Theory. a virtual vector is created from the origo to the point on the sphere that project to the point on the screen.. Figure 2.7: Navigation with Microsoft 3D Tools image source: http://viewport3d.com. 2.3. SICK machine vision. Almost all production lines have stations that control the results. Since the industrial revolution and the invention of the serial production assembly line the verifying of the products has been in the hand of human controllers. There is always a margin of error when human control is involved, not to mention the cost and restrictions in workable hours. By developing more advanced technology, more and more tasks previously performed by humans can be controlled by computers and machines faster, cheaper and longer times between services. The technology is in many regards even more reliable than humans, especially when error margins are small and tasks involve heavy calculations. A computer can work around the clock and always with the same result. Along with the development of faster computers so can the speed of controls and measurements increase.. 2.3.1. Machine vision product family. Vision sensors The vision sensor is a system used for specific tasks at high speed for example read text, check for color or orientation. These systems are very small and robust, and due to their rather simple tasks they are very simple to setup. The vision sensor system is 2D and provides bitmap images..

(22) 2.3 SICK machine vision. 11. Figure 2.8: Picture of two Inspector cameras.. Smart cameras. The smart cameras are separated into 2D and 3D cameras. Both systems contain built-in image processing capabilities and can be used without a connection to external computers. The 2D cameras are used for image analysis of object positions or patterns. The image processing tools can for example rotate an object, match measurements of details and classify them as pass or fail. Areas of use include label positioning, detail and hole positioning. The 3D cameras make use of a calibrated laser for calculating the height of the object. This is used for measuring the objects height, volume and shape. There are many situations where 3D analysis is more powerful than using 2D, for example when analyzing connector pins on a cable. On the other hand, the IVC3D cannot analyze the placement of the image on a candy wrapper. Other areas of usage include, surface inspection (dents), volume inspection (food industry) and height/depth of details on the object. The smart camera uses an graphical GUI to program them, by dragging image analysis components into the program an advanced series of programming steps can be constructed very easily making the cameras very easy to use and fast to install in a production line. An example of a IVC-3D camera can be found in figure 1.1.

(23) 12. Theory. PC-based cameras The PC-based cameras are very advanced cameras which collect large amounts of data that requires to be transfer to a computer for analysis. These systems are used in many different industries, amongst these are in sawmills; calculating where to make the cut in the logs to get the optimal amount of timber. The PC-based cameras also have the advantage of being able to multiscan, meaning that the image sensor can collect several types of images in one scan. This enables the PC-based cameras to collect for example a grayscale image as well as 3D data. The types of images that can be collected are; color images, scatter images, grayscale images and 3D images. Another advantage of these cameras is their speed. The cameras are capable of scanning many thousand profiles per second, making the level of detail very exact. By combining different image analysis components, these camera systems are very capable and can be customized to every need. Since the amount of data transferred from the PC-based cameras is so large, a computer is needed to control the camera and analyze the results. The programs required are often so unique that they are individually created for the specific tasks. There are several components(image types) available for the PC-based cameras, each designed for different purposes. Some components deliver several images and some are specialized on speed. The components are: • Hi3D - High resolution 3D • HorThr - 3D Horizontal Threshold • HorMax - 3D Horizontal Maximum • HorThrMax - 3D Horizontal Threshold Maximum • Gray - 2D Grayscale image • Scatter - 2D Scatter image The areas right above and below the laser line contain the amount of the scattered light transported inside the material currently scanned. This is valuable when scanning non transparent objects (i.e. medicine pill containers)or objects with varying light scattering properties (i.e. wood) • HiresRGB - 2D RGB1 color image. 2.3.2. Data acqusition. All the cameras manufactured by SICK have the advantage of being quite simple in their setup. The main deployment of the relatively small camera systems is in an assembly line. The camera system consists of the camera housing, electrical sensors and in the case of advanced 3D systems, a calibrated laser. The laser 1 Red,. Green, Blue.

(24) 2.3 SICK machine vision. 13. sweeps over the object illuminating the surface variations. The camera registers the laser line and can extract a profile of the object. After the object has been scanned the profiles can be added to create the 3D data.. Figure 2.9: Example of scanned laser profiles that converts into a height map. When scanning an object there are some situations which affect the detail and information of the scan: • Reflectance When the object surface is very reflective it might, at certain angles, reflect all light away from the camera lens, resulting in no profile being registered. This is common when scanning polished metal. • Laser occlusion When the laser is positioned so that the object to be scanned shades some areas from the laser. • Camera occlusion When the object shades areas making them not visible to the camera. Rigs In the test environment a hand controlled conveyor belt is used as shown in figure 2.11. The test rig consists of an encoder, the conveyor belt, a sensor, the camera and a calibrated laser. The encoder provides information about the movement of the conveyor belt. The sensor can be used for controlling the camera. When using a Ranger based camera the laser is placed freely while IVC has the laser built in. The resulting data is transferred through a fast gigabit Ethernet cable..

(25) 14. Theory. Figure 2.10: Laser and camera occlusion.. Figure 2.11: Test rig for Ranger camera.. Data formats The data returned from an IVC-3D camera is formatted with a header describing the basic information of the file. After the header the first segment of data is found. The 8-bit data has low precision and is an aged format but still active since there are many applications that use it. The next segment is the 16-bit data which is the most exact and usable data today. The 16-bit data is also often accompanied with the last segment; the extended header. The extended.

(26) 2.4 Windows Presentation Foundation. 15. header includes more information about settings used for the camera at the time of the data acquisition. This information is also vital for scaling the 3D mesh and positioning of the model.The file format of an IVC-3D camera is IMG.. Figure 2.12: IVC file format. In the case of the more advanced Ranger cameras, the format can be a bit different depending on the settings. The most common layouts are Scan- and Subcomponent-layout. In the case of subcomponent layout the different components are next to each other as seen in figure 2.14. In Scan-layout the components are arranged above and under each other as seen in figure 2.13. This determines the way the file should be read. The Ranger cameras deliver two files, one DATfile containing all the raw data and one XML-file declaring what components are included and their respective size.. 2.4. Windows Presentation Foundation. Windows Presentation Foundation, WPF, is a subsystem for developing graphical user interfaces for the Windows platform. It is now a part of the .NET Framework.

(27) 16. Theory. Figure 2.13: schematics for a Ranger data format in scan layout.. Figure 2.14: schematics for a Ranger data format in subcomponent layout.. and is seen more or less as a replacement of winforms. Instead of relying on the GDI2 engine to render all graphics, WPF make use of DirectX. The user interface is constructed through WPF using XAML3 code. Another thing making WPF 2 Graphics. Device Interface Application Markup Language.. 3 Extensible.

(28) 2.4 Windows Presentation Foundation. 17. a very competent development environment is the data bindings which are very customizable and every user control can contain other controls.. 2.4.1. Extensible Application Markup Language. The Extensible Application Markup Language, XAML, is developed by Microsoft and is based on standard XML. It is designed to be simple and the files are editable without any special compiler. In XAMLs case, they describe how the user interface and interactivity is constructed. There are many similarities with HTML4 coding. Every element starts and ends with respective tags. And element can contain input parameters (i.e. width and height for images or work areas). In WPF, elements and objects can be created either in "code-behind" or in the XAML interface. One of the ideas behind WPF is to be able to separate the workflows involving the user interface with the functionality programming.. 2.4.2. WPF3D. WPF is a very good tool for developing windows applications due to its many predefined graphical presentation controls (i.e. sliders, buttons and canvases). Since it uses DirectX as renderer it also enables some basic 3D capabilities. To display a 3D visualization in WPF there are some steps that needs to be taken: • Viewport3D The viewport describes the area in which the 3D objects can be displayed. • Camera The camera is needed to tell the viewport3D in what direction the user shall face. There are several different cameras; perspective camera, giving a realistic display with objects getting smaller depending on placement from the camera. Orthogonal camera displaying all lines parallel and no vanishing point. • Navigation Decorator The viewport is static and the camera is declared at a certain point, facing a certain point. To add interactivity to the viewport a decorator can be used, managing all inputs when the viewport is active (i.e. clicking and dragging in the viewport). • Light The viewport can be seen as a closed box and the content of the viewport will not be visible if there is no source of light. There are four standard lightsources; point light, spot light, directional light and ambient light. All giving different effects to the scene. 4 HyperText. Markup Language.

(29) 18. Theory • VisualModel3D The VisualModel3D is the object to be displayed in the viewport. The model is a 3D object containing data describing, point positions, triangle indices, normal directions and texture coordinates.. Figure 2.15: Structure of WPF3D. VisualModel3D The 3D object contains a number of lists that describe its position, its shape, how and in what direction it reflects light and how textures are applied. To populate these lists many steps are required. These steps are visualized in figure 2.16. In WPF all graphical operations use double precision which in turn increases size of data that needs to be processed. The fundamental piece of data, the x-, yand z-coordinates of the vertices, are stored as doubles in a list. This list can then be used to populate the triangle indices by selecting each vertex in the right order. The triangle indices are stored as int32. The normals can be explicitly calculated and stored but, if no value is calculated, WPF derives the normal from the order of the triangle. All the information needed for the texture coordinates are also read from the original point list. The textures are only 2D and describe what coordinates in the texture image shall be mapped to each triangle. The xand y-coordinates of the texture are stored as doubles. These elements create the geometry of the model. The texture image is part of the material which also is needed. The material describes how light bounces off the model and what.

(30) 2.4 Windows Presentation Foundation. 19. properties the surface shall have. Together, the material and the geometry, make up the 3D model.. Figure 2.16: Structure of a VisualModel3D object.

(31)

(32) Chapter 3. Implementation 3.1. Language choice. As stated in chapter 2 the implementation language framework used is WPF. This is however not the only potential language to use. When developing 3D-orientated applications there are several languages to use. In the environment of managed code there is some support for Managed DirectX (MDX, SlimDX(open source), SharpDX(open source),). The Microsoft version was however discontinued and is now replaced with Microsoft XNA. XNA is a development framework specialized on game development for Microsoft systems (X-BOX, windows and windows mobile). These examples are all viable since they are connected to the .NETplatform and easily integrated with the current applications used at SICK. Since the project involved an application part, the pros of UI1 -coding and WPF resulted in the choice of .NET 3.5 and WPF as the development platform.. 3.2 3.2.1. Software development XML reader. The XML-reader is a class developed especially for the Ranger data format. It separates the vital information needed to extract the model from the information tags. There are other methods available for reading an XML-file but this method provided a more controlled and tailor-made process since the ranger format can come in very many varieties. The order in which components and subcomponents are declared is very important, and there is no standard in order or how much information is declared in each component. Therefore this class implements a robust XML-parser. To implement a new component a switch-statement needs to be altered to read the new component tag, and then place the valid information in their correct list. The lists are: 1 User. Interface. 21.

(33) 22. Implementation • IconSize Size of the Icon buffer • ComponentList A list describing all available components in a file. The data interval amount is structured in separate lists depending on the component. In every interval list five to six values are stored: • Offset before data • Size of interesting data • Offset after data • Data format (byte, int, float, double) • Scan format (subcomponent or scan) The interval lists are: • RangeInterval • IntensityInterval • ScatterInterval • RGBInterval • ColorInterval Apart from the interval data, the XML-files also contain some camera settings that can be used for scaling and prepare the model for the visualization. This information is placed in the list corresponding to the files components. • DCMsensortraits • DCMworldrangetraits • DCMworldrangetraits3d Listing 3.1: Example of XML code. <icon_data_format v a l u e t y p e=" " name=" " > <parameter name=" s i z e " >6144</ parameter> <parameter name=" v e r s i o n " >1</parameter> <parameter name=" l a y o u t " >SCAN</parameter> <component v a l u e t y p e="DCM" name=" Hi3D␣ 1 " > <parameter name=" s i z e " >6144</ parameter> <parameter name=" h e i g h t " >1</parameter> <s e n s o r r a n g e t r a i t s > <parameter name=" c o o r d i n a t e s " ></parameter>.

(34) 3.2 Software development. 23. <parameter name=" f o v ␣ x0 " >0</parameter> <parameter name=" f o v ␣ x1 " >1535</ parameter> <parameter name=" f o v ␣ x2 " >0</parameter> <parameter name=" f o v ␣ x3 " >1535</ parameter> <parameter name=" f o v ␣ z0 " >1</parameter> <parameter name=" f o v ␣ z1 " >1</parameter> <parameter name=" f o v ␣ z2 " >1023</ parameter> <parameter name=" f o v ␣ z3 " >1023</ parameter> <parameter name=" o r i g i n ␣x " >0</parameter> <parameter name=" s c a l e ␣x " >1</parameter> <parameter name=" o r i g i n ␣ z " >352.875 </ parameter> <parameter name=" s c a l e ␣ z " >−0.0625</ parameter> </ s e n s o r r a n g e t r a i t s > <subcomponent v a l u e t y p e="BYTE" name=" S c a t t e r " > <parameter name=" s i z e " >1536</ parameter> <parameter name=" width " >1536</ parameter> </subcomponent> <subcomponent v a l u e t y p e="BYTE" name=" I n t e n s i t y " > <parameter name=" s i z e " >1536</ parameter> <parameter name=" width " >1536</ parameter> </subcomponent> <subcomponent v a l u e t y p e="WORD" name=" Range " > <parameter name=" s i z e " >3072</ parameter> <parameter name=" width " >1536</ parameter> </subcomponent> </component> </icon_data_format>. 3.2.2. 3D models. The 3D data, as well as all visual information, stored in the files can be interpreted as 2D images. What differs between the intensity gray scale image and the 3D image is what the camera "sees". When interpreted as a gray scale image the 3D data pictures a height image. The whiter the pixel is, the higher is its vertical position. This is the basic step for creating the model. When the 3D data has been interpreted into a flat grid image, the triangulation only needs to position each vertex according to the height declared by the pixel value. Since the cameras only see the object from one direction the occlusion that occurs in areas hidden from the camera will be stored as missing data. Polygon optimization Since the data stored in the files can become very large, some kind of optimization is needed to reduce loading times and visualization speed. In many cases different implementations of decimation are used. When an object is far away from the camera, a lower resolution mesh could be used without the sense of lost detail..

(35) 24. Implementation. (a) 3D height data image. (b) Traingulation grid overlay. Figure 3.1: Vertex placement and height based on height image.. This technique is called level-of-detail, LoD. There are many different ways to create the mesh used for different levels, many of which are very effective and fast. However, these methods are of little use in this implementation since the 3D objects in WPF are constructed through lists. The possibility to search through and modify these lists would require too large of an effort and would reduce the frame rate. Another reason is the data format used in WPF, all graphical operations uses double precision. This together with the memory clearance issue makes such advanced functions unsuitable. The optimization is still required since the amount of data easily can exceed WPF limitations in regard of memory. To sort out this problem a subsampling of the mesh can be used. In this project two different methods were tested; the first method used a two-by-two pixel kernel to calculate the mean value of all vertex points, the second method resamples by removing every other column and row. The second method was found to be better since it does not modify the mesh as much as the calculated optimization. If the data is extremely large every third or fourth row and column can be used, but at that point details in the model will be affected. One of the reasons making this sort of basic sub sampling viable are the textures. Since the textures include many details not needed to be extruded from the model, a coarser model can be used. This line of thinking is frequently used in the gaming industry adding more detail in form of 2D textures applied to a low resolution mesh. Before implementing this solution, another method was tried. By using multiple threads on the CPU many tasks could theoretically be completed at the same.

(36) 3.2 Software development. 25. time thus resulting in lower loading times. However, WPF has the restriction of no threading of graphical objects. All graphical objects used are based on different arrays or lists which could be seen as non graphical objects, these are threadable but to merge them into the final graphical object still requires more computational power than gained from threading its parts.. 3.2.3. Textures. Since the details are reduced in the geometry the importance of the applied textures increases. The image data is extracted the same way as the height data except the images stay as a two dimensional array. When working with gray scale images only one channel is used as opposed to three channels, or even a fourth alpha channel, when dealing with color images. The intensity images from the Ranger cameras are stored as 8 bit per pixel indexed images. By declaring a palette ranging from 0 to 255 all gray values are accounted for. For the scatter data or other types of image data the same procedure can be used. However, there is some image processing that can be done to increase the information that will be displayed. By combining several types of data (i.e. Intensity and scatter) a more interesting texture can be created. Color images are often separated in three layers; Red, Green and Blue. Each layer is a gray scale image describing what areas are influenced and how much by respective layer. This setup is standard and also the way WPF constructs its color images. Another way of describing a color image is the HSV encoding. HSV stands for Hue, Saturation and Value. This encoding can also be seen as three single layer gray scale images describing their layer impact on the final color image. By using the HSV encoding and selecting the scatter image as hue, the intensity image as the value and setting the saturation to 1 (100%) the result is a new color image which depending on the scatter value will shift color while still displaying the details from the intensity map. One thing to keep in mind is that the texture images are always in full resolution since the details are needed to compensate for the reduced geometry and since there might be many active images, the memory allocated for managing these image objects increase very fast.. (a) Intensity texture. (b) Scatter texture. (c) Intensity and Scatter combined through HSV. Figure 3.2: Texture examples Another texture function used is the gradient. Apart from the image textures.

(37) 26. Implementation. this texture has its texture coordinates set along the height of the model. In WPF there is a type of brush (LinearGradientBrush) which can be used to create a gradient material. The LinearGradientBrush sets a start color, a stop color and as many color as needed between them. By setting at what point the middle colors will be at the gradient is created. This is also an interactive texture since the middle colors can be controlled by sliders to modify at what point they will be positioned between the start and stop colors. There is also support for full color textures, implemented at the very end of the thesis project. It combines three separate gray scale images into a full color image. The previous problems with color textures were not software based but more how to calibrate the camera sensor to sync the color image with the height data.. Figure 3.3: Full color texture. 3.2.4. Tools. While the focus of this master thesis has been the development of the .NET based visualization, there are some tools available in the implemented application. The first requirements included the ability to capture a screenshot of the visualization. This function simplifies usage of visualization data in documents and for support errands where the visualized image can focus on a certain area indescribable with text or without sending large datasets. The scale tool is vital since all data is not correctly stored in the files. The scale tool uses a simple scale matrix which controls the x-, y- and z-amplification. The received meta tags will often be enough for the scaling to be correct. But for example when scanning from different directions, the data could be mirrored and objects, including text, will be visualized wrong. By setting the x- or the y-amplification to negative one, a mirror effect will be created. The model translation tool controls the point describing the center of the model. This is needed mostly for the animation tool, but can also be used to.

(38) 3.3 User Interface development. 27. control the point at which the camera is focused on. In some usability tests the origo placement tool was confused with the panning functionality. For this reason the optional help buttons was implemented and a visual glyph. The Shear tool is used for tilting the model. This is useful when the camera and laser setup differs from the standard. To compensate from geometric placement of the camera all slices of the model can be tilted by a set degree. The Light Rotation control will let the user rotate the standard light source around the object; this is mostly a cosmetic tool but can clarify certain details when set with the right angles. The Animation tool is a slider which controls the speed for an animation of the model rotating 360 degrees. This tool is almost exclusively used for fair and demonstrations. The Intensity texture control is used for increasing the visibility of the intensity texture. By increasing the slider value a lighter version of the texture is created. To apply the effect the intensity texture needs to be reselected. The Range texture control is an extension of the intensity texture control. By setting both controls and clicking the update texture button a new height based gradient texture is created. The Gradient Texture tool controls at what points the different colors will be active. The control is interactive and the result can be viewed directly on the model if the gradient texture is applied.. 3.3. User Interface development. To create the visualization application user interface fast prototyping was used. By drawing areas onto a piece of paper, different solutions were developed. Since the focus is on the visualization the UI development consisted mostly of toolbar placement and research how users interact with the application. Through usability test the most frequent errors or time sinks were discovered and could be altered to increase novice user speed. For example grouping certain tools and dividing tools in different menus based on types. A toggled help system was also implemented for clarification when needed. The toolbar is divided into three parts each covering a certain type of tools. Some files (IVC) do not contain any texture information and will not make use of the texture tools. The different tool menus have their own size since some tools require a bigger visual feedback but the main idea is to always have the largest focus on the visualization. There is also an option in the top menu bar to remove the toolbars for increased visualization size..

(39)

(40) Chapter 4. System overview The resulting product is separated in three parts; the class library, the user control and the application. The class library contains the fundamental functions required by the user control and the application. If the user for some reason wants to develop their own user control or functionality the different modules are separable.. Figure 4.1: Hierarchy of the system.. 4.1. Class library. All the functions used to extract, modify and sort the data from the files are called from the class library. Some functions are self-explanatory and just return a value otherwise private. Other functions are more complex and make use the data to process and generate new information to be used in the visualizations. The most important function is the extractData which receives the lists and data from the XAML or IMG-header readers. Depending on the format of the scanned models, a certain portion of the function is executed. In the beginning of each extractData-sub function, a size control is conducted. If the size of the 29.

(41) 30. System overview. data is above a certain level a subsampling will occur. If the data is even bigger, a coarser subsampling may occur. After the height data is extracted the same function extracts the textures and creates respective texture objects. Another advanced function is createRGBthroughHSV. This function receives two 8 bit per pixel gray scale images and combines them into a 24 bits per pixel full RGB image. This function is quite memory intensive since all three images (the two received and the third, generated) will have to be read and processed. Each pixel in the resulting image is calculated as followed; C = chroma value, set to the texture value (intensity texture in the application) since saturation is set to max; 1. To decide what Red, Green and Blue values should be used a variable is created then tested against an interval. H’ = Hue prime. Phase shift of the texture used as hue, in the applied solution the hue texture is divided by a factor of 20 then a modulus six is used to get H’ in the right interval. X = Color variable, calculated as C * (1 - abs((H’ mod 2) - 1)) The resulting RGB values per pixel is calculated as followed:  (0, 0, 0) if H is undefined     (C, X, 0) if 0 ≤ H0 < 1     if 1 ≤ H 0 < 2  (X, C, 0) (0, C, X) if 2 ≤ H 0 < 3 R, G, B =   (0, X, C) if 3 ≤ H 0 < 4     (X, 0, C) if 4 ≤ H 0 < 5    (C, 0, X) if 5 ≤ H 0 < 6. 4.2. User control. The user control is the first interface between the user or application and the class library. It prepares functions such as navigation, load sequences, lighting and animation. The user control can be seen as an application with many loose threads, it is up to the application that implements the user control to tie theses loose ends to controls. From a visual point, the user control only presents the visualization window.. 4.3. Application. The developed application ties up the loose ends from the user control and extends a visual interface. There is a horizontal menu, a vertical menu and a top menu. The horizontal menu contains all available tools, the vertical menu contains all available textures and the top menu is standard application activities like help, about and file. The tool menu is separated based on area of functionality and.

(42) 4.4 Use case scenario. 31. required size. The texture menu is based on visual representation with clickable areas and an extra option to view the texture as a normal image. The application is installed through a standard windows installer that includes an uninstaller.. 4.4. Use case scenario. The typical usage of the application is as a sales or demonstration tool. It can present all standard formats from the SICK camera series and with basic tools for edit and modify the visualization. The user control can be inserted to any .NET application with great ease and will add functionality to any program in need of visualizations. The class library is a blue print of one way to solve the 3D visualization of camera data and could be transferred to other development environments for increased performance. In the scope of the application the use case scenario is a demonstration of a file or a first look of a customer’s data..

(43)

(44) Chapter 5. Results The result of this master thesis can be viewed from two directions, by fulfilled requirements and by the delivered application. By following the requirement specification all requirements marked "MUST" have been accomplished, followed by all "SHOULD" and most of the "NICE" requirements. The requirements not fulfilled were considered too time consuming or not in focus of this project. So from a requirement perspective the result of this thesis is better than expected. From an application and user perspective there are some points that might reflect a more correct result. The task of creating, displaying and interact with the visualization is accomplished but with great strain on the hardware. Due to memory leaks from objects not being deleted completely between file loads there is a threat of crash and a decrease in frame rate. In the application a hotfix has been applied to at least prolong the time to potential crash. The fix consists of clearing the viewport explicitly when clicking the load button and again right after the file is selected; this apparently gives the application enough time to remove more of the old objects. Since this project, a new version of .NET has been released and since it contains a completely new garbage collection it might address this problem. A reason for this problem being so unique and almost unheard of is the nature of the WPF 3D objects. The models created with the Ranger cameras can easily contain 4-5 million polygons (all graphic objects are described in doubles) and 3-4 textures. When reading standard tutorials and even more advanced 3d applications there are seldom examples over 10000 polygons, so the memory usage is many times larger in this case. Adding the bitmap operations, any potential memory leak is much more obvious in this project. However, the scenario of multiple file loads excessing 20-40 different files is outweighted by what the application can accomplish. The future of new garbage collections, better 3D-object representation and faster hardware is also something to take into account, but at current time it is clear that using WPF 3D to handle this amount of data is pushing the edge to the limit. It is also clear that the format delivered by the cameras at SICK is quite easy to interpret and there is no need for advanced triangulation methods. 33.

(45) 34. Results. Figure 5.1: Visualization result from the new application(left) and the old visualization(right).. 5.1. The new visualizer. The application, Sick3DViewer, is a Windows based tool for displaying camera data from most configurations and series produced by SICK. Compared to the existing solutions the new visualizer is more of a universal solution. It does not require special settings or simulated cameras to display the data.. Figure 5.2: The application user interface..

(46) 5.2 Performance. 5.2. 35. Performance. The data returned from the cameras come in all variations and sizes. A scan of a small object can be of very high resolution resulting in a large data set and in the same way can a large object be scanned in low resolution. In the requirement specification a performance requirement of a ten second loading time for a 500x1000 pixel IMG1 image2 is introduced. This requirement is met and the frame rate is more than acceptable. As stated in chapter 1.3 the platform for the application is a standard laptop used for example by sales persons, the application has been tested on several laptop systems with acceptable frame rates and load times. There is however a performance issue while loading several files in sequence. The larger the data set is the faster the memory is occupied and the application might terminate. Frame rate is not a problem even when rendering very large objects. The bottleneck in the visualization is the creation of the object, once created; the object can be used with little extra memory allocation. The functions which affect the performance are those who edits the different blocks of the object, for example change texture coordinates. The tools modifying the geometry of the model does not affect the frame rate since they are applied as transformation matrixes and do not modify the object itself.. 1 IVC3D 2 pixel. camera format, 16bit data. size refers to original height map..

(47)

(48) Chapter 6. Discussion The results from this thesis meet the requirements set and the resulting product is ready for usage. There are some issues that points to the disadvantage of the framework chosen and some advantages. Advantages: • Usability The WPF framework is very powerful in its graphical user interface structure. The XAML code specifying windows and controls can create very advanced yet user friendly interfaces. It is very extendable and new tools and controls can be implemented without great effort. • Portability The .NET environment provides good support for implementation of the user control in other third party .NET applications developed by SICK. The application is standalone and very easy to install and use. • Extensibility The class library is with the separate header readers designed to be very extensible. Since data is stored in more or less the same way in the files, only differs where, new components and functions can be implemented in a straight forward way.. Disadvantages: • Speed The loading times when dealing with large files can become quite long and without the possibility of multi threading graphical objects the visualizer can become slow and unresponsive. If the possibility of using float precision 37.

(49) 38. Discussion instead of double precision for graphical operations should arise it would cut the amount of data considerably and thus the time it would take to shuffle data into the different arrays and lists. • Memory The application has problems returning all the memory allocated from previous object. This might be resolved in .NET 4.0 since it has a new garbage collection. This problem can result in a crash after several consecutive file loadings. One thing that could be the root of this problem is the WPF references, as long as an object is referred to in some way (sliders, buttons, textures) it will not be flagged for the garbage collector to remove it. In the case of ranger objects, which also are the largest form of data, an IDisposable pattern has been implemented but did not show any large improvements..

(50) Chapter 7. Concluding remarks and future work Since the limitations in WPF and the structure of its 3D formats affect the performance in such a way, it is clear that WPF 3D never was constructed to manage this kind of content. WPF is still being developed and new releases might address the problems and limitations previously mentioned. In the latest version, 4.0, the support for new pixel shaders is greatly enhanced which could be an alternative way of getting the result. Parallel to WPF, Microsoft develops another platform connected to .NET; XNA. This framework is especially built for game development for windows and Xbox360 and includes more efficient way of handling and rendering polygons and 3D content. Several of the limitations in WPF do not exist in XNA, for example graphical operations in XNA are done with float precision instead of double precision. The method of extracting data and constructing the 3D objects are however relevant to most implementations of a visualizer and could serve as a guide. Some further development of functionality, tools and future work would include: • A more set standard of the different file formats • Upgrade to .NET 4.0 platform to analyze performance enhancement • Review other graphical platforms for faster loading times • Hit testing of visual result for individual polygon values • Export 3D model capabilities to different formats • Add functionality to merge several files into one model(code stub for this is implemented) • Add animation support and more polygon precise modifying tools to enhance interactivity • Extend WPF code to Silver Light for online version of the visualizer 39.

(51)

(52) Bibliography [1] MacDonald, Matthew Pro WPF in C# 2008 Windows Presentation Foundation with .NET 3.5. Apress, 2 edition, 2008. ISBN 978-1-59059-955-6. [2] Troelsen, Andrew Pro C# 2008 and the .NET 3.5 Platform. Apress, 4 edition, 2007. ISBN 978-1-59059-884-9. [3] Lehenbauer, Daniel Rotating the Camera with http://viewport3d.com/trackball.htm. 2004. Relations.. the. Mouse.. URL:. [4] Hodnick, Mike Windows Presentation Foundation (WPF) 3D Tutorial. URL: http://kindohm.com/technical/WPF3DTutorial.htm. 2007.. 41.

(53)

(54) Appendix A. System Requirement Specification A.1. Introduction. This is the software requirement specification for a master thesis, involving visualization and software development at SICK IVP, Linköping. The requirements in this document are as followed: Req. no X changed in version Requirement no X information Priority. A.1.1. Purpose and goal. The goal of the master thesis is to create a class library and an application that will be able to process all image formats from SICK IVPs cameras and present them in a user friendly way. The purpose is to make it easier to demonstrate the capability and results from the different cameras, the result of project can also make it easier in a quick support situation when analyzing a saved image (i.e. grab setup settings shown in the application) instead of analyzing a whole program step by step.. A.1.2. Product usage. The application will be deployed with an installer and will require at least a standard laptop. The class library will be a "stand-alone" package that shall be able to return the visualization.. A.1.3. Product background. There are many technical advanced cameras at SICK IVP, and they all have their own formats and data. Today when a result from a camera shall be displayed for a customer or at an exhibition you would need a station with the right software and the right settings to see the result. By creating an application and a library 43.

(55) 44. System Requirement Specification. that will be able to load every format, and displaying the information, it will make it easier for demonstrations. Also, the application could make easier customer support errands faster by showing the camera technicians the exact grab settings active when the file was saved, and thereby speeding up the troubleshooting.. A.2. General system idea. Figure A.1: The complete system.. A.2.1. Coarse description of the product. The product shall be able to load different camera formats and visualize the data. This is realized by loading the file and with the developed class library presenting the result in the application.. A.2.2. Product components. The product will consist of a class library and an application. The class library is designed for loading data and creates the visualization. The application will utilize the class library in a user friendly way.. A.2.3. Dependencies. The class library will be "stand-alone" and shall be able to integrate with other applications. The product will be developed on, and will therefore need, .Net 3.5..

(56) A.3 The class library. A.2.4. 45. Design philosophy. The product shall be developed in WPF to comply with SICK IVPs computer environment.. A.2.5. General requirements on the complete system. Req. no 1. A.3. 0.1. Create a class library which reads a file and returns a visualization. MUST. The class library. The class library shall be able to process image data and return a visualization object.. Figure A.2: Overview of the class library.. A.3.1. Initial description of the class library. The object returned by the class library is a complete visualization that will offer navigation support.. A.3.2. User interface. Req. no 2. A.3.3. 0.1. The user shall be able to rotate/zoom/pan the visualization. MUST. Design requirements. Req. no 3. 0.1. Req. no 4. 0.1. The class library shall be compatible with Windows XP The class library shall be developed in WPF. MUST MUST.

(57) 46. A.3.4. System Requirement Specification. Functional requirements for the class library. Req. no 5. 0.1. Req. no 6. 0.1. Req. no 7. 0.1. Req. no 8. 0.1. Req. no 9. 0.1. Req. no 10. 0.1. Req. no 11. 0.1. A.4. The class library shall be able to read from IMG The class library shall be able to read from XML The class library shall be able to read from BMP The class library shall be able to read from JPEG The class library shall be able to read from RAM The class library shall be able to visualize 3D information with intensity mapped The class shall be able to visualize 3D information with scatter mapped. MUST MUST MUST MUST MUST SHOULD SHOULD. The application. Figure A.3: The application.. A.4.1. Initial description of the application. Req. no 12. The application shall be compatible with Win- MUST dows XP The application shall use the class library to load different files and provide the user with visual information and settings.. A.4.2. 0.1. Design requirements. Req. no 13. 0.1. The application shall be developed in WPF. MUST.

(58) A.5 Performance requirements. A.4.3. Functional requirements for system 2. Req. no 14. 0.1. Req. no 15 Req. no 16. 0.1 0.2. Req. no 17. 0.1. Req. no 18. 0.1. A.4.4. The application shall be able to save the current view as a 2D screenshot (i.e. jpeg) The application shall have lighting The application shall show camera settings contained in the saved file The application shall have the ability to modify the lighting The application shall have an installer. MUST MUST MUST SHOULD SHOULD. User interface. Req. no 19. A.5. 47. 0.1. The application shall be tested from a user friendly perspective. SHOULD. Performance requirements. Req. no 20 0.1 Load time to visual result, max 10sec. MUST Req. no 21 0.1 Frame rate of visualization, min 5 fps. MUST For the product to be useful the following requirements must be met while processing an IMG-file 500x1000 pixels.. A.6. Further requirements. Req. no 22 Req. no 23 Req. no 24. 0.1 0.1 0.1. Req. Req. Req. Req.. 25 26 27 28. 0.1 0.5 0.5 0.5. Req. no 29 Req. no 30 Req. no 31. 0.5 0.5 0.5. A.7. no no no no. Develop for unit testing Apply texture Load and process HDR-format (ranger camera) Export visualization to other formats (PDF) Combine Scatter and Intensity data Combine Intensity and Height data Develop the class Library as a Windows Forms compatible user control Appending models Add support for sheared models Add animation support (spin). NICE NICE NICE NICE NICE NICE NICE NICE NICE NICE. Documentation. Document Technical documentation User manual (simple). Languange English English. Purpose Description of the development Introduction of the system. Format PDF PDF.

(59) Appendix B. System Design Specification B.1 B.1.1. Introduction Purpose of this document. The Software Design Specification (SDS) document will describe the framework used to build the class library and viewer application. It will also describe a list of requirements used to test the final product.. B.1.2. Scope of the development project. The 2D/3D Viewer is an application that will be used to display any of the image formats from SICK IVPs camera rigs. The viewer will use the class library to accomplish a navigational visualization of the data. The library will also allow integration with other applications to provide visualization and navigation support.. B.1.3. Definitions. • GUI - Graphical User Interface • PC - Personal Computer (desktop or laptop) • WPF - Windows Presentation Foundation. B.1.4. References. SRS - Software Requirement Specification 0.5. B.1.5. Overview of document. The System architecture description section will provide an overview of the complete system and its most important parts, as well as the interaction between the application and the user. 48.

No results found