Konstruktion av generisk testmiljö för inbyggda system

(1)

Construction of generic test environment for embedded systems

MAGNUS DORMVIK GUSTAV LEESIK

Master of Science Thesis

(2)

(3)

Construction of generic test environment for

embedded systems

Magnus Dormvik Gustav Leesik

Master of Science Thesis MMK 2013:85 MDA 423

(4)

(5)

Examensarbete MMK 2013:85 MDA 423

Konstruktion av generisk testmiljö för inbyggda system

Magnus Erik Leonard Dormvik Gustav Felix Leesik

Godkänt

2013-11-11

Examinator

Mats Hanson

Handledare

Sagar Behere

Uppdragsgivare

Data Respons AB

Kontaktperson

Stefan Stendahl

Sammanfattning

Data Respons AB utvecklar skräddarsydda inbyggda system. Systemen består av allt mellan modifierade datorer baserade på x86-arkitekturen till enklare mikrokontroller. Syftet med detta examensarbete var att utvärdera och utveckla metoder för att testa dessa system. Målet var att identifiera testmetoder/lösningar för att göra test av inbyggda system snabbare, mer träffsäkra och med bättre spårbarhet.

Arbetet omfattade analys av befintliga hårdvarutester och testdokumentation, design av kompleterande testmetoder samt implementation av dessa.

Analysarbetet innefattade hur Data Respons arbetar med tester av inbyggda system, från utveckling och montering till felanalys. Intervjuer gjordes, stöddokumentation och rutiner analyserades. Identifierade problem omarbetades till krav som användes för att utvärdera befintliga produkter samt behovet av skräddarsydda lösningar.

Ett testexekveringsramverk utvecklades i C++ Qt för att automatisera hårdvarutester genom att minimera manuella rutiner kring utförandet av test och dokumentation av test.

En utvärdering av testmetoden Boundary Scan utfördes. Testmetoden möjliggör hårdvarutester av kretskort med hjälp av kretskortskomponenternas inbyggda testfunktioner. Metoden möjliggör testning av delar av kretskortet som normalt inte nås med någon annan testmetod.

Boundary Scan gör det möjligt för utvecklare att isolera fel i hårdvaran utan att blanda in dess mjukvara.

En utvärdering av testverktyget elektronisk last utfördes. En sådan kan emulera olika typer av elektriska laster, vilket används för att testa kretsar som levererar ström. Många fel i elektriska system kan härledas till en felande strömkrets. Att testa dessa är avgörande för att uppnå hög tillförlitlighet.

(6)

(7)

Master of Science Thesis MMK 2013:85 MDA 423

Construction of generic test environment for embedded systems

Magnus Erik Leonard Dormvik Gustav Felix Leesik

Approved

2013-11-11

Examiner

Mats Hanson

Supervisor

Sagar Behere

Commissioner

Data Respons AB

Contact person

Stefan Stendahl

Abstract

Data Respons AB develop custom embedded systems, ranging from modified PCs to micro controllers used to do specific tasks. The purpose of this master thesis is to analyse how to make tests of embedded systems faster, more accurate, easier to reproduce and quality assured. The purpose is also to implement a solution to these needs as much and well as possible.

The work covered how Data Respons work with tests of embedded systems, from development and assembly to fault analysis. It starts with a general analysis of improvements regarding tests.

Interviews were held, support documentation and routines regarding test were examined.

Identified problems and needs for improvements were reworked to requirements. Requirements were mapped to existing products and the need for custom made test software.

The result was a solution package containing:

A test execution framework. This test software is used to automate tests and reduce manual procedures and interactions with the system under test. The solution also dramatically reduces the work involved with logging the tests.

An implementation of the test method Boundary scan. This tests method enables hardware tests on a physical layer of a printed circuit board (PCB) and enables testing of parts of the PCB unreachable with any other test method. Boundary scan also enables developers to identify if a fault is caused by hardware or software.

An evaluation of electronic load. An electronic load is an equipment that can emulate the load of a circuit that produces electric power. Testing power circuits with the real load can be dangerous, noisy or inconvenient. Many failed systems have a root cause in a failed power circuit. Testing these are crucial for a high reliability.

(8)

(9)

FOREWORD

This Master thesis was done for Machine Design at KTH (Royal Institute of Technology), Sweden. It was done at Data Respons AB, mostly at the Kista office, Sweden, during the period June 2011 to February 2012.

We would like to thank:

Jeanette Fridberg, manager of the Solutions department when this master thesis began, for valuable everyday guidance.

Stefan Ohlson, our technical supervisor at Data Respons, for technical guidance on embedded systems.

Sagar Behere, our supervisor at KTH, for tips on programming and organisational issues.

Finally we would like to thank the whole Data Respons office in Kista for an inspiring environment, a challenging task and a good time.

Gustav Felix Leesik Magnus Erik Leonard Dormvik Kista March 2012

(10)

(11)

NOMENCLATURE

Abbreviations

AC Alternating Current ISO

AD Analog to Digital ISP In System Programming

ARENA JTAG Joint Test Access Group

ARM CPU architecture KTH

BASH a UNIX shell LGA Land Gate Array

BGA Ball Gate Array MCU Micro Controller Unit

BIST Burn In Self Test MOSFET type of power transistor

BOM Bill Of Material MTBF Mean Time Between Failure

BS Boundary Scan NI National Instruments

BSDL OEM

CLI Command Line Interface OS Operating System

CPU Central Processing Unit PCB Printed Circuit Board

DA Digital to Analog PGA Pin Gate Array

DC Direct Current PWM Pulse Width Modulation

DFT Design For Test PXE

DOA Dead On Arrival QMS Quality Management System

DUT Device Under Test RAM Random Access Memory

EMC Electro Magnetic Compliance RMA

EMI Electro Magnetic Interference SPI Serial Peripheral Interface

ESD Electro Static Discharge SVN

FMEA SW

FMFI Fucking Magic Fixed Itself TAP Test Access port

FTA Fault Tree Analysis TCK Test ClocK pin

FW FirmWare TDI Test Data Input pin

GUI Graphical User Interface TDO Test Data Output pin

HDD Hard Disk Drive TEX Test Execution framework

HW TMS Test Mode Select pin

IC Integrated Circuit TRST Test ReSeT pin

ICT In Circuit Testing USB Universal Serial Bus

IEEE PLM Product Lifecycle Management

International Organisation for Standardisation

Software to handle BOMs and products

Kungliga Tekniska Högskolan (Royal Institute of Technology)

Boundary Scan Description

Language Original Equipment

Manufacturer

Preboot eXecution Environment

Return to Manufacturer Authorisation

SubVersioN, version handling software

Failuer Mode and Efftect

Analysis SoftWare

HardWare

Institute of Electrical and Electronics Engineers

(12)

Terms

Board bring up The process of getting software to run on new hardware, i.e. making sure a printed circuit board behaves as specified

Operations The department at Data Respons that assembles and repairs products, builds prototypes and handles logistics

System integrations The department where system integration is done Solutions The in-house development department

Services The consultant part of Data Respons

Loop-back plug A device plug that connects sending pin to receiving pin

(13)

1 INTRODUCTION

This chapter gives a short background to Data Respons AB, the initially defined problem and the result of the solution. The chapter also describes the work of the master thesis in short and what methods were used to achieve the goals. A short explanation of the disposition is also given.

1.1 Background

Data Respons wants to improve the quality of their products by creating better tests and methods for testing. All products that are shipped to a customer must be tested. The quality requirements are very high. This final hardware test and testing of the product's functions is done in-house and is often time consuming. The products that need to be tested are often very different from each other. This makes it hard to re-use testing equipment and procedures, thus the wheel is being re- invented every time a new product is tested for the first time. This makes development and assembly expensive. Also, since many tests are done more or less manually, human mistakes occur.

1.1.1 Data Respons

Data Respons is a Norwegian company that was founded in 1986. Data Respons develop and deliver customised embedded solutions to leading OEM companies. Data Respons has offices in Europe and Asia. The different offices have different areas of competence. This master thesis is done at the offices in Sweden, with main focus on Kista, Stockholm. The Kista office has both development and final assembly.

1.1.2 Problem

The developers lack routines for verification and documentation. The testing equipment that developers use now is either typically a very generic equipment, such as a oscilloscope or a very specific, such as a Ethernet analyser.

This can be condensed to three distinct problems

• Test takes too much time. Both in the development and assembly phase, mainly because of high human interaction, lack of support-documents, templates regarding test procedures and documentation.

• There are doubts about the relevance, quality and repeatability of many tests.

• It is difficult to find results and statistics from old tests and detailed information about how the tests were carried out.

1.2 General goals

• Analyse the different needs for improvements regarding testing at the offices in Sweden.

• Speeding up repetitive tests of multiple units.

(17)

• Create methods and documentation to make tests traceable and repeatable.

• Automate and minimise human interaction in test procedures.

• Improve test methods.

1.3 Purpose

The purpose of this master thesis is to analyse how to make hardware tests of embedded systems faster, more accurate, easier to reproduce and quality assured. The purpose is also to implement a solution to these needs. This will result in better documentation of tests, better procedures for testing in all parts of the product life cycle, better tools for product quality analysis, which will result in higher product quality.

1.4 Disposition

When looking at the disposition of the report, it can be noted that the problem definition does not appear until chapter 5. It reflects the high level of work done on identifying the task. The definition could not be done earlier in the process on a level where it could be used to set any specific requirements to the solution.

1.5 References

Since a lot of material was gathered during the analysis phase, many of the conclusions in the report refer to the conditions at Data Respons and are not general assumptions from other research. This is done intentionally to illuminate how the conclusions from this master thesis can be analysed and compared to other studies, best practices and literature from the area.

1.6 Priority

Priorities were made for all tasks. The two main products of this master thesis, the test software and hardware were chopped up in smaller tasks. All of these were prioritised on a tree grade scale. The test software had higher priority which resulted in the hardware not being finalized.

See APPENDIX 4: PRIORITY PLAN.

1.7 Method

1.7.1 The V-model

The development was done using the V-model. The V-model is a well known development model, initially intended for software development. It says among other things that all the requirements need to be testable. See Illustration 1: V-model.

(18)

A modified V-model was used to fit the project. The model was expanded to include general goals, a target area evaluation and three individual tracks for test improvement. See Illustration 2: Specific V-model.

1.7.2 The modified V-model

Illustration 1: V-model

Illustration 2: Specific V-model

General improvement evaluation

System requirements testing

Delivered subsystem

C++ code and Product testing Time Planning

priorities Analysis and

Interviews

Electronic load Boundary Scan Test execution framework

Electronic load evaluation Boundary Scan evaluation Test execution framework

testing

(19)

1.7.3 Improvement area analysis

When the work started at Data Respons, an analysis of what type of test related improvements, was needed. This was done by making interviews at Data Respons and analysing documentation regarding testing and quality assurance available at Data Respons.

1.7.4 Decision on approach

Two main tracks of improvement were identified.

Top down approach covered the routines, support documentation and communication.

Bottom up approach covered known problems regarding practical issues.

An advice for decision was created where the two approaches were analysed. The advice for decision was used to provide Data Respons with the information needed to decide which approach to choose.

The approach chosen by Data Respons solution manager was the bottom up approach. This was also what was recommended by the authors of this master thesis.

1.7.5 Research on subject

A research on embedded systems testing was done to get the information needed to properly identify the problems and find suitable solutions. The in depth studies Boundary-scan and Electronic load were also covered here.

1.7.5.1 Boundary scan in depth study

The boundary-scan testing was first mentioned during the interviews with the developers at the Örebro office. It was presented as an expensive method that was complex to use but after a short pre-study on the subject and tools available today, it appeared as highly interesting. It was identified as a possible solution to one of the problems regarding board-bring-up and also a suitable target for an in depth study, see “APPENDIX 3: LIST OF TESTING ISSUES“.

1.7.5.2 Electronic load in depth study

The procedure of testing power circuits was during the interviews identified as a time consuming task, see chapter 11.1 “Sammanfattning av intervjuer av utvecklare och montörer på Data Respons“. As of now, power circuits are either not tested at all, tested using the real actuator or by using a power resistor. Since very few different devices are specified to draw the same current at the same voltage, the power resistors are hard to re-use. It also involves calculations, and construction of a chassis to hold the power resistors. A digital electronic load, which is adjustable in both voltage and current, could be a solution to this problem. It will be investigated how they work, how to build one and recommendations on existing products.

1.7.6 Analysis of the refined task

A second analysis phase was initiated after the decision to perform the work bottom up. This analysis involved discussions with developers and the engineers performing functional tests at the end of the assembly line. A problem definition was made.

(20)

1.7.7 Solution package design

A solutions matrix ( APPENDIX 5: SOLUTIONS MATRIX) was created to identify suitable solutions with good problem solving coverage. A system specification was made and flowcharts were made. A early stage user manual was created to get a clear view of the solution. The three identified solutions were an automated test execution framework (TEX), a Boundary-scan solution and an Electronic load solution. The concept was tested by running the software (TEX) on an embedded system.

The solution package consist of: TEX, boundary-scan and electronic load.

1.7.8 Test execution framework Design

An evaluation was made to decide which programming language should be used considering system requirements and cross platform functionality. C++ with the Qt library was chosen because C++ is a well proven language and is familiar to the authors of this thesis. The Qt library provides a convenient way to cross compile to both Microsoft Windows systems and Linux systems, using the same source code.

1.7.9 Boundary-scan testing

The boundary-scan solution was tested by installing two products and testing the functionality on a embedded system developed by Data Respons.

1.7.10 Electronic load testing

The electronic load solution was not tested explicitly due to lack of time. Earlier experiences with electronic loads show that they are easy to use and give repeatable results

1.7.11 Test execution framework evaluation

The test execution framework was evaluated by presenting the solution to the developers at Data Respons. The evaluation was also done continuously with design meetings.

1.7.12 Boundary-Scan evaluation

The Boundary-scan evaluation was done as product evaluation of available boundary-scan products. The products was tested by creating boundary-scan tests on an embedded system developed by Data Respons.

1.7.13 Electronic load evaluation

An evaluation of an electronic load could not be performed. Since electronic loads are fairly expensive, it was thought likely that one would not be purchased during this master thesis. The plan was instead to build a very simple electronic load, a prototype, as a proof of concept. The prototype was not built due to lack of time.

(21)

1.7.14 General improvements evaluation

The general improvements were evaluated by checking the requirements from the general goals.

Because of time limits, the final solution was never practically implemented or evaluated in a case study.

(22)

2 PROBLEM ANALYSIS

This chapter describes how Data Respons work with tests. The analysis consists of interviews with Data Respons personel and file server documentation analysis of routines and wikis.

When the work started at Data Respons, it was not clear what needed to be done to best solve the problems regarding tests. At first, the task given was to build a “test box” that somehow made tests of embedded systems easier. Due to the unclear purpose of this box, an analysis of the general needs regarding tests for all the offices in Sweden was done. This analysis included both the need for better documentation and the need for new or improved tools to execute the tests.

2.1 Business analysis

2.1.1 Interviews

The analysis consisted of a series of interviews. The interviews were held at Kista, Lund, Västerås and Örebro, covering 18 persons, mainly developers and integration engineers but also project managers and sales personnel. The interviews contained, among other things, questions regarding the needs for improvement on a both technical and administrative level, how tests and test documentation is handled and stored today. This provided a good overview and played a central role in deciding what was needed and what could be done. The analysis generated a report, Advice for decision.

2.1.2 Support document analysis

The Support document system is relevant because the documentation of products and their tests are important to the quality of the products. Data Respons has a Quality Management System (QMS). This QMS complies with ISO 9001. The QMS has three levels:

• Level 1: Strategic

Management process, Organization charts, Quality management process, Description of the company and its goals in short.

• Level 2: All procedures

Flow charts for specific procedures like Order handling, Data backup, Document control, Sales procedure, etc.

• Level 3: Specific instructions, forms, schemes, etc.

Coding standard for C and C++, ESD (Electro Static Discharge) policy, Calibration rules, Hardware and Software development instructions (settings in development software, symbols to use etc.), General Development instructions (time plan, milestones, work-flow, etc.)

2.2 Summary interviews

This is a summary, office by office, of about 18 hours of interviews. Observations regarding the

(23)

thesis. This information and knowledge is here condensed to a summary of what is characteristic for each office and a description of how the business is run.

Although the employees are tied to their home offices, the projects are often shared between offices. It is quite common for developers and project managers to visit other offices and work there for short periods.

2.2.1 Stockholm Kista

At the Kista office, two HW developers, the Operations manager, one Integration engineer and one salesman were interviewed.

2.2.1.1 Office description

The Kista office has 30 employees, of which three are developers, eight are System Integration engineers or technicians and the rest belong to Administration and Sales.

Solutions department:

• Does in-house development projects. The projects can be of very different sizes. Small projects can be minor upgrades of existing products or previous projects. Large projects can be development of a completely new product about which the customer only has an idea but no competence to develop on their own.

• The most common type of project is custom built embedded systems.

• The environmental requirements are often high. The products need to operate in

environments that are for example damp, rapidly changing in temperature, vibrating and dusty.

Operations department:

Operations has several sub-departments, among them System Integration and RMA.

RMA department:

• RMA Handles products that are returned to Data Respons for upgrades or repairs, both off-the-shelf products and OEM products.

System Integration department:

• Assembles prototypes and pre-series developed by Solutions.

• Assembles final series of products that are complicated to build, and are not produced in big enough volumes to be profitable to assemble abroad.

• Repairs many of the products that Data Respons sells.

• Performs functional tests on products that are assembled abroad, before they are shipped to customers.

2.2.1.2 Summary of what was said during the interviews

• A general opinion was that there is a lack of guide documents on how to run projects and how to create and perform tests.

• Feedback to developers about product problems in the field and during assembly is poor.

(24)

• Statistics over common faults and problems require some work to extract since almost all information from tests are in spreadsheets.

• Things are mostly working fairly well, but that is because most projects only involves a handful of people at a time.

• Many things are done ad-hoc and not always well documented which means that if the engineer who knows all about a specific product leaves the company, a lot of important information is lost.

• There is a need for better test specifications.

• Less human interaction in tests would be an improvement, to avoid careless mistakes.

• Some system for reusing old tests would speed up the process of creating new tests.

• Better communication between developers and System Integration is needed.

• Administrative improvement probably gives most benefits for Solutions.

• Practical improvement probably gives most benefits for System integration.

• Some products are unnecessarily complicated to assemble and disassemble.

• A company-wide test strategy would do a lot to increase quality and efficiency.

2.2.1.3 Conclusions from the office

The Solutions department would benefit from a way of testing products during development that automatically logged the tests. They would also benefit from some sort of modular testing library so they wouldn't have to “re-invent the wheel” for every project.

Operations would benefit from a way of doing repetitive tests faster and more accurately. They could also use a tool for systematic troubleshooting. It would also save time if Operations could re-use the tests already designed by Solutions.

2.2.2 Västerås

Two developers were interviewed. One SW and one HW developer.

Västerås has about 15 employees. The office in Västerås only has Services which means that all consultants work at the customers' sites.

• About half of the times, there are good requirements to follow when testing.

• Testing is one of the last things in a project, so there is seldom enough time to perform proper testing when the deadline is drawing near.

• There is almost always a lack of guide documents regarding how projects and specifically tests are supposed to be done. This is probably because creating those documents is an

(25)

• Clear and testable system requirements would make administrative work easier concerning test.

• It is important to get help from people who know the product inside out. For example, a nurse knows the medical equipment very well and should be involved in prototype testing a new piece of medical hardware.

Since Västerås has no in-house developers they will probably not be the target for this master thesis.

2.2.3 Örebro

At the Örebro office, four HW developers and one office manager was interviewed.

There are about 15 employees in Örebro, six people were interviewed. The Örebro office has almost everyone of their developers working in-house. Many of the developers in Örebro came from the same company before they were hired by Data Respons. This is positive because they work well together and know each others' strengths and weaknesses. It can be negative because they are used to work in a certain way and that can make synchronisation with the rest of Data Respons difficult. Örebro has one developer whose main competence is in testing.

• It is easier to prioritize testing if the customer has good technical knowledge.

• The customers are more willing to pay for testing if they understand why it is important.

• If the product needs a safety classification, well specified and well documented tests are an absolute requirement.

• If Data Respons has total responsibility it is more common that only the most essential part of testing is done. This is because when the project hours are all used, testing is still left to be done. Therefore, testing is sometimes done for “free”.

• Projects are often so different from each other that it is hard to have a template of how projects should be managed.

• If it is known from start that it will be a high volume product, more effort is put into finding cheap components, mounting sequence, DFT (Design For Test), etc.

• It is hard to find relevant information to re-use when searching in old project folders. It is probably more time consuming to find and re-use old templates and pieces of code than to create it from scratch when starting a new project.

• When starting a new project, it should be decided from start what processes and document templates should be used, to make sure all file names and documents use the same style and nomenclature.

• Construction verification and production testing takes a lot of time. Especially if an error is discovered that requires changes to the hardware construction.

(26)

• One of the HW developers estimates that he puts about 20% of his time into testing.

• EMC problems can be very difficult to resolve. It is important to be thorough when constructing the PCB. It pays off later in the project.

• Early troubleshooting is difficult. It is hard to know if the problem is in the HW or SW.

From experience it is about 50/50 HW and SW errors.

• Intermittent errors are a problem, especially if the error is difficult to recreate.

• A good way of testing ports is using loopback plugs. Making the device communicate with itself tests both ingoing and outgoing ports.

• Board bring-up is often so different between boards and developers that there is no general way to do it.

• When testing new HW, it is important to use small pieces of code to make sure the SW is not the problem.

• Tests used by developers during construction are sometimes passed on to production with little or no modification. This is seldom very good, since these tests are not made to be run fast and in series or parallel. Logging from these tests is often not suitable for production logs.

• It might be good if someone always was responsible for test.

• It would probably do more good if the administrative issues were resolved, than to implement some type of new testing hardware.

• A test manager for Data Respons Sweden would perhaps work well. That person could pretty soon build a library of HW and SW tests and would be responsible for seeing to that testing is properly handled in all projects.

The Örebro office is similar in many ways to the Kista office. Except that they don't have production or final assembly.

If some kind of hardware device is developed, they will probably not be the primary user. If an administrative solution is the result of this master thesis then Örebro will most likely benefit from it.

2.2.4 Lund

At the office in Lund, five developers were interviewed.

There are about 25 employees in Lund. The company Lundinova was bought by Data Respons in 2008. They kept the name Lundinova because it is well known in that region. Lund has most of its developers working in-house. They often do high-tech compact solutions and have made a number of accessories for a large mobile phone manufacturer.

(27)

• A customer comes with an idea

• A rough sketch for a solution is done quickly

• If the customer likes what they see, the project moves on

• A brainstorming with everyone that is available

• Calm thing down and get realistic solutions written down

• Choose what people should be in the project, depending on expertise and availability

• Scrum is a project form that involves 5 minute daily meetings (scrums) where everyone answers the questions: What have I done since the last scrum? What will I do until the next scrum? What obstacles are in my way? These meetings are good because they keep the customer's ideas and developers' ideas from diverting.

• A lot of work is done ad-hoc.

• The size of projects varies a lot. From making a complete product out of a vague idea to just designing a PCB from a wiring diagram without any functional responsibility.

• Testing is often thoroughly discussed at the start of projects but is easily down-prioritized as the project moves along.

• Tests are often chosen from experience if the customer doesn't have specific requirements.

• There is access to a simple EMC/EMI chamber where the developers can run the

equipment by themselves. The chamber is not certified but it speeds up development a lot to be able to run EMC/EMI tests during development to check that everything is within reasonable limits.

• EMC/EMI certification of products is done by a certified test-house.

• Some companies have specific test templates, that makes it easy to construct tests.

• It is rather uncommon that the customer demands test logs and test documentation from the development process.

• Test results are checked against the requirements in a document that specifies all values from different measurement points. Often referring to component data sheets.

• It is not trivial what tests to pass on from development to production.

• Mentorship is unofficially implemented, a junior developer works side by side with a senior developer for a while.

• Some test templates are re-used, but that is highly individual from developer to developer.

• Having HW and SW development at different sites complicates the process. There is a risk that it becomes a “us and them” feeling when problems start to surface in the projects.

(28)

• Having tests done abroad may be cheaper, but can complicate thing because of language barriers and the fact that it takes time to get there if there is an issue that can't be resolved over telephone or video chat.

The Lund office is quite different from Kista and Örebro. When the interviews were made they were not using the same documentation system as Kista and Örebro. Lundinova has a very different product portfolio containing more compact solutions often created from scratch whereas Kista and Örebro makes more modular solutions that are more PC-like. Lund has many skilled developers and they are very creative in their solutions.

As with Örebro, if some kind of hardware device is developed as a result from this master thesis, they will probably not be the primary user. If an administrative solution is the result of this master thesis then Lundinova will most likely benefit from it.

2.3 Company-wide areas of improvements

• Testing needs to get a higher priority when planning the working hours for the project.

• A testing group within the company would quickly accumulate knowledge and help project groups deciding what tests are relevant and necessary.

• There is a need to better port tests from development to production.

• Documentation of testing needs improvements.

2.4 Test's role in a greater perspective

All too often, time runs out and testing, the last thing done in a project, suffers. This is a problem for the entire industry, not just Data Respons. Competition is hard and everyone wants (and needs) to push prices down in order to win a request for quotation. This phenomenon is even more obvious in consumer products. The first version of a new product almost always contains a large number of bugs because it is cheaper and easier to let the customers find bugs and it is so important to be amongst the first on the market with a new model or product. This is a necessary, but difficult and dangerous balancing act. Too incomplete and untested products will cause bad- will, but testing too thoroughly will make the product obsolete by the time it reaches the market.

(29)

3 TARGET AREA AND APPROACH

This chapter explains the general approach on the problem, what angle the problem was engaged from. It also defines what is possible to achieve within the frames of a master thesis.

In order to reach the best possible result, the identified improvements needed to be delimited. A target area needed to be identified to make the solution form a handleable package that could be processed. The pre-study showed that there were two main areas of improvement, administrative improvements and technical solutions to problems regarding tests. Therefore, two different approaches were used. The two approaches were created to get a clearer view on what the actual work would be. This was necessary in order to make a qualitative decision on which target was best suited for both Data Respons and the students. The advice for decision was also a requirement from Solutions manager Jeanette Fridberg in order to continue the work.

Approaches in short:

• The top-down approach focus on routines, methods and documentation which later may result in the need for a technical solution to complement the administrative solution.

• The bottom-up approach focus on a generic technical solution to solve known problems regarding how tests are done today. Support documents and routines will be created mainly to complement the use of the technical solution.

The definitions of problem, approach and improvements were identified in the pre-study by analysing the interview material, see “APPENDIX 1: Interviews, brief“.

3.1 Top-down approach

The top-down approach was defined as finding the cause to the “test related issues” (problems) at hand and solving the underlying problems and/or finding other larger problems that can be solved with less effort. It focuses on the routines for how things are done and what resources are at hand to solve the problems. The method allows to find and solve the right problem. Effort would be put into pinpointing the problems and giving advice on how to solve them, rather than actually solving them.

3.1.1 Identified issues

• There are uncertainties regarding what the most needed improvements are regarding system testing to improve product quality at Data Respons Sweden.

• There are uncertainties about how tests are administered and what routines exist concerning tests.

• A lot of time is consumed on recreating test-documents.

• The developers use their own templates for test documentation. This makes quality assurance difficult. Documents become hard to handle and interpret by anyone but the creator.

• Many of the developers and project managers don't know what templates are available.

• For smaller projects, the sales person in some cases becomes project manager. This often makes documentation and tests for off-the-shelf systems inadequate. This causes

(30)

problems if the person responsible is no longer available and there are no documents at hand.

• Many of the developers from Örebro and Lund claim that better documents and

guidelines regarding tests probably would improve more than getting a technical solution for the hands-on testing.

3.1.2 Approach

• Find out what templates and project models are available at the different offices.

• Analyse:

• Several projects and look at what templates have been used.

• What has gone wrong and if it could have been avoided using better tests and templates for tests.

• What the most cost efficient ways are to make sure similar errors are not repeated.

• Improve:

• Analysed material and make improvements, starting with the most important material.

3.1.3 Delimitations

• This method would not include quality-management such as routines for cross-checking and routines for how routines are created. It would concern only testing routines and documentation of test methods.

• Effort would be put into pinpointing the problems and giving advice on how to solve them, not necessarily solving them.

3.1.4 Main Goals

• Shortening the time spent on tests by:

• creating templates for board bring-up, logging, prototype test documentation, production test documentation.

• making sure the tests that are made are relevant. Looking at what goes wrong with existing products (RMA, test logs) gives a picture of what is prone to fail, thus what is important to test.

• developing or investing in new test equipment. The equipment will either be bought off-the-shelf or developed in-house.

• Ensuring the quality of tests by:

• providing instructions for how traceable and repeatable tests should be executed

(31)

• Making sure data from tests is stored appropriately and is easy to access by:

• analysing ways of implementing automatic documentation into new test equipment.

• analysing ways of making test logs searchable and easy to pull statistics from.

3.2 Bottom-up approach

The Bottom-up approach was defined as dealing with the problems directly. Not looking into what caused the problems in the first place or identifying other greater problems. In this case focus on developing a solution to known problems, limitations and minimising human interaction. Only support-documents and routines regarding the solution would be produced.

3.2.1 Identified issues

• Functional testing is highly manual.

• Functional testing is time-consuming.

• The method to carry out tests in production is very product specific.

• The way of logging tests in production are highly manual.

• No voltage, current or load tests are done in production.

• Monotonous tasks may cause human error.

• There are no methods to reuse tests between products.

• There are no methods to do the exact same test again, years later.

3.2.2 Approach

• Analyse what the Solutions and Operations departments in Stockholm needs regarding tests and automation.

• Analyse how tests can be done more easily.

• Develop a method for automated logging of voltage and current of external ports and power-feed during functional tests.

• Develop a dynamic software for test automation and logging.

3.2.3 Delimitations

• The end user of the solution will primarily be Operations and Solutions in Stockholm/Kista.

• Issues regarding test routines at the Operations and Solutions department will not be handled

• No deeper analysis of the backgrounds to the problems will be made.

(32)

3.2.4 Main goals

• Make it possible to quickly make physical layer tests

• Speeding up repetitive tests of multiple units

• Minimise errors caused by monotone tasks for the tester

• Make tests traceable and repeatable

• Automate documentation of tests

3.3 Advice for decision

To evaluate the approaches, an advice for decision was created, see “APPENDIX 2: ADVICE FOR DECISION“. The document contains a description of the approaches and an advice on which approach to take. The bottom-up solution was considered the best approach. The motivation was that Data Respons benefits directly from a automated and unified test-platform.

The solution can also be developed further to create automated installations of custom operating systems and other features that would ease the work for both Operations and Solutions personnel. It should also be noted that a light version of the top-down had already been done in the pre-study.

On the contrary, the need for working with the problem area covered by the top down approach was identified as highly necessary. Issues regarding support-documents and routines for both developers and project managers were found in almost all the interview material. The conclusion, however, was that this work should be done on a strategic level with respect to the the different customers' projects and ways of working at the different offices. The work should be done by someone who has long experience with managing development of embedded systems.

3.4 Decision

Based on the Advice for decision report , Jeanette Fridberg, Solution Manager in Stockholm, followed the advice for decision and decided that the bottom-up approach should be used for this mater thesis.

(33)

4 TESTING METHODS OF EMBEDDED SYSTEMS

This chapter contains the research regarding test methods available for embedded systems. The Boundary-scan and power loading methods are explained in detail (in-depth studies), the other methods are briefly explained. Test coverage areas are defined and methods are mapped to the different areas.

4.1 Testing methods

4.1.1 Black box testing

Black box testing is a test method where the functionality is verified by sending test patterns to the DUT (Device Under Test) and comparing the response to an expected response. How the DUT produces the response is not of interest. The main focus in black box testing is to verify how the DUT acts according to the expected behaviour. Black box testing is normally done during production testing.

4.1.2 White box testing

The white box testing method focuses on testing how the DUT is working internally. The method ranges from analogue and digital component tests to investigating buffer levels, and data flow in a program. White box testing is normally done during product development testing.

4.1.3 Examples of different tests

• In-circuit test (ICT)

• Stand-alone JTAG test

• Automated X-ray inspection (5DX/AXI)

• Automated optical inspection (AOI)

• Continuity or flying probe test

• (Board) functional test (BFT/FT)

• Burn-in test

• Environmental stress screening (ESS) test

• Highly Accelerated Life Test (HALT)

• Highly accelerated stress screening (HASS) test

• Ongoing reliability test (ORT)

• System test (ST)

• Final quality audit process (FQA) test Some of these are explained later in this chapter.

(34)

4.2 Areas of testing

There are several test-methods used when developing embedded systems. Which type of test to use, and why, depends on DUT usage and production volume. The method also depends on which part of the product life-cycle the product is in. The tests also cover different areas of the DUT in different ways. When studying the different test methods and the hardware of the systems available at Data Respons Kista, the components and logic of a DUT were divided into eight areas, see Illustration 3: Embedded systems test areas.

The Applications block represent testing the applications delivered with a system. The tests are used to verify that all configuration files, licenses and binaries required to run the applications are installed and correct.

Usually the application tests are used to test the lower level blocks as well. Testing an application that initiates a network connection over GPRS to the Internet includes testing both the application itself, part of the OS, the driver and the GPRS modem hardware functionality such as logic connections and signal quality. It is however very difficult to get a full test of the underlying blocks with this method.

The OS and driver block represents testing that the devices and resources are available to the operating system. The difference between application level and driver level verification is not always clear. When testing a serial interface at application level, the application Hyper-terminal can be used. The equivalent OS/driver level would be to access the device directly through the device OS buffer.

The Logic circuitry block represents testing the PCB IC´s physical connections between logical components on the PCB. The tests are used to verify the mounting and soldering of the logic components on the PCB. It also tests that all nets on the PCB are intact.

The Power electronics block represent testing the circuitry used to provide power to the PCB and to external devices from the PCB. Testing these circuits usually involve applying an external load to the system, or running the system at its maximum performance and analyzing the quality of the provided power. Terms such as ripple, level and heat are objectives to these tests. The tests are used to verify the mounting and soldering of the power circuit connections and heat sinks on the PCB.

The Signal quality block represents testing the quality of the different analogue and digital Illustration 3: Embedded systems test areas

OS, drivers Logic

circuitry

Power electronics

Signal Quality Mechanics

Robustness EMC

Applications

(35)

The EMC block represents analysing how sensitive the DUT is to electromagnetic interference and how strong electrical signals are propagated from the DUT. The tests are used to verify that the PCB design is correct according to the EMC requirements.

The Mechanics block represents testing water resistance, pressure, vibration, temperature and other mechanical entities. These tests are usually done during product development but some IP classified products require additional testing during production according to Data Respons' policy.

The Robustness block represents several robustness areas. The identified robustness areas at Operations at Data Respons Kista are mechanical, electrical and logical(computational). The robustness tests analyse how well or long the system operates during non ideal conditions.

Mechanical robustness tests cover physical quantities such as vibrations and heat. Electrical robustness tests cover EMI and EMC and electrical overload tests. Logical robustness tests includes applying logic load to the system such as network traffic and CPU operations.

4.3 Functional test

Functional test is a test method that runs the system at native speed. Tests are run on a test-OS or on the DUT's intended OS. This is a common test method in a production environment since the DUT usually have a functional OS in this product stage and multiple systems can be tested in parallel with no extra hardware required. Functional tests can be used to test parts of both logic circuitry, power electronics, signal quality and robustness at the logical and electrical level. The test method is preferably used to test OS, drivers, applications, and robustness. See Illustration 4:

Functional test areas.

The disadvantage is that functional tests usually does not completely test all the blocks. This creates a situation where the partially tested blocks often are left partially tested see,

“APPENDIX 1: Interviews, brief“. It is also difficult to pinpoint the blocks that actually failed during a failed functional test. If the developers involved with creating the PCB do not have access to the DUT OS and drivers, the cost of creating functional tests is fairly high.

Illustration 4: Functional test areas

OS, drivers Logic

circuitry Power

electronics Signal Quality Mechanics

Robustness EMC

Applications Good coverage

Partial coverage

Poor coverage

(36)

4.4 In Circuit Testing

In Circuit Testing (ICT) started to appear in the 1970s and was the first step to automate the procedure of manually testing individual components separately after PCB mounting [Park1].

Here the ICT-techniques bed of nails and flying probe is discussed

4.4.1 Bed of nails

Bed of nails, uses a test fixture with several nails to connect to the PCB components. This enables testing each component individually. A similar manual action would be connecting a multimeter or oscilloscope to strategic positions on the PCB. The method enables testing of resistors, capacitors, inductors and open/shorted nets.

The PCB is put on top of the bed of nails and pressed down, connecting the components to the nails. The ICT method can be used to simultaneously connect hundreds of strategical position and then use programming logic to connect to the nails through the main plate. The nail connections are used to test the components by apply signals, currents or voltages and measure the result. ICT is also used to produce and read logical patterns on ICs [Fein1]. See Illustration 6: Bed of nails [www.ami.ac.uk].

Illustration 6: Bed of nails [www.ami.ac.uk]

Illustration 5: In circuit tests areas

OS, drivers Logic

circuitry

Power electronics

Signal Quality Mechanics

Robustness EMC

Applications Good coverage

Partial coverage

Poor coverage

Konstruktion av generisk testmiljö för inbyggda system

Construction of generic test environment for embedded systems

Construction of generic test environment for

embedded systems

Magnus Dormvik Gustav Leesik

Sammanfattning

Construction of generic test environment for embedded systems

Abstract

FOREWORD

NOMENCLATURE

Abbreviations

Terms

TABLE OF CONTENTS

1 INTRODUCTION

1.1 Background

1.2 General goals

1.3 Purpose

1.4 Disposition

1.5 References

1.6 Priority

1.7 Method

2 PROBLEM ANALYSIS

2.1 Business analysis

2.2 Summary interviews

2.3 Company-wide areas of improvements

2.4 Test's role in a greater perspective

3 TARGET AREA AND APPROACH

3.1 Top-down approach

3.2 Bottom-up approach

3.3 Advice for decision

3.4 Decision

4 TESTING METHODS OF EMBEDDED SYSTEMS

4.1 Testing methods

4.2 Areas of testing

4.3 Functional test

4.4 In Circuit Testing