Human error management 4.0: Augmented Reality Systems as a tool in the quality journey

INDUSTRIAL MANAGEMENT, SECOND CYCLE, 30 CREDITS STOCKHOLM SWEDEN 2018,

Human error management 4.0

Augmented Reality Systems as a tool in the quality journey

DANIAL ETEMADY QESHMY

JACOB MAKDISI

Human error management 4.0

Augmented Reality systems as a tool in the quality journey

by

Danial Etemady Qeshmy Jacob Makdisi

Master of Science Thesis TRITA-ITM-EX 2018:188 KTH Industrial Engineering and Management

Industrial Management SE-100 44 STOCKHOLM

Hantering av mänskliga fel 4.0

Augmented Reality som ett verktyg i kvalitetsresan

Danial Etemady Qeshmy Jacob Makdisi

Examensarbete TRITA-ITM-EX 2018:188 KTH Industriell teknik och management

Industriell ekonomi och organisation SE-100 44 STOCKHOLM

Human Error Management 4.0

Augmented Reality Systems as a tool in the quality journey

Danial Etemady Qeshmy Jacob Makdisi

Approved

2018-06-05

Examiner

Luca Urciuoli

Supervisor

Jannis Angelis

Commissioner

Scania CV AB

Contact person

Christer Wilhelmsson

Abstract

The manufacturing industry is shifting, entering a new era with smart and connected devices. The fourth industrial revolution (Industry 4.0) is promising increased growth and productivity by the Smart Factory and within the enabling technologies is Augmented Reality (AR). This is a technology that can be used to augment the reality with digital information. At the same time as the technology is introduced, errors in manufacturing are a problem which are affecting the productivity and the quality. The errors can be caused by humans and companies strive to eliminate the errors caused by humans.

This research aims to find the main causes of human errors in assembly lines and thereafter explores whether AR is an appropriate tool to be used in order to address those issues. Based on a literature review that identified and characterized a preliminary set of root causes for human errors in assembly lines, these causes were empirically studied in an exercise that covered an in-depth case study at a multinational automotive company. Data in form of interviews and deviation reports have been used to identify the causing factors and the result showed that the main causes of human errors are the amount of thinking, deciding and searching for information which affected the cognitive load of the operator and in result their performance. Several interviews with experts in AR allowed to verify if this technology would be feasible to solve or mitigate the found causes.

Besides that, in repetitive manual assembly operations, AR is better used showing the process in order to train new operators, at the same time as for experienced operators AR show information only when an error occurs and when there is a need of taking an active choice is more appropriate. Nevertheless, while theoretically able to managing human error when fully developed, the desired application makes the augmentation of visual objects redundant and increasingly complex for solving the identified causes of errors which questions the appropriateness of using AR systems. However, the empirical findings showed that for managing human errors, the main bottleneck of an AR system is the software and AI.

Key-words

human error management, artificial intelligence, human performance, vehicle assembly, augmented

Hantering av mänskliga fel 4.0 Augmented Reality som ett verktyg i

kvalitetsresan

Danial Etemady Qeshmy Jacob Makdisi

Godkänt

2018-06-05

Examinator

Luca Urciuoli

Handledare

Jannis Angelis

Uppdragsgivare

Scania CV AB

Kontaktperson

Christer Wilhelmsson

Sammanfattning

Den tillverkande industrin skiftar och går in i en ny era där smart och uppkopplad teknologi introduceras i de operativa delarna av tillverkningen. Denna fjärde industriella revolution (Industry 4.0) som den även kallas för med smarta fabriker, utlovar ökad produktivitet och tillväxt.

Bland de teknologier som representeras i detta nya landskap återfinns Augmented Reality (AR), vilket är en teknik som används för att förstärka verkligheten med digital information. I samband med att denna nya teknik introduceras, är avvikelser i produktion ett problem som påverkar företags produktivitet och kvalitet.

Den mänskliga faktorn är en bidragande del till detta problem och företag strävar efter att eliminera felen orsakade av människor.

Denna studie syftar till att hitta orsakerna till att människor orsakar fel i produktion och därefter utforska om AR är ett lämpligt verktyg att använda för att råda bot på dessa orsaker och därmed eliminera felen.

Genom en litteraturstudie har det identifierats ett antal faktorer som påverkar den mentala belastningen hos människor i produktionssammanhang. Dessa faktorer har därefter undersökts genom en fallstudie hos en multinationell tillverkare av kommersiella fordon. Datainsamling i form av intervjuer och avvikelsedata har använts för att identifiera de påverkande faktorerna och resultaten pekade på att behovet av att behöva tänka, leta efter information och fatta beslut påverkade den mentala belastningen mest. Intervjuer hölls med forskare och montörer för att definiera en lämplig AR funktion som sedan undersöktes genom flera intervjuer med forskare inom AR för att verifiera om AR är en lämplig teknik att använda för de identifierade orsakerna.

I termer av AR i en arbetsmiljö med repetitiva aktiviteter efterfrågas en funktion som visualiserar fel för montörer som är erfarna medan det för oerfarna montörer är bättre med visualisering av hela arbetsprocessen. Men, trots att systemet i teorin är lämpligt att använda för att hantera orsakerna till att felen uppstår så är den efterfrågade funktionen överflödig då visualisering kommer visas väldigt sällan samt att tekniken är väldigt komplex. Detta gör att det går att ifrågasätta hela funktionen av att använda AR system i det fall som studerades. Dessutom visade sig tekniken vara olämplig att använda i den miljö fallet utspelar sig i på grund av svårigheter med artificiell intelligens (AI).

Nyckelord

human error management, artificiell intelligens, human performance, fordons montering, augmented

W

Page

List of Tables xi

List of Figures xiii

1 Introduction 1

1.1 Introduction . . . . 2

1.1.1 Background . . . . 2

1.1.2 Problematization . . . . 3

1.1.3 Purpose . . . . 3

1.1.4 Research question . . . . 4

1.1.5 Delimitations . . . . 4

1.1.6 Expected Contribution . . . . 5

2 Positioning of the Study 7 2.1 Positioning of the Study . . . . 8

2.1.1 Previous Research . . . . 8

3 Literature Review 11 3.1 Literature Review . . . 12

3.2 Engineering Psychology and Human Errors . . . 12

3.2.1 Human Information Processing . . . 12

3.2.2 Signal Detection, Sensitivity and Vigilance . . . 14

3.2.3 Rasmussens skill-rule-knowledge framework . . . 15

3.2.4 Mental workload, Attention and Performance . . . 17

3.3 Augmented Reality . . . 20

3.3.1 What is Augmented Reality? . . . 20

3.3.2 Augmented Reality systems - Conceptualization . . . 21

3.4 Framework Connecting the Theories . . . 25

4 Method 29 4.1 Method . . . 30

4.2 Method to fulfill the purpose of the study . . . 30

4.3 Scientific Approach . . . 31

4.4 Research Design . . . 31

4.5 Data Collection . . . 33

4.5.1 Observations . . . 33

4.5.2 Documents . . . 34

4.5.3 Interviews . . . 34

4.5.4 Data Analysis . . . 35

4.6 Quality of the Study . . . 36

4.6.1 Validity . . . 36

4.6.2 Reliability . . . 37

4.6.3 Research Ethics . . . 38

5 Result 39 5.1 Result Outline . . . 40

5.2 Pre-Study Findings . . . 40

5.3 Human Errors . . . 40

5.3.1 Exposed Positions . . . 41

5.3.2 Observations at the main-line and RS3 . . . 42

5.3.3 Mental Workload at RS3 . . . 43

5.4 Augmented Reality . . . 49

5.4.1 Observations on the setting of RS3 position . . . 49

5.4.2 Desired Functionality from the Assemblers Point of view . . . 50

5.4.3 Researchers Point of View on Augmented Reality Systems . . . 52

6 Analysis 59 6.1 Analysis . . . 60

6.1.1 Answering SRQ1 . . . 60

6.1.2 Answering SRQ2 . . . 62

6.1.3 Answering SRQ3 . . . 64

7 Discussion 69 7.1 Discussion . . . 70

7.2 Sustainability . . . 73

8 Conclusions 75 8.1 Conclusions . . . 76

8.1.1 Answering MRQ . . . 76

8.1.2 Managerial Implications . . . 77

8.1.3 Contribution . . . 77

8.1.4 Study limitations and future work . . . 78

A Appendix A 81

A.0.1 Appendix - Interview Questions . . . 83

B Bibliography 85

TABLE Page

4.1 Key words in literature search . . . 32

4.2 Interviews conducted . . . 35

5.1 Type of Errors at RS3 . . . 42

7.1 Settings where HMD Augmented Reality would be more appropriate . . . 73

FIGURE Page

3.1 Information Processing (Wickens and Hollands, 2000) . . . 12

3.2 Rasmussens Performance Level and Errors . . . 15

3.3 Relationship between Mental Workload and Performance (Wickens and Hollands, 2000) 19 3.4 Mixed Reality Continuum (Milgram and Kishno, 1994) . . . 21

3.5 Concept of Augmented Reality . . . 21

3.6 Key Pieces of Augmented Reality . . . 22

3.7 Schematics of Augmented Reality and Key Components . . . 25

3.8 Mental Workload - Waterfall . . . 26

4.1 Research Design . . . 31

4.2 Research Activities . . . 33

4.3 Data Analysis Process . . . 36

5.1 Deviations and Stoppage time on each MO . . . 41

5.2 Human error on each position at MO4 during studied two months . . . 41

5.3 Mental Workload Factors . . . 44

5.4 Summary of the mental workload aspects at the RS3 Position . . . 48

5.5 Summary of assemblers thoughts on Augmented Reality systems . . . 52

5.6 Summary of researchers thoughts on Augmented Reality systems . . . 56

6.1 Distribution between slips and lapses at RS3 . . . 61

6.2 Schematic Figure of The causes of errors . . . 62

A.1 Interview Questions - Human Error . . . 83

A.2 Interview Questions - AR . . . 84

CHAPTER

1

I

1.1 Introduction

This study explores the possibility of using Augmented Reality systems in manufacturing as a mean to manage human made errors. The study was conducted as a case study at a multinational vehicle manufacturer to determine what the characteristics of the errors are, the causes of the errors and later on, if Augmented Reality is an appropriate tool to manage those causes.

1.1.1 Background

Throughout the history, technology has been in the epicenter of what has driven different indus- trial revolutions, with increased productivity and growth as a result. These industrial revolutions have had their underpinnings in the mechanization of the industry, to the electrification and recently the automation of the industry (Boston Consulting Group, 2015; McKinsey, 2015). How- ever, due to the rapid development of different smart and digital technologies, the industry is now entering the fourth industrial revolution, or Industry 4.0 as it has been known for (Boston Consulting Group, 2015). According to both consulting firms, McKinsey (2015) and Boston Con- sulting Group (2015), entering this new form of industrial revolution will bring benefits and value to companies embracing its potential by utilizing the digital technologies throughout the value chain. McKinsey (2015) mentions that among the different disruptions that is driving this industrial transformation is the different human- machine interaction, where augmented reality systems being part of this. The Boston Consulting Group (2015) defines nine different technological trends that are part of the fourth industrial revolution where Augmented Reality is one of those.

The concept of augmented reality is something that has been around for a long time, although it has not had any breakthrough until recently when the enabling technology has been developed for it to be used (Gilchrist, 2016). In its simplest description, augmented reality is used to augment the reality with information to enhance the performance of the task being conducted in the Smart Factory (Syberfeldt et al., 2017). It is within assembly settings where augmented reality has the potential improving the assembler performance. For instance, Accenture (2017) has in collabora- tion with Airbus developed a wearable augmented reality system with the aim of decreasing the complexity in the assembly and reducing the assembly time of seats by utilizing visualization.

Another actor who has applied the concept of augmented reality in the operations is Boeing, who has applied the technology for the installation of wiring with the aim of reducing the errors of the technicians and increase the productivity (Boeing, 2018). For Boeing, it is of highest importance that no errors occur and augmented reality has according to them the potential of reducing it (Boeing, 2018). In terms of utilizing augmented reality in warehouse logistics, field tests has according to Heutger and Kuckelhaus (2014) and McKinsey (2015) the potential of reducing the errors caused by humans with as much as 40 percent in picking activities thanks to augmented

reality. Moreover, other research has stated that AR has the possibility of reducing errors caused by humans, as it can work as a guide system (Porter and Heppelmann, 2017; Abraham and Annunziata, 2017; Regenbrecht et al., 2005; Boston Consulting Group, 2015; Scalabre, 2018).

Errors in manufacturing occurs and one factor causing errors is the human and are said to be more difficult to manage than technical ones (Groover, 2007). Human errors have effect on the efficiency on manufacturing, and in a study it showed that 23 percent of the unplanned downtime is caused by the workers in manufacturing, indicating that there is a need to manage it (Wright, 2017). Scania CV AB (Scania) is a global manufacturer of trucks and engines and is a part of the Volkswagen Truck and Bus Group (Scania CV AB, 2017; Volkswagen AG, 2018). Scania is a world leader within its field and has continuously aimed to improve their operating processes through their corporate culture "The Scania Way" (Scania CV AB, 2018).

Since Volkswagens acquisition of Scania CV, the aim of Truck and Bus Group has been de- veloped to becoming a global champion when it comes to commercial vehicles (Lundin, 2015).

This includes synergy effects in the group where Scania will play an important role (Scania Corporate Relations, 2014). For a company like Scania CV, which is well-known for their high quality products and their efficient processes, further pressure is put on them to continuously look solutions in order to manage human errors in their processes in order to live up to their reputation.

This study has been conducted at Scania CV during a period of 20 weeks.

1.1.2 Problematization

As new technologies are approaching and making their way out from laboratory settings, there is a need for an understanding what value they can actually bring. Augmented Reality is one promising technology within the new Smart Factory landscape that is promised to be able to reduce human errors. At the same time companies such as Scania CV are facing challenges with human errors as they have achieved a plateau in their quality journey where human errors are being more difficult to manage through traditional quality management and waste elimination methods. The assemblers on the production line are causing errors which have an effect on the productivity as well as the quality. With the entrance of new technology, there is a need of understanding if this new technology is appropriate to use in order to manage the errors caused by humans at a production line.

1.1.3 Purpose

The purpose of this study is to investigate what the causes of human errors are in manual assembly operations at a vehicle manufacturer. Further on, the purpose includes investigating if an Augmented Reality system is an appropriate tool for managing those causes.

1.1.4 Research question

In order to fulfill the purpose of this study, the following research question was defined:

MRQ: Is an Augmented Reality system an appropriate tool to manage human errors in manual assembly?

For the main research question to be answered, three sub research questions were developed. To understand what the characteristics of errors made by assemblers were, SRQ 1 was defined as following:

SRQ 1: What are the main types of human errors at the production line?

Once the error types were known, we sought to find the reasons why they occurred, hence SRQ2 was defined as:

SRQ 2: What are the main causes of human errors at the production line?

Since the characteristics of Augmented Reality are well covered in the literature and are generic, the focus of the next sub research question was not on the characteristics of Augmented Reality.

The focus was rather if Augmented Reality can deliver the solutions needed to manage the causes of human errors, thus SRQ3 was defined as:

SRQ 3: Is an AR system a feasible tool to manage those causes?

1.1.5 Delimitations

This study was carried out at the Gearbox Assembly plant at Scania CV in Sodertalje. As the research period was limited to 20 weeks, some delimitations were made in order to make the problem researchable. The study limited itself on one single position on the production line in order to manage the time-scope of the study. As the research was conducted within the field of Industrial Management and Economics, the study did not develop any software or tested an specific hardware to verify any conclusions or hypothesis. Moreover, as this study was conducted within the field of Industrial Management and Economics, it did not conduct a deep investigation on the enabling technologies of an Augmented Reality system but rather considered if an Augmented Reality system practically can provide the desired function generated by the in depth study of human errors.

1.1.6 Expected Contribution

The expected contribution of this study is divided into two, one academic contribution and one company contribution. From the academic perspective, was expected to provide with empirical findings for smart factory research and the appropriateness of Augmented Reality systems in manual assembly.

Moreover, this study was expected to contribute to the case company with an understanding of the root causes of the errors conducted on the line, how to target the root causes and conceptual methods on how to manage them. Lastly, the case company is expected to be provided with guidance on the appropriateness of the usage of Augmented Reality systems in manual assembly.

CHAPTER

2

P

S

2.1 Positioning of the Study

This literature review aims to describe and comment previous studies in the field of Augmented Reality and human errors and thus describe the positioning of the study in relation to previous research and the identified academic gap.

2.1.1 Previous Research

Hudoklin and Rozman (1992) studied the human errors in a man-machine system and its re- lation to stress and errors. Their conclusions were that human errors are the most probable cause of failures and that stress has a high impact on human performance. This is further elaborated by the work of Yeow et al. (2014) where they evaluated the impact of stress, fatigue and work environment on human errors in manufacturing industries. They concluded that there is a relation between stress and human errors in the manufacturing industry. They said that stress makes industrial workers divert their attention and lose their focus on the objective of a task. A study made by Wilson and Russel (2003) describes that the mental workload of the human operator is critical to optimal performance. More studies that have been conducted in various settings has also shown that the mental workload of an individual has a clear impact on attention and thus their performance (Yurko et al., 2010; Wilson, 2002 ;Bailey and Iqbal, 2008).

A study made by Tang et al. (2003) showed that in an laboratory assembly setting, 3D in- structions through head mounted displays reduced the test subjects perceived mental workload as some mental processes in the assembly task was offloaded to the augmented reality systems and thus reduced the error rate by 82 percent. In the same study, their conclusions were that augmented reality systems can relieve mental workload in assembly tasks and that since "expert"

assemblers are difficult to train, augmented reality systems are the next step in the process of augmenting human attention. The authors also stated that there is some work that needs to be done, both technical, as well as fitting augmented reality systems to the actual problem it is intended to solve. The study made by Beitzel et al.(2016) tested an augmented reality system in a laboratory setting where the result showed that through the aid of an augmented reality system, the test subjects perceived a decrement in stress and mental workload and their conclusion was that augmented reality systems are in particular beneficial in time-sensitive operations.

In a study made by Syberfeldt et al. (2015), an augmented reality prototype was tested in a laboratory environment where they simulated assembling tasks. They concluded that their augmented reality system consisting of head-mounted displays and 3D visualized objects that were overlaid on the real world did positively assist their test subjects in assembling procedures when compared to no visual aid. In the same study, the authors also concluded that one of the reasons for augmented reality systems has not yet had a breakthrough, excluding the technologi-

cal immaturity, is the user acceptance. They argue that the current augmented reality systems being tested in laboratory settings are not designed to do what the assemblers actually need and that the end-users are still not convinced that augmented reality systems will make the users more efficient.

In the work made by Fite-Georgel et al. (2011), he describes that only a small portion of aug- mented reality systems has successfully moved to real-user tests, and he argues that one of the reasons for this is that there is a need of a deeper understanding of the industrial setting in order to make augmented reality systems applicable in the industry. He further argues that more involvement in the development of augmented reality systems is needed, in particular from industry stakeholders which includes the end-users. His recommendation is that academic researchers must strengthen their collaborations with industrial partners in order to improve augmented reality prototypes in an iterative process of development that is based on the user’s feedback and viewpoints.

The statements of Fite-Georgel et al. (2011) is in line with what Regenbrecht et al. (2005) says about augmented reality systems. They begin with describing that most prototypes have been developed and tested in laboratory settings with a focus on service, maintenance, design and development, and training. They argue that it has been shown to be difficult to bring research out from laboratory settings in regard to Augmented Reality. One of the reasons is that in laboratory settings, the augmented reality projects and tests have taken place in pre-configured and specially prepared hardware and software environments, but when it comes to the real trial, most cases have failed, and this could be because the researchers have worked in "silos", meaning not having an end-user-centric design or application. They elaborate on this, saying that in many instances the end-users have not been integrated into the development of augmented reality systems and that their viewpoints have been overseen. They conclude, that from researcher’s point of view, the best-augmented reality solutions might not be the ones with the highest level of originality or novelty. The end users might find that there are simpler and more elegant solutions for their specific problems (Regenbrecht et al., 2005). Nee and Ong (2013) is on the same page, saying that in order to make augmented reality applicable in manufacturing, the desired functions must consider the users and have them in the center to bridge the issues of acceptance.

To conclude, although mental workload and its relationship to performance and errors is a topic that has been researched, not much study has been directed toward the industry and the mental workload of assemblers in manual assembly and the causes of high/low mental workload.

It is with great effort the presented studies have been found, and it is this gap in research that increases the originality of this study. As presented, augmented reality systems have a great potential for reducing the mental workload of operators and thus increase their performance.

However, a common theme among the presented research is that all tests have been conducted in a laboratory setting where the operators have been new to the task to be conducted. Moreover, an additional theme in the research presented is that there is an inquiry for a deeper under- standing of the end-users viewpoints, the real environmental setting and the real world problem augmented reality systems shall manage and solve. This study bridges this gap of understanding, by providing what the real world manual assembly needs in Augmented Reality systems and what functions is desired in order to manage common issues and causes of errors with the hope of accelerating the development of Augmented Reality systems in manual assembly.

C

3

L

R

3.1 Literature Review

This section first presents literature relevant to understand why and how human errors occur.

This include literature in the information processing of humans [3.2.1] and [3.2.2], followed by the different cognitive performance levels described by Rasmussen [3.2.3]. Thereafter, mental workload and its relationship to attention and performance is presented [3.2.4]. This is then followed by a generic conceptualization of Augmented Reality [3.3] and last, a chapter connecting the presented theories [3.4].

3.2 Engineering Psychology and Human Errors

The field concerning the design of machines that accommodates the limits of the human users is referred to as the field of Human Factors Engineering (Wickens and Hollands, 2000). The human factor has various definitions but Meister (1989) provides a simple but explanatory description where he says that it is how people accomplish work tasks in the context of human and machine operations and how the behavioral and non behavior dimensions affect the accomplishment. The main goal of the academic field is to reduce errors, increase productivity, and enhance safety and comfort when humans interact with machines. Engineering Psychology arises from the intersection between Human Factor Engineering and Psychology where much focus is on the information-processing capacities of the human brain which will be further elaborated in the following chapters.

3.2.1 Human Information Processing

Wickens and Hollands (2000) presents a model for stages of the human information processing process (see figure 3.1). This model has been widely used and is similar to the model presented by Groover (2007). The authors describe that the information processing of humans is represented by a series of stages and they argue that there is no fixed starting point in the sequence of stages.

Figure 3.1: Information Processing (Wickens and Hollands, 2000)

Sensory: Through the human senses, information and events in their environment can gain access to the brain of humans. Properties of human visual and auditory receptors have a

great impact on the quality of the information received by the brain (Wickens and Hollands, 2000).

Perception: For the human performance to be efficient, it is not sufficient with only the sensory.

The information that is received by the brain must be interpreted and translated in order to create a perception. There are two key features in the perception stage. Firstly, it generates a perception automatically and rapidly which means that the perception stage needs minimal attention resources allocated to it. Secondly, it is driven by both the input (information through the sensory) and it is also driven by its long-term memory. The speed of this stage and the automaticity is what distinguishes this stage from the other stages in the information processing model (Wickens and Hollands, 2000; Groover 2007).

Cognition and memory: Cognitive operations requires, in general, a greater time and more attentional resources than the other stages. This is due to the fact that cognitive operations are initiated by the working memory and other sets of activities such as rehearsal, reasoning and/or image transformation. The working memory of humans is a sensitive temporary storage of acti- vated information. Working memory is a key feature when conscious activities are needed which is dependent on the processing and transformation of information. The reason behind making the working memory sensitive is the fact that it is vulnerable when other activities need attention and mental presence allocated to them. In some cases, the well-rehearsed working memory can transform to long-term memory which is much less vulnerable (Wickens and Hollands, 2000 ; Groover 2007 ).

Response selection and execution: When an individual has gained an understanding of a situation which has been achieved through the earlier stages and augmented by cognitive transformation, a selection of response is done by the human. The selection of an action is different from executing a task since the execution requires physical motions to be done in order to reach the goal of the selected action (Wickens and Hollands, 2000).

Feedback: In the feedback loop, the actions chosen by an individual are directly sensed by the human. One of the implications of the feedback loop is that the flow of information can be initiated at any point and the flow of feedback information is continuous (Wickens and Hollands, 2000 ; Groover 2007).

Attention: The last property of the human information processing model is attention. Many men- tal operations do require the selective application of these limited attention resources and they are not carried out automatically. To the left of the model, the attention is selectively allocated to channels of sensory materials to be processed, which is also referred to as selective attention. In the case of visual information, the limited resource is called foveal vision which is/can be directed

to different channels in the existing environment of a human. In the case of selective attention, the scanning process of the eyes are many times driven by the past experience were the human knows where to look. But this experience can result in being the cause of errors since humans can miss the details in the rest of the environment. If an individual has many operations going on at the same time, the individual must have a strategy for dividing the attention resources among two or more activities (Wickens and Hollands, 2000 ; Groover 2007).

3.2.2 Signal Detection, Sensitivity and Vigilance

The information processing of humans presented in the chapter above usually begins with the detection of some environmental event. The associated problems of information processing of humans are those of detection, recognition, and diagnosis. Signal detection theory can be applied in any situation where there are two discrete states of the world, signals and noise, that cannot easily be distinguished (Wickens and Hollands, 2000).

Signals could be referred to as something that initiates the information processing of humans, it could be an instruction, light or any situation. The ability to detect a signal that initiates the information processing of humans is highly dependent on the sensitivity. Sensitivity refers to the separation of noise and signal distribution of the environment. The sensitivity measure is called d’ which corresponds to the degree of ability to separate signals from noise. It is said that a major cause to deviations from the intended goal of an action to occur in assembly settings is because the operator’s poor memory for the precise physical characteristics of the signal that initiates the information processing. When memory aids are provided to remind the operator of what and how the signal looks, the d’ value approaches its optimal value (Wickens and Hollands, 2000).

In vigilance tasks, the operators are required to over a long period of time detect signals that are periodic, unpredictable and infrequent. Two conclusions have derived from studies in the area of vigilance: Operators show lower vigilance levels than desired, and the vigilance levels fall steeply during a shift (Wickens and Hollands, 2000; Groover 2007). As a subject’s target signal is reduced e.g. the target signal is hard to detect, the sensitivity decreases and vice versa (Parasuraman et al., 2008) . Sensitivity usually decreases when there is uncertainty about time and location of when the subjects target signal will appear (Parasuraman et al., 2008; Wickens and Hollands, 2000) . When the even rate is increased (number of activities per unit time), the sensitivity level decreases of operators (Wickens and Hollands, 2000).

Wickens and Hollands (2000) and Parasuraman et al., (2007) explain that vigilance tasks and performance is influenced by factors such as the display, task type, environmental stressors and design of a work-station. Further more, they have concluded that in settings where visual signals have been applied, the sensitivity decrement has still taken place due to the need of remain

focused on the visual signals during longer periods of time extract a till in fatigue (Wickens and Holland, 2000; Parasuraman et al., 2007) . Because of the fatigue, there are difficulties in sustaining the attention and the sensitivity to detect signals decreases and that activities that require more mental effort will lower the sensitivity of the individuals. They argue that there are in general four techniques to combat the loss of vigilance and hence increasing the sensitivity:

1. Show target examples, 2. Increase target salience, 3. Vary event rate and 4. Train observers when novice (Wickens and Hollands, 2000).

3.2.3 Rasmussens skill-rule-knowledge framework

Rasmussen (1983) has developed a framework which is directed toward operators in industrial settings. Rasmussen (1983) presents three type of performance levels and their associated errors.

The performance levels are divided into three cognitive processing models that is used by an operator when he or she performs a task.

The framework describes that an operator first perceives and interprets information in the processing system described earlier and that the information is processed cognitively in one of the following levels: Knowledge-based, Rule-Based or Skill-Based (Reason, 2009; Rasmussen, 1983). Furthermore, the associated errors in the three performance levels can be divided into three categories: Mistakes, Slips, and Lapses (see figure 3.2).

Figure 3.2: Rasmussens Performance Level and Errors

Knowledge-Based Performance: At this performance level, the operator has been pre- sented to an unfamiliar situation and an environment where previous know-how or rules are not available and thus the control of the performance needs a higher conceptual level and a knowledge-based approach. The individual must assess the situation and the goal is explic- itly formulated which derives into an action plan (Rasmussen, 1983; Reason, 2009; Groover 2007).

The errors associated with this performance level occur due to failure in understanding the

situation. What is typical for these types of errors is that the operator’s memory limits are overloaded, and the operator fails to consider different alternatives. This form of error may also be a result of the operator failing to interpret complex information (Wickens and Hollands, 2000).

In regard to knowledge-based errors, typical control measures are clear displays, diagnostic tools and decision-making aids and organizational learning and training (Stewart and Grout, 2001)

Rule-Based Performance: The rule-based performance level refers to a sequence of routines in a familiar work situation which is typically controlled by an operator’s stored rule or procedure which the operator empirically has derived through the previous occasion or instructions. Further more, the rule based performance level is characterized by the use of checklists during the work, telling the operator what to do and in what order. At this performance level, the goal is not explicitly formulated but is generated implicitly by being in a situation where the rules are initiated (Rasmussen, 1983; Reason, 2009; Groover 2007).

The errors associated with this performance level differs from the knowledge-based errors in the way that these occur even though the operator is more confident and sure about the situation. Since the operator thinks he/she knows the situation, they invoke a rule or plan of action to deal with the situation. However, the operator fails to involve the correct rule or plan of action (Wickens and Hollands, 2000; Reason, 2009; Groover 2007). The control measures in order to tackle rule-based errors are similar to the control measures of knowledge-based errors but with regular drills and exercises (Stewart and Grout, 2001).

Skill-Based Performance: At this performance level, the human performance is linked to stored patterns of pre-programmed instructions. The skill-based level represents performance during acts or activities where the individual follows a statement of a created intention and the performance take place without conscious control and smoothly follows an automated and highly integrated pattern of behavior (Rasmussen, 1983; Reason, 2009; Groover 2007).

The errors associated with the skill-based level are slips and lapses. These errors occur due to the perception and interpretation of the situation (Wickens and Hollands, 2000; Reason, 2009). A very common slip are what is referred to as capture errors (Wickens and Hollands, 2000). This mean that the intended stream of behavior is "captured" by a similar, well-practiced behavior and routine. These capture errors are usually initiated and take place for three reasons: (1) the action sequence involves a slight departure from the more frequent action, (2) some characteristics of the stimulus environment or the action sequence is related to the wrong (but more frequent) action; and (3) The action sequence is relatively automated and therefore not monitored by one’s attention.

What differs slips from lapses is that slips occur when an operator is carrying out the right intention incorrectly. A slip represents a deviation from the intended action, however, a lapse takes it one step further where the operator does not execute any action at all . Lapses can be tightly connected to the loss of memory, meaning that the operator has understood what to do, know how to do it, but forgot to do it. What is considered a major factor causing these forms of errors (Lapses and Slips) is an interruption of the operator and loss of attention causing the oper- ator to forget what to do (Wickens and Hollands, 2000; Reason, 2009). Ishi et al. (2013) describes that slips are caused by mistakes in judgment such as misidentifying or misunderstanding a situation and when an operator forgets to do an activity it is considered as a lapse.

Ishi et al. (2013) argue that in contrast to the measures that can be taken at the previous performance levels (knowledge-based and rule-based), slips and lapses cannot be prevented by traditional training or discipline for thorough conformance to manuals. They argue that these errors do not depend on the years of experience of an operator but depends more on their physical and psychological conditions. Tafton et al. (2011) argues that skill-based errors are prevalent and have fundamental cognitive underpinnings and says that these types of errors can’t be reduced by policy or training but can rather be solved with robust systems by applying cognitive theory to the design of systems. Wickens and Hollands (2000) says that checklists and reminders where procedures with "place markers" which tick off each step in the process as well as warnings and alarms to detect errors is a way of tackling skill-based errors. Ishii et al. (2013) are in the same line saying that double-check assistance with a computer is a way of reducing skill-based errors.

Stewart and Grout (2001) says that by using mental aids such as computer-based intelligent decision support systems and memory aids that reduce the mental workload one can control skill-based errors.They also argues that for skill-based errors, error detection systems are good tools that help to reduce the mental workload of the individual as long as it is easy to trace back where the error occurred. This error detection can be carried out by the operator themselves or an external system. Zhang et al. (2004) describes that memory aids, decreased multitasking, decision support, action tracking, information reduction and immediate feedback are among the methods to control skill-based errors.

3.2.4 Mental workload, Attention and Performance

There is no unilateral definition of mental workload, but the most self-explanatory definition of the mental workload is provided by Parasuraman et al. (2008) defining mental workload as "The relationship between the function relating the mental resources demanded by a task and those resources available to be supplied by the human operator", p.145. The definition presented is in line with how Wickens and Hollands (2000) refers to mental workload, meaning that mental workload is the demands a specific task put on the cognitive capacity, similar to the relationship

between physical demand and the amount of energy put into pulling, pushing and moving.

There are both objective and subjective measurement techniques in order to assess the mental work of an individual or group (Wickens and Hollands, 2000). Nasa Task Load Index (TLX) is a subjective method that was developed by the Human Performance Research Group at Nasa’s Ames Research Center (National Aeronautics and Space Administration (NASA), 2017). It has been cited in many studies and is considered to have a high reliability and applicability (Wickens and Hollands, 2000). The Nasa TLX assesses the perceived workload of an operator among seven dimensions, mental workload being one of them. In this study, the mental workload is of primary interest hence the question to be answered has been derived only from that dimension and been customized to fit our case. The Nasa TLX describes that there are a number of factors that affect the mental workload of an individual and among these are: how much thinking, deciding, calculating, looking and searching a task requires (Wickens and Hollands, 2000). The validity of these factors is further strengthened by a study made by Brolin et al. (2017) where they studied the cognitive aspects affecting human performance in manual assembly. Their study showed that when assemblers needed to look for components, searching for the right information and deciding what to assemble, the mental workload of the assemblers increased and the task performance decreased. Berlin and Adams (2017) also describes that the degree of searching for information affects the mental workload of an operator and that the process of always thinking if the operator has done a task correctly, in turn, affect the mental workload of the operator.

Wickens and Hollands (2000) describes that there is a supply and demand relationship (see figure 3.3) associated with the mental workload and that there is a "red line" where the task performance of an operator decreases as the mental workload increases and passes the red line. If an activity or task requires a high degree of cognitive effort that surpasses the cognitive resources of the operator, a decrement in task performance will be noted. Kantowitz (2000) takes it one step further stating that whenever the mental workload is too high or too low, the performance of the operator will decline.

Figure 3.3: Relationship between Mental Workload and Performance (Wickens and Hollands, 2000)

When discussing mental workload, attention cannot be neglected since these concepts are related (Kantowits, 2000; Wickens and Hollands, 2000). According to Kantowitz (2000) the two concepts overlaps each other in a Venn diagram, but that there is a disagreement about the size and form of this intersection. Further more, the concept of stress can not be neglected, as Wickens and Hollands (2000) explains, the mental workload that exceeds the cognitive resources of an operator increases the stress level and hence affects their ability to sustain the desired attention level. This, in turn, affects the operators’ ability to detect the signals that initiate the information processing of humans by decreasing the sensitivity level which was discussed in earlier chapters.

According to Endsley et al. (1999) an increased mental workload can affect the attention and thus the situational awareness of an operator as only a subset of the information received can be attended or also lead to a misperception of information provided.

Automation is often introduced to alleviate the mental workload of an operator or to augment system performance to reduce human errors (Groover, 2007). Wickens et al. (2012) refers to automation as "The performance by machines (typically computers) of functions that previously carried out, whether fully or partially, by humans", p. 378. The principal benefit that comes with automation is that if it is carefully designed, the systems can reduce human workload, both mental and physical. The workload reductions can occur in execution stages, e.g. an automated screwdriver, in decision choices and/or in the acquisition of information (Wickens et al., 2012).

The potential for automation to reduce mental workload by providing cognitive support makes the concept attractive in environments where an operator has a high time pressure and where cognitive efforts need to be minimized in order for an operator to carry out many tasks (Parasura- man et al., 2009; Wickens et al., 2012; Brolin et al., 2017).

According to Norman (1990) automation is efficient to the user when the user receives feed- back if a task is carried out correctly. Receiving multimodal feedback as a result of automation

has shown to be an efficient method in supporting assemblers by reducing the mental workload of users. There are in general three types of feedback that can support assemblers: haptic feedback, visual feedback and auditory feedback (Vitense et al., 2003; Ishii et al., 2013). Auditory feedback refers to the process where an operator is provided with audio support, haptic feedback is when operators can sense a vibration on their skin, and visual feedback refers to when operators visual senses are exposed to some form of stimuli (Vitense et al., 2003; Ishii et al., 2013) . In their study, Vitense et al. (2003) found that in an assembly setting, haptic and visual feedback are more efficient when used alone as well as in a combination with each other. When these feedback types were implemented, the assemblers perceived mental workload decreased and their performance increases.

To conclude, the mental workload of an operator is determined by how much thinking, deciding, calculating/counting, looking and searching they must do. If the mental workload of an operator exceeds his or her capability, the task performance will decline as a consequence of the increased level of stress and decreased the level of attention. As the workload, stress, and attention is not at its optimum level, it is more difficult for operators to detect signals and/or misperceive signals in their environment that initiates the information processing. There are different techniques to combat the high level of mental workload. Automation is often introduced to alleviate the mental workload by replacing redundant activities of an operator. The cognitive support the operator can get is in general haptic, auditory and visual feedback. The feedbacks main target is to reduce the aspects that affect the mental workload of an operator to an optimum level since too low or too high mental workload reduce task performance and increase the probability of human errors to occur

3.3 Augmented Reality

This chapter will present Augmented Reality on a conceptual level in order to understand what the concept and aim of Augmented Reality is and the key parts to consider.

3.3.1 What is Augmented Reality?

As the name is stating, Augmented Reality is a medium that has the ability to enhance events in the real work by adding digital content. Poelman and van Krevelen (2010) described Augmented Reality systems as a set of technologies that together enables humans to see and hear more than they otherwise would do. According to Wang et al. (2016) Augmented Reality can be considered as a set of innovative and effective human-computer interaction techniques. Poelman and van Krevelen (2010) and Palmarini et al. (2018) base their perceptions of Augmented Reality on the definition made by Milgram and Kishino (1994) and their mixed reality continuum as a way of Augmenting the real world with virtual objects (3.4).

Figure 3.4: Mixed Reality Continuum (Milgram and Kishno, 1994)

Wang et al. (2016) have conducted a comprehensive survey of Augmented Reality systems in the domain of assembly processes and they concluded that the concept of Augmented Reality has a potential future within the assembly, but the technology has to be developed within different fields.

In the study made by Ong et al. (2008) where Augmented Reality applications in a manufacturing setting were studied, they found that Augmented Reality has made much progress in the recent years but it is still on an exploratory stage. In their research they found that one of the basic fundamental issues that has to be addressed when designing an Augmented Reality system for assembly assistance, is to determine the factors when, where and what. According to Ong et al.

(2008) when having the answers to these factors, the function of the visual AR system will be determined.

3.3.2 Augmented Reality systems - Conceptualization

Each and every time an Augmented Reality system is going to enhance the reality with digital content for the user, the system undergoes a process consisting of several steps from acquisition of the reality to the visualization and augmentation of it (Peddie, 2017; Heutger and Kuckelhaus, 2014; Furth, 2011; Horejsi, 2015). However, for this thesis we will use the the process described by Craig (2013) as it is less technical. This simplified process is divided into two steps:

Figure 3.5: Concept of Augmented Reality

1. Determining what is happening in the reality. This means that the system acquires data from the operating environment through various sensors. This data is then processed in order to make use of it.

2. Visualize the processed data. This means that the system is displaying the content that is derived from the data that has been processed on a display. This augmentation can take its shape in different forms depending on the purpose and functionality of the system.

In the research made by Peddie (2017) regarding visual augmented reality, he makes a dis- tinction between the different forms of visual augmentation.The author argues that there are generally two ways of augmenting the reality: (1) either the system provide general information and/or (2) providing specific instructions with digital overlays. What distinguishes the two types of augmentation is the degree of passivity or interaction the user has with the digital content.

Informative Augmented Reality is considered more passive in the sense that the user do not have any direct interaction with the digital content. Such form of augmentation takes its shape in form of information side-by-side with the reality, and is not integrated in to the real environment.

On the other hand, the instructional Augmented Reality is the method of incorporating digital overlays that are laid upon the reality and the objects within it. This type of Augmented Reality is also more interactive, where the user can manipulate the augmentation and move them around.

According to Peddie (2017), one must remember that even though these two types of visual augmentation exists, there is nothing that says that the system utilizes one over the other, it is equally common to have a mix between them.

However, in order to generate augmentations on the reality, there is a need of an Augmented Reality system. According to Craig (2013) and Wang et al. (2016) there is no universal Augmented Reality system that suits all the use cases, but rather this is something that is engineered to specific cases. But what Craig (2013) emphasizes on is that for all systems, there must be a conjunction between hardware and software. The software will tell the system what to do, while the hardware is the piece that conducts the task. According to Poelman and van Krevelen (2010), the enabling technologies for an functional Augmented Reality system has remained the same ever since the first Augmented Reality system was developed back in the 60s. Conceptually, the key pieces are (Poelman and van Krevelen, 2010; Azuma, 1997; Peddie, 2017; Craig, 2013):

Figure 3.6: Key Pieces of Augmented Reality

Hardware

The hardware of Augmented Reality systems can roughly be divided into displays, sensors and processors.

Displays

Mainly for vision based augmented reality systems, the taxonomy can be divided into two main categories of display types, namely wearable devices and non-wearable devices (Peddie, 2017).

Normally, when people refer to Augmented Reality systems, they refer to wearable devices, as these have been used in industry, military, medics, logistics and more (Cirulis and Ginters, 2013;

Heutger and Kuckelhaus, 2014; Furth, 2011; Wang et al., 2016; Poelman and van Krevelen, 2010).

Wearable systems are also referred to as Head-Mounted-Displays (HMD) and could be integrated into helmets, contacts and headsets, were headsets typically refer to smart glasses. What these HMDs does, is that they place the virtual environment, in conjunction with the real world, in the sight of the user (Heutger and Kuckelhaus, 2014; Peddie, 2017; Furth, 2011). The advantages of using wearable devices are that they allow the human to use both his/her hands in working conditions and provides the user with more flexibility in overall (Syberfeldt et al., 2017).

On the other hand, the non-wearable devices can be clustered as hand-held mobile devices, stationary devices, and head-up displays. The mobile devices refer to smartphones, tablets etc.

According to Furht (2011) and Peddie (2017) it is the mobile devices that are driving the develop- ment of the concept of Augmented Reality forward and is a key contributor to why Augmented Reality will most probably be a commonality in the future. The stationary systems refer to Augmented Reality system that usually uses some set of projection or hologram. These systems are more rigid and often requires a bit more space than other AR systems (Peddie, 2017; Furht, 2011; Poelman and van Krevelen, 2010; Azuma, 1997; Wang, et al., 2016). Head-up displays have been used in different settings, and they are mostly known for being used in aircraft, both fighter jets, and commercial planes, and are also being introduced to commercial vehicles (Peddie, 2017)

Sensors

The sensors role in the Augmented Reality system is to provide information of the reality to the system, with the purpose of making the system aware of what is happening in the environment.

There are several different sensor types that can be used, ranging from camera sensors to GPS sensors and these can work either in isolation or in conjunction with each other (Craig, 2013).Pri- marily the sensors are used for giving input to the tracking system, which is considered the heart of the system. The tracking system is what enables the system to determine the position (localiza- tion and orientation) of the user relative to the reality. For an accurate tracking, it is suggested that several sensors are used in conjunction with each other (Craig, 2013; Peddie, 2017; Poelman and van Krevelen, 2010; Azuma, 1997; Furth, 2011). What Peddie (2017), Poelman and van Kravelen (2010) and Craig (2013) are stating, is that since the development of high-performance

smart phones camera systems has achieved exponential development in terms of resolutions, size and price, these have pushed the optical tracking systems of Augmented Reality forward, and there is a promising future within this technology for vision based tracking and sensing.

However, for this to work, the system must have computer vision software that can analyze the captured reality accurately, either by using fiducial markers (marker-based vision system) or natural features of the environment (markerless vision system) (Craig, 2013; Poelman and van Krevelen, 2010). According to Lima et al. (2017), the marker less vision system is what will push the acceptance level of Augmented Reality forward.

However, not only shall the sensors acquire information for the tracking system. Sensors are also used in order for the system to get an understanding of what the user is doing. For this purpose, there are both passive as well as active sensors. The use of physical buttons on the system is a more active way for the system to get an input from the user, while using the camera to detect gestures is a more indirect way of giving input to the system (Craig, 2013).

Processors

The role of the processor is to function as the brain of the system and conduct the computation for the system along with other computer units (Craig, 2013). The computing can be carried out in different ways, and the different ways has their own benefits and drawbacks. Mainly, the two ways of processing the information is either onboard or external. For instance, in their study, Hashem et al., (2015) mentions that cloud computing will provide devices with computational power that will further enhance their advances in computing. Furth (2011) mean that since the technology has rapidly developed with mobile systems, a lot of computing power is today available in a small package. However, what is crucial for the processor is that it is powerful enough to be able to provide augmentation in real time. This mean that the system must understand what is happening and proceed without any lag/latency (Craig, 2013).

Software

The software of the Augmented Reality system plays a vital part for the system, as this has the role of making use of the gathered information from the environment. The software must be capable of integrating with the sensors in order to turn their input into something valuable. For instance, if mainly the input is gathered through camera sensors, there is a need of using vision system AI algorithms that are powerful enough to make something valuable of the input (Craig, 2013).

Interaction

The interaction with the Augmented Reality system is the way the user is giving inputs to the system. As the Augmented Reality system gives information and/or instructions to the user, it