15 Credits
Survey of Virtual and Augmented Reality
Implementations for Development of Prototype for
Practical Technician Training
Tobias Lindvall & Özgun Mirtchev
Computer Engineering Programme, 180 Credits
Örebro, Sweden, Spring 2017
Examiner: Franziska Klügl
Örebro Universitet Örebro University
Institutionen för School of Science and Technology Naturvetenskap och Teknik SE-701 82 Örebro, Sweden
Virtual training is a vital way of educating technicians to make them prepared for real maintenance work. Technicians that are educated to perform maintenance work on JAS 39 Gripen, complete parts of their training through the application Virtual Maintenance Trainer (VMT), which can provide a detailed simulation of the aircraft during operation. The technicians are able to complete courses and lessons with specific procedures such as debugging internal computers and parts of the aircraft and performing maintenance work to fix errors. However, the application is desktop-based and to make the education even more effective, there is a desire to explore the possibilities in virtual and augmented reality.
This report explores the alternatives of education tools in virtual reality and augmented reality through a survey. In the survey, the advantages and disadvantages of current implementations are examined to provide an optimal system which could work to give technicians a realistic practical training simulation experience.
Based on the results of the survey and by using the game engine Unity, a prototype application is built which can simulate technician training procedures on a model of JAS 39 Gripen. HTC Vive and Leap Motion were used to immerse the user into the simulation world and to enable realistic interaction. A technician may be able to learn through completing different training procedures in the simulation by walking around and interacting with a full-scaled Gripen aircraft.
Sammanfattning
Virtuell träning är ett viktigt sätt att utbilda tekniker för att förbereda dem för underhållsarbete i verkligheten. Tekniker som utbildas för att utföra underhållsarbete på JAS 39 Gripen, genomför delar av utbildningen genom programmet Virtual Maintenance Trainer (VMT), som kan återge en detaljerad simulering av flygplanet under drift. Teknikerna kan delta i lektioner med specifika uppgifter som inkluderar att felsöka interna datorer och delar av flygplanet samt utföra underhållsarbete för att åtgärda fel. Programmet är dock skrivbordsbaserat och för att göra utbildningen mer effektiv, finns det en önskan om att utforska möjligheterna i virtual och augmented reality.
Denna rapport undersöker alternativen för utbildningsverktyg i virtual reality och augmented reality genom en teoretisk undersökning. I undersökningen vägs fördelar och nackdelar för nuvarande implementeringar för att tillhandahålla ett optimalt system som kan fungera för att ge tekniker praktisk erfarenhet i en realitisk träningssimulering.
Baserat på resultaten från undersökningen och genom att använda spelmotorn Unity, har en prototypsapplikation skapats som kan simulera teknikerutbildning på en modell av JAS 39 Gripen. HTC Vive och Leap Motion användes för att låta användaren kliva in i simuleringsvärlden och för att möjliggöra realistisk interaktion. En tekniker kan lära sig att utföra underhållsåtgärder genom att genomföra olika träningsförfaranden i simuleringen genom att gå runt och interagera med ett fullskaligt Gripen-flygplan.
A big thank you to our supervisor at Saab, Johan Gustafsson, for all the help, encouragement and support during the exam work. A special thanks to Linus Lindberg at Saab, who helped us by providing valuable technical knowledge regarding 3D modelling.
Many thanks to our supervisor at the university, Andrey Kiselev, for giving feedback for writing the report and for enabling us to use available hardware for the prototype. Also thank you to our examiner, Franziska Klügl, for help with writing the report.
List of Figures iv
Abbreviations v
1 Introduction 1
1.1 Background . . . 1
1.1.1 Saab . . . 1
1.1.2 Virtual Maintenance Trainer . . . 1
1.2 Project . . . 3 1.3 Objective . . . 3 1.4 Requirements . . . 4 1.5 Division of Labour . . . 4 2 State-of-the-Art Survey 5 2.1 Introduction . . . 5
2.2 Clarification of Virtual and Augmented Reality . . . 5
2.3 Guidelines for Training in VR and AR . . . 6
2.4 Architecture . . . 6
2.5 Virtual Reality . . . 7
2.5.1 Introduction . . . 7
2.5.2 Hardware and Software . . . 7
2.5.3 Related Studies and Implementations . . . 12
2.6 Augmented Reality . . . 17
2.6.1 Introduction . . . 17
2.6.2 Hardware and Software . . . 17
2.6.3 Related Studies and Implementations . . . 20
2.7 Result . . . 24
2.7.1 Multi-user Capabilities . . . 24
2.7.2 Graphics and System Performance . . . 25
2.7.3 User Interactions . . . 29
2.7.4 Miscellaneous . . . 30
2.8 Conclusions . . . 31
2.8.1 Result Evaluation . . . 31
2.8.2 Summary . . . 32
3 Methods and Tools 33 3.1 Methods . . . 33
3.1.1 Development Planning . . . 33
3.1.2 Event-Driven Implementation . . . 33
3.2 Tools . . . 35
3.3 Other Resources . . . 37
4 Prototype Development 38 4.1 Refuel Procedure Training . . . 38
4.1.1 Ground Crew Panel . . . 38
4.1.2 Refuelling Access Panel . . . 38
4.1.3 Refuel Equipment . . . 39
4.1.4 Instructions . . . 39
4.2 Result . . . 40
4.2.1 Virtual Model Representations . . . 40
4.2.2 Interaction Design . . . 43
4.3 Discussion of Implementation . . . 48
4.3.1 Expenditure . . . 48
4.3.2 Evaluation of Used Hardware . . . 48
4.3.3 Development Potential and Future Developments . . . 49
5 Discussion 50 5.1 Compliance with the Project Requirements . . . 50
5.1.1 Summary of the Requirements . . . 50
5.1.2 Fulfilled Requirements . . . 50
5.2 Impact on Society . . . 51
5.3 Project Development . . . 52
5.4 Reflection on Own Learning . . . 52
5.4.1 Knowledge and Comprehension . . . 52
5.4.2 Proficiency and Ability . . . 52
Bibliography 53 Appendix A Setting up Unity with HTC Vive and Leap Motion 62 Appendix B Prototype System Diagram 64 Appendix C Demonstration of Implementation 65 C.1 Menu Interaction . . . 65
1.1.1 An overview of the VMT system . . . 2
2.2.1 The RV Continuum . . . 5
2.4.1 Basic VR and AR System Overview . . . 7
2.5.1 Oculus Rift with Touch . . . 8
2.5.2 HTC Vive with controllers and Lighthouse basestations . . . 9
2.5.3 Samsung Gear VR . . . 10
2.5.4 Plugged Leap Motion controller . . . 10
2.5.5 Microsoft Kinect V1 & V2 . . . 11
2.6.1 Microsoft HoloLens . . . 18
2.6.2 Meta Company’s Meta 2 . . . 18
2.6.3 Google Glass . . . 19
2.6.4 Google Cardboard . . . 19
2.7.1 Image illustrating the FoV of HoloLens for displaying virtual objects . . . 28
3.0.1 A top-down view of the tracked area by using the Room Overview Window in SteamVR . 34 3.2.1 Basic Prototype System Overview . . . 35
3.2.2 HTC Vive with a front-attached Leap Motion . . . 35
4.1.1 Panels . . . 39
4.2.1 JAS 39C Gripen . . . 41
4.2.2 Refuel equipment . . . 41
4.2.3 Hand models . . . 42
4.2.4 Picked up refuel hose nozzle . . . 43
4.2.5 Rotating the refuel valve protection cap . . . 44
4.2.6 Button interaction . . . 44 4.2.7 Menu . . . 45 4.2.8 Laser pointer . . . 45 4.2.9 Navigation sequence . . . 46 4.2.10 Refuel manual . . . 46 4.2.11 X-Ray mode . . . 47 4.2.12 Multiple users . . . 47
A.1 View of finished project setup of VR camera and Leap Motion modules . . . 63
AR Augmented Reality. 3–6, 13, 17–25, 27, 29–33, 50–52 AV Augmented Virtuality. 5, 6
CAVE Cave Automatic Virtual Environment. 12–14, 24, 25, 31, 32 COTS Commercial off-the-shelf. 12, 24, 26, 31, 32
DOF Degrees of Freedom. 8, 10, 14, 15, 17, 18, 23, 27, 28, 31
FOV Field of View. 8–10, 15–19, 21, 22, 24, 26–28, 31, 48, 49 HMD Head-Mounted Display. 7–11, 13–20, 24–33, 35, 48
IMU Inertial Measurement Unit. 8, 26
SDK Software Development Kit. 10, 15, 20, 24, 33
VMT Virtual Maintenance Trainer. 1–3, 12
Educating technicians in practical training procedures often requires the use of expensive hardware which also might have a limited availability. A training procedure may include the technicians to do certain practical tasks in order to learn about maintenance and inspection work of a particular hardware such as a vehicle or other machines.
With the swift development of virtual reality and augmented reality, a desire among many companies has emerged to find out how the technologies could be utilised for practical training procedures as accessible and cost-effective educational tools.
This thesis includes a survey of the state of the art in virtual and augmented reality technologies. The survey focused on existing implementations for practical training and was conducted using a qualitative method. Using certain keywords related to the topic, data was congregated and then compiled to extract important information by considering different parameters relating to the objective.
A prototype was created (based on the results of the survey) to evaluate the technical capabilities and to show the concept of a practical procedure training in virtual or augmented reality.
1.1
Background
1.1.1 Saab
This thesis has surveyed the current technologies in virtual and augmented reality. Based on the result of the survey, a suggestion was made of how Saab could develop their technician training tool. Saab AB is a military vehicle manufacturer, founded 1937, by the Swedish government with the purpose to secure the production of Swedish fighters. One fighter in particular called JAS 39 Gripen, one of the most known fighters, started to be produced 1988 and has since then undergone four generations (A, B, C and D) and the latest in development is the E generation.
To maintain a Gripen fighter, extensive knowledge about the aircraft is required by educated
technicians, provided by the specialised maintenance technician training application for Gripen, Virtual Maintenance Trainer (VMT) 39, which is developed by the business area Saab Aeronautics. It allows the training of technicians who will specialise in inspecting and maintaining the Gripen fighter.
1.1.2 Virtual Maintenance Trainer
VMT 39 is a desktop-based education application developed by Saab Aeronautics that runs on the VMT platform, which supports other VMT training applications for other vehicles. The platform, also developed by Saab, can be run on the most common operating systems, such as Windows, Linux and UNIX. The VMT 39 application can be used to cost-effectively train Gripen technicians or pilots. It covers virtual based maintenance training through offering education of the aircraft system, procedure
Figure 1.1.1: An overview of the VMT system
training, system diagnostics and fault localisation. Tailored training sessions can be created using videos, sounds, charts, interactive simulations, 3D animations, etc. The system can be used partly by a technician student for self-education but also by a teacher under instructional training course. The system runs on a dedicated workstation, with three screens and a joystick. A diagrammatical overview of the system may be seen in figure 1.1.1.
1.1.2.1 Users
Trainees and instructors constitute the types of users that can use VMT 39. Trainees connect to the application server using the Java-based client where they are allowed to practice certain tasks. Instructors use the same client program but have the rights to create lesson setups for trainees to participate in.
1.1.2.2 Course Creation
The VMT platform is using the Shareable Content Object Reference Model (SCORM) [1] standard to make e-learning content and Learning Management System (LMS) work together. This enables creation of learning material which can be used in other SCORM conformant systems.
All the created courses in an application are backed up to the server. When an instructor has created a course in a VMT application, it is stored on the server enabling the selected trainees to access the same course through client workstations. In the next version of VMT, the Tin Can API [2], will be implemented instead of SCORM, which allows for more LMS functionalities.
During the use of the VMT 39 application as an instructor, it is possible to create new courses and custom lessons. For each course, different learning modules can be created with different components.
Course Modules define a lesson as a training, examination, guidance type etc. Different modules only shows relevant components that can be added to that type of module.
Course Components are added to a chosen course module. Different course components can in real-time display cockpit panels, show 3D models of the aircraft or show diagrams and panels to monitor electric, hydraulic and fuel flows. From the components the instructor is also able to decide if there are going to be any errors during run-time for the trainee to handle.
1.2
Project
VMT uses computer displays to visualise the maintenance work during a procedure for a technician student. To make this area of the training more efficient and realistic, a specific education environment was needed to provide realistic interactions with virtual models. This was achieved by using available techniques in Virtual Reality (VR) and Augmented Reality (AR) to be an auxiliary component in VMT to create a practical virtual education environment.
This work included doing a survey of different approaches and implementations of already existing solutions for using VR and AR for educational purposes. The result of this survey would reveal whether VR or AR is appropriate to use as an educational tool with the current technology, through comparisons showing the different advantages and disadvantages. Possible hardware recommendations will also be proposed and a prototype will be developed showing the possibility for technician training by using appropriate application development tools.
The application prototype would enable a user to execute a procedure on the aircraft by interacting with virtual models through the use of controllers or their hands. The kind of hardware or software that would be used were dictated by the survey. This application will not be integrated with VMT but will instead be a proof of concept to show that practical education in virtual or augmented reality could work as a complement for VMT. It will be developed as a standalone application but could perhaps in future VMT systems, be integrated as a separate lesson or configuration.
1.3
Objective
The objective of the project was to investigate the possibilities of providing a realistic practical procedure training concept for aircraft technicians. The investigation would cover VR and AR technology and be based on three different training situations:
a) A teacher should be able to demonstrate a procedure on a virtual engine, while the students stand around this object, all wearing head-mounted displays. The teacher should be able to rotate and turn the virtual/superimposed object. The students will in turn be able to observe the model in more detail.
b) An inspection procedure should be done by the student alone, e.g; be able to inspect the aircraft before take-off, such as controlling air-pressure in tires, refuelling the aircraft or troubleshooting cockpit errors. Different parts of the aircraft should also be able to react to interactions, such as opening a door when making a gesture etc.
c) The system should provide the tools to complete a maintenance task. The student should receive a set of instructions from the teacher and fulfil the requirements to complete the task. This needs to be done by using the instructions manual together with virtual tools such as a screwdriver etc.
1.4
Requirements
The project should have completed an in-depth survey of the current situation of VR and AR
technologies, with a documentation of advantages and disadvantages. From the survey, investigations should indicate which appropriate technology is suggested for a practical technician training
application.
A prototype application should be created to demonstrate the capabilities of the recommended technology concept and attempt to implement the three scenarios mentioned in the 1.3 Objective section.
1.5
Division of Labour
Doing the survey, the workload was shared equally where each of the authors, on their own, collected and summarised the relevant data which then was discussed. The major part of the work that had to be done for planning and implementing the prototype was accomplished in an agile way using pair programming. Occasionally the labour was separated where the authors shared the work equally with manipulation of 3D models and producing code for the prototype application.
2.1
Introduction
This survey used a qualitative research method to find appropriate data for the topic, by searching for related papers using backward reference searching. Keywords that were used are related to virtual maintenance training. Expansion of the research field was enabled by the backward reference search, to find new areas to analyse where fields such as medical and other areas were included.
This chapter starts with a brief introduction of the definition of Virtual Reality (VR) and Augmented Reality (AR) in Section 2.2 with a follow-up of the guidelines for implementing an application in both fields in Section 2.3. The survey is divided as such that in Section 2.5 and Section 2.6, virtual reality and augmented reality are introduced with an explanation of available hardware before presenting available implementations regarding practical virtual training. It is recommended to read through both of these sections in order to get a better view of the available technology in both fields before continuing on to the results of the survey. Section 2.7 is devoted to the results of the presented implementations and how virtual or augmented reality may perform with regards to different factors in practical procedure training. Finally, the conclusion of the study is presented in Section 2.8.
2.2
Clarification of Virtual and Augmented Reality
The following section will further describe the meaning of VR and AR and where it ends up in the Reality-Virtuality Continuum. The presented continuum (Figure 2.2.1) is a prescribed system to elucidate the meaning of the extremes that are included. Four classifications of environments and realities are shown, where mixed reality includes environments between the real and the virtual environments called AR and Augmented Virtuality (AV) [3].
The real environment is the natural environment one can see with their own eyes, consisting of only real objects, devoid of any artificial amplifications. VR is described as an environment consisting entirely of virtual objects, which can be observed through a display [3].
Between the real and the virtual environments, there are environments where the virtual objects are in coalescence with the real objects, offering the ability to, in real-time, use real objects as virtual
Figure 2.2.1: The RV Continuum Source: [4]
objects and vice versa. These environments are called AR and AV. There is no definite agreement on the definitions of these two environments, however, three factors have been defined to enable a decisive explanation of how one may differentiate between them. The three factors are divided as:
Extent of the World Knowledge (EWK) The recognition of the world and complementary objects Reproduction Fidelity (RF) The visual quality of the reproduced real and virtual objects
Extent of Presence Metaphor (EPM) The amount of presence or immersion the observer should feel From this information, it is possible to define AR as a virtual environment for the observer, allowing the possibility of displaying virtual objects in a real environment - making it closer to the real environment side in the Reality-Virtuality continuum graph. In contrast, Augmented Virtuality enables the
observation of objects or data, from the real environment, in a virtual environment setting. Depending on the weight of each previously mentioned factors, it either brings one closer to the real environment or the virtual environment [3].
While there are studies in specific areas of mixed reality for AV, this chapter will focus on VR and AR from a first-person perspective, exploring how the technologies work in industrial, educational and other similar contexts.
2.3
Guidelines for Training in VR and AR
Implementing a system in VR and AR requires several factors to be fulfilled to offer a realistic experience fit for technician training environments. Gavish et al. [5] has introduced four different guidelines to follow when designing VR or AR applications for maintenance training. By following the guidelines it is possible to increase the training efficiency, enhance skills acquisition and enable a development of a useful model of the task for the users. The four guidelines include:
• Proper integration of observational learning
• Enactive learning by combining physical and cognitive fidelity • Providing guidance aids
• Providing sufficient information about the task
Having these guidelines as well as guidelines from Oculus [6] and Leap Motion [7], it will provide a proper way of analysing the different related implementations and help with design decisions for the prototype.
2.4
Architecture
VR and AR implementations that are observed, have a similar architecture which takes input data and computes the data in a simulation, producing some output for the user. As an example, the user may input data by moving their hands or their position. Based on this input, the simulation will calculate the new state of the virtual entities and output a new representation of the virtual environment to the user. A diagrammatical overview of the inputs and outputs between a user and the devices (Figure 2.4.1) may give a better understanding of how a typical VR or AR system work.
Figure 2.4.1: Basic VR and AR System Overview
2.5
Virtual Reality
2.5.1 Introduction
There have been multiple technological developments and experiments in the recent years in the world of Virtual Reality (VR). New innovations are continuously emerging to bring the development forward. The VR technology has found its way into many areas of application, such as education, health care and maintenance training, to mention a few. Which specific technologies to use, ultimately depends on what task is at hand [8].
An important aspect of VR is the visual perception, enabled by different visualisation devices which may offer the user a visual experience of the virtual environment. Another aspect is user interaction, achieved by several devices today, providing different levels of immersion. For instance, using glove-or camera-based tracking techniques, the user is able to interact with a simulation by perfglove-orming hand gestures or remotely controlling a virtual hand [9].
Haptic feedback is also used to increase the immersion in a virtual reality setting. A way to achieve this is through built-in actuators in hand controllers that respond to certain events or through special vests which can produce body contact simulations by electrical impulses [10].
2.5.2 Hardware and Software
This section will present hardware and software used in the world of VR today. For hardware, the description of the most prevalent devices today in the market will be included, together with any tracking capabilities they may have. The software subsection will include the description of most used engines and development frameworks to create VR applications.
2.5.2.1 Device definitions
Wired Head-Mounted Display (HMD) in VR, available in the current market are dominated by Oculus Rift and HTC Vive. The HMD’s are often used in a spacious room, connected to an external workstation with camera trackers in the corners of the room. Positional tracking is enabled by the
cameras so that the user may walk around within artificial borders and experience a six degree of freedom positioning. The artificial borders aid the user from walking too far from the boundaries of the virtual environment [10].
Non-wired HMD’s are the second type of HMD’s. Google Cardboard, Google Daydream and Samsung Gear VR are all non-wired HMD’s and require a smartphone. Compared to wired devices, these devices (without additional external tracking devices) only provide three Degrees of Freedom (DOF) rotational tracking.
Tracking and Input Devices are essential input methods in VR since it enables the virtual recreation of a user’s physical actions. Technologies today enable tracking of the head, hands or the entire body of the user to represent their actions in a responsive way. Besides tracking there are other primordial non-tracked input devices, which will be mentioned towards the end of this subsection.
2.5.2.2 Hardware
Oculus Rift is the third iteration HMD from Oculus and was released 2016. It has a combined resolution of 2160x1200 with a refresh rate of 90 Hz and a Field of View (FOV) at 110° [10]. A built-in gyroscope, magnetometer and accelerometer in the Oculus Rift HMD enables tracking of the user’s head orientation in three DOF. Through a method called sensor fusion [11], which uses inputs from the three mentioned sensors, the user’s in-game head orientation (yaw, pitch and roll) is determined. Positional tracking of the user’s head is achieved through the Constellation system developed by Oculus. This system uses micro-LED on-board the HMD, which are tracked by at least one external IR-camera (Oculus Sensor), enabling optical tracking of the user in six DOF [6, 12]. In late 2016, Oculus released Oculus Touch, which is a native handheld input device of the Oculus Rift. In a virtual environment, Oculus Touch can represent the user’s hands accurately and in an intuitive way. Positional tracking is achieved by the built-in LED’s and utilisation of the same Constellation system, used for the Oculus Rift HMD. The Oculus Sensor can be used for both Oculus Rift and the Touch. Rotational tracking for the Oculus Touch is performed using a built-in Inertial Measurement Unit (IMU), similar to the HMD, enabling tracking of the user’s hands in six DOF. The device also has touch triggers, push buttons and an analogue stick enabling user interactions e.g. gesture recognition, user orientation, grabbing virtual objects etc. [13]. An image of the HMD and the Touch controllers can be seen in Figure 2.5.1.
Figure 2.5.1: Oculus Rift with Touch Source: [14, 15]
HTC Vive was released in 2016 and is the main competitor to Oculus Rift. Similar to Oculus Rift it has a combined resolution of 2160x1200 with a refresh rate of 90 Hz. The FOV is at 110° [16, 10]. The distinct difference to the Oculus Rift is the used tracking technique.
The HTC Vive HMD uses a technique for positional tracking called ’Lighthouse’, developed by Valve. This technique works similar to how it is used in maritime navigation, where ships navigate near the shore by counting the time between flashes from nearby lighthouses. Similar to that, Lighthouse requires the use of so called base stations (acting like lighthouses), which are accompanied with HTC Vive. The base stations, attached to each corner of the wall of an open room, tracks peripherals by performing an omnidirectional flash. Each base station then transmits alternating sweeps of horizontal and vertical IR-lasers of the room to trigger the photo-sensors on the peripherals. By comparing the timing of when the sensors are activated, the exact position and orientation can be calculated with high accuracy and low latency [17, 18].
Communication with the workstation is done through a Linkbox, which powers the HMD and connects through USB and HDMI. The Linkbox also communicates with the basestations through bluetooth to activate or deactivate the devices since they only requires power to operate.
Vive Tracker is an upcoming device which can be used to track anything of choice. It may be attached to a specific item or to the body and provides an easy way for integrating full body experience into the virtual environment. However, for tracking the entire body, it requires multiple Vive Trackers to be able to give an accurate estimation of the body in the virtual environment. It uses the same ’Lighthouse’ technique for positional tracking as the HTC Vive and the controllers [19].
The HTC Vive controllers was released together with the HTC Vive HMD in 2016 and includes 24 sensors, a multi-function trackpad, a dual-stage trigger and haptic feedback. It is used for interaction with the virtual environment using the hands but gives a visual representation of the controllers instead of the hands [20]. Figure 2.5.2 displays the Lighthouse basestations together with the HMD and the controllers.
A new implementation of a wireless HTC Vive HMD has been developed in cooperation with Intel, which will be released in 2018. This relieves the need of a workstation and opens up further possibilities for developing an efficient training system with this device.
Figure 2.5.2: HTC Vive with controllers and Lighthouse basestations Source: [21]
Samsung Gear VR was developed in cooperation with Oculus and Samsung. It is a mobile, tetherless HMD that requires a connected Samsung smartphone to run. Only novel Samsung smartphones (Galaxy S6 or later) are supported. The HMD provides the user with a horizontal FOV of 101° [22].
By utilising the built-in gyroscope/accelerometer and proximity sensor in the connected smartphone, Gear VR enables rotational tracking of the user’s head (three DOF).
The cooperation between Samsung and Oculus has also resulted in a VR controller, intended for the Gear VR. This controller uses the same tracking technique as the Oculus Rift through sensor fusion. Additionally, the Gear VR has a built-in touchpad on the side to interact with the virtual environment and communicates with the smartphone through bluetooth. Figure 2.5.3 shows the GearVR HMD.
Figure 2.5.3: Samsung Gear VR Source: [23]
Leap Motion is a pocket-sized, rectangular shaped and a USB-connective device that is marketed for being an accurate hand-tracking tool in a VR or desktop context (Figure 2.5.4). This device in conjunction with the associated API, can in real-time determine the position of a users hands and fingers in a Cartesian space. The Leap Motion controller has three IR-LED’s and two IR-cameras, providing stereo vision, which enables optical tracking in six DOF [24]. The captured data is of a grayscale stereo image, which is streamed through the USB-connector to the computer where the Orion Software Development Kit (SDK) takes over and performs mathematical calculations with tracking algorithms to reconstruct a 3D representation of what the device sees [25].
Figure 2.5.4: Plugged Leap Motion controller Source: [26]
Microsoft Kinect V1 enables tracking of the user’s entire body through an RGB-D camera (Figure 2.5.5a). The captured data is computed to a depth map through structured light which infers the body position through machine learning. The later released model, V2 (Figure 2.5.5b), uses a built-in Time-of-Flight camera, which emits light and detects the reflected light. The time between emission and detection is then measured to determine the position of predefined objects [27].
(a) V1 (b) V2
Figure 2.5.5: Microsoft Kinect V1 & V2 Source: [28, 29]
2.5.2.3 Software and Development Frameworks
There are several software applications which are widely used when designing for VR. The prominent game engines today are Unity [30] and Unreal Engine 4 [31], which are cross-platform engines used for game development. Unity and Unreal Engine supports plugins and packages to be imported to make it easy to create a VR application for different HMD’s.
Software Development Kits (SDKs) Short description of a few selected SDKs and used APIs. • OpenVR [32] is developed and controlled by Valve to support communication with HTC Vive
and other virtual reality headset devices to run the same VR application. Essentially OpenVR consists of two APIs and acts as a mediator between other software and other devices. The First API, OpenVR API, is to allow communication between application and a software which communicates with the drivers of an HMD. The second API, OpenVR Device API, allows communication between the HMD software, such as SteamVR, and the drivers of the HMD. • Open Source Virtual Reality (OSVR) [33] is developed by Razer and Sensics and is similar to
OpenVR to enable different devices to work with the same game but with the main difference being that it is open source.
• Oculus SDK [34] is used to enable installation and communication of drivers for using Oculus Rift. Oculus have one API which communicates with Oculus Runtime, which in turn directly communicates with the HMD and the tracking system.
• SteamVR [35] is a closed-course software by Valve and is used to enable communication with drivers of different HMDs, but primarily for HTC Vive, by using the OpenVR API. Communication between the HMD device, HTC Vive, and SteamVR does not require the use of the OpenVR Device API however, since the drivers for the HMD and tracking devices already are implemented into SteamVR.
The mentioned SDKs allows flexible integrability with different software such as Unity and Unreal Engine.
2.5.3 Related Studies and Implementations
The following section will describe several studies in the field of VR in the recent years and what conclusions they came to with their implementations.
Before presenting the different studies a clarification of mentioned platforms may be useful. Often when talking about virtual systems there are two types that are mentioned: Commercial off-the-shelf (COTS) and Cave Automatic Virtual Environment (CAVE). COTS platforms often describe a system that is able to be purchased commercially for a low cost, and often used by only one user. CAVE platforms is a projection-based system where a virtual environment is projected on surrounding walls by projectors, where several users may experience the same virtual environment together. These acronyms will be used frequently throughout the text but focus will mostly be on COTS platforms.
2.5.3.1 Maintenance and Inspection Operations
A practical training environment called Virtual Teaching and Training System (VTTS) was developed by Quan et al. [36], to teach aircraft maintenance. VTTS is a computer based on VR technology and real-time education, similar to a CAVE. This particular VTTS uses a projective display and an LCD display with workstations and an aircraft simulator. The VTTS configuration has the possibility to realise numerous teaching contents and subjects. Similar to Virtual Maintenance Trainer (VMT), it possess the ability to implement a system with various routine operations, emergency situations and examinations. It is also possible to render a virtual scene similar to the real world operations and work conditions. By using this system the authors have come to the conclusion that this kind of application for maintenance training, reduces man-made errors, aircraft damage and accidents. VTTS enables to use virtual objects instead of real aviation equipment which lowers the cost, while essentially maintaining the same study efficiency. This type of learning environment will also increase the study efficiency compared to traditional methods e.g. text and multimedia.
For maintenance, Chang et al. [37] has suggested a solution for training in a virtual environment. In their implementation of a practical procedure training system for substation maintenance, they present a concept closely similar to VMT. Their system simulates models of equipment, failures that may occur and procedures to be completed by the student. A training and examination management tool is built into the system to help the teacher to keep track of each student. A mouse and keyboard are used as input devices for the platform. The platform communicates with a server, running on a Linux operating system, where all the computing is done. The scenario data (models, faults or tests) used by the server is stored in a database. The clients (teachers and students) are using workstations with the Windows operating system.
The system provides many tools for the teacher to monitor training procedures and manage training archives and teaching plans. Compared to the teachers, students have an interface with limited access, but with the modules for training or examination of a procedure. A virtual toolbox (pliers, screwdrivers, duct-tape etc.) is also provided during the simulation which enables the student to use the
appropriate tool during a certain procedure. In this current implementation, the information is relayed on a monitor but may very well be to an HMD instead.
A recent study of an immersive VR maintenance training conducted by McNamara et al. [38] used Oculus Rift (Development Kit 2) and Kinect V2, and the game engine Unity. The study focuses on tracking a human body by using the Kinect V2 and examining the positional accuracy and system delay during use. Positional accuracy offers realistic representations of corresponding user actions. System delay is the time between a physical user action and the time for the action to be displayed on the HMD. It is an important factor to avoid simulator sickness to prevent the user from becoming dizzy [39]. Understanding the problems with system delay may aid in the understanding of the limitations of a VR system.
Ten participants interacted with a virtual model depicting an Arresting Gear engine (an engine to decelerate an aircraft when it lands). Interactions included walking around the model and to try different body positions to test the capabilities of the Kinect V2.
Results showed that positional accuracy of the Kinect V2 performed well, for a realistic movement in the virtual environment, on tracking a single user with the sensor in front. However specific body positions such as crouching, turning away from the sensor or blocking the view of the sensor would cause the Kinect to lose its tracking capabilities. Furthermore, it was unable to detect small gestures, which made it improper for the use of training where high accuracy was required.
On the other hand, a considerable amount of the system delay was caused, between a performed action by the user and the rendering of the action on the HMD. Kinect V2 has a data transition rate at 30 Hz compared to the HMD at 75 Hz. As a consequence, user-performed actions, with this type of system, may take longer to update and cause disorientation which may cause simulator-sickness [6]. Moreover, a contributing factor is the stereoscopic rendering on the Oculus Dk2, which requires the computer to render a scene twice per frame, causing an excessive demand on the CPU and GPU. To alleviate this, the authors suggests using a computer with better specifications, with emphasis on better performing CPU, GPU and RAM. The time spent to prepare the image for the HMD would then be reduced significantly. Additionally, the system performed better on a computer running the operating system Windows 8.1 and a dedicated graphics card, due to the different compatibilities with the drivers between the hardware at the time.
This type of training system may be suitable for when the user will need to become familiar with an equipment in a virtual environment by walking around it, or by interacting with it by using large distinct motions.
Continuing the topic of maintenance Borsci et al. [40] made a study of a train car service procedure with an implemented virtual-based training tool called LAugmented Reality (AR)TE-VBT. It is based on the HoloVis game engine (InMo) and enables users to visualise and interact with CAD models in different devices. The devices in this study compared between a CAVE platform, Oculus Rift and a tabletop display.
In the CAVE system, there were four walls with each wall having two projectors which projected the virtual environment. Nine workstations were used to visualise the virtual world. The user had to use polarised glasses with built-in trackers for interaction. The tabletop display was a device named zSpace, a 24 inch LCD running at 120Hz. A stylus device was used as a controller and similar to the CAVE, polarised glasses were used as a tracking device. In the case of Oculus Rift, it was used together with desktop controllers and joysticks. 60 participants were divided into three groups and were given the task to do random maintenance operations on parts of a train car. One group would use the CAVE
system, second group the tabletop display and the third group the Oculus Rift.
The conclusion of the study presents all three of the devices as good learning experiences with the largest difference coming down to price. While the CAVE is a largely expensive alternative (>50k €), the tabletop display has a more reasonable price (< 6k €) and the Oculus Rift with a workstation at the cheapest price point (< 2k €). The authors also mention the importance of acceptance of the end-users when using VR systems for learning since the success of a system like this depends on how good the user believes the virtual training is comparable to the real world.
Cabral et al. [41] presents a simulator system for maintenance of power grids distribution lines which follows most of the guidelines in section 2.3 well. Oculus Rift is used to visualising the environment. Interaction with the environment is enabled by tracking real tools, which receives a virtual depiction, to give the user a realistic feeling during learning. The tracking system is based on an extremely accurate infrared camera system called OptiTrack. The body of the student is also tracked and is depicted as an avatar in the virtual environment allowing the teacher to assess if the maintenance is done in a correct way. Both the teacher and the student has their own program interface.
The teacher can through their perspective control of the training allow different situations to occur to allow different challenges on the maintenance lessons. Challenges may include doing the maintenance at different weather situations, structures being on fire, experiencing electrical arcs and short-circuits and other obstacles such as trees, cars etc. From that, the teacher is also able to see how the student reacts to and solves the problem.
The student performing the maintenance training uses the Oculus Rift to see the virtual world and do the operations required. Other students may, through an external screen, see what the training student is doing, to learn from each others mistakes.
DAssault Systèmes is a worldwide aerospace company, producing both passenger and fighter jets. Recently they presented a virtual simulator developed for virtual maintenance training for their Falcon jet, by using their own 3DExperience platform. It is designed to be an inexpensive alternative for aircraft maintenance training to their already implemented CAVE system. It uses CATIA CAD data to provide an immersive training experience for engineers and technicians.
The system uses Oculus Rift HMD together with a front-attached Leap Motion, to provide hand tracking capabilities, enabling real-time interaction for the users. Constellation tracking is also used for the Oculus Rift to provide six DOF virtual experience. Multiple instructors or trainees are allowed in one session, each represented by an avatar in the virtual world. The Leap Motion tracker also allows the user to see their own hand and the hands of others, useful when the teacher needs to point at a certain object, or when a trainee points at an object while asking questions about it.
Interaction in the world is enabled by a menu, spawned by facing your left-hand palm towards the Leap Motion controller. From the menu, mentioning only a few things, the user is able to (i) set transparency of the virtual model, (ii) select different data to show, (iii) recolour models to make some parts more distinct from others, (iv) bring the trainees to the instructor position, (v) spawn a pointer steered by the HMD motion to point or select an object further away from the hand. Movement in the virtual world is done by the arrow-buttons on a traditional keyboard [42].
2.5.3.2 Educational Benefits
A proposal for a virtual education environment was put forward by Crespo et al. [43] by testing the benefits of a VR simulator. The simulator allows engineering students to perform an interactive
operation on an industrial robot. Used hardware was Oculus Rift HMD and Razer Hydra Joysticks. The application was designed using the game engine Unity. Oculus SDK was used for configuring the Oculus Rift while Sixense SDK was used to allow programming for six DOF motion tracking of position and orientation with the Razer Hydra controllers. Students that tested the system executed a number of interacting tasks of the virtual robot with the controllers, by moving the robot around in the virtual environment or making it perform certain tasks, such as moving items.
It was concluded that this system has promising possibilities of becoming a reliable engineering learning tool. The used software and hardware enable the system to be highly flexible by working with different platforms, operating systems and controllers. Furthermore, it allows for mistakes to be made and makes it possible for the student to experience what will happen if something goes wrong. Through the immersive interaction, it enabled a swift learning of specific routines’ and reduced the need to use real hardware.
Alhalabi [44] also made an educational study with 48 students, by comparing different VR systems. The VR systems were provided from Worldviz, a worldwide provider of VR systems. It included The Corner Cave System (CCS, a projector based VR system) and an Oculus Rift DK2. The CCS system requires the users to wear special shutter glasses equipped with precision position tracking ‘eyes’ to see and interact with the virtual world through the projected screens. The students were divided into groups with (1) no VR, (2) HMD with tracking, (3) HMD without tracking and (4) CCS with tracking. The tracking system was set up with eight cameras for user tracking in a spacious room. The students were asked to study different engineering materials and learn about new concepts. The results showed that the learning efficiency increased significantly by using VR systems. The CCS system would be an appropriate alternative for multiple users at a time, however, the HMD system is more inexpensive, with the additional advantage of providing a more immersive environment for the user.
Additional virtual training implementations has included Oculus Rift together with Leap Motion (and Kinect V2) to provide an effective way of training for situations in construction and fire emergencies. [45] used Unreal Engine 4 in their implementation of an inexpensive virtual environment for an engineering and construction setting. [46] used OpenGL to implement a web-based training system for fire wardens to offer a safer alternative for training the cognitive decisions during fire in a building. Both of the projects used Industry Foundation Classes (IFC) to define CAD-graphic data, related to architecture and construction, as 3D objects.
2.5.3.3 Input Practices
In terms of user control, Khundam [47] implemented a system using the Oculus Rift DK2 together with Leap Motion to provide a way of controlling the movement inside a virtual environment with hand palm gestures. Using Unity, an application was created to provide a virtual space for the user to move in freely. 11 participants tested the system with two different scenes for each of the input methods. In the experiment, the subjects’ mission was to move around the virtual world in a fixed order. The evaluation showed that using the Leap Motion controller the users were able to complete the missions faster with the palm control gesture, enabling finer control of the movement speed, than the gamepad. However, due to the smaller FOV of Leap Motion, the users had to face the HMD towards their hands when performing the specific gestures.
limitations of current available input methods and to expensive wearable devices. In particular, the FOV limitations of Leap Motion was mentioned to be less effective. The SmartVR platform was used together with the Microsoft Band 2 wristband, a VR workstation and Oculus Rift DK2. A Bluetooth transceiver was used for communication between the workstation and the wristband. Unity was used to develop the application of a virtual store shop, where the user would be able to pick two modes, switching between them by performing a distinct gesture. First mode is a selection mode, which lets users select virtual items, and a navigation mode, which enables movement in the virtual environment. Second mode is a navigation mode, where the user may perform a swiping gesture to move around, moving towards the direction of the swiped action. In the selection mode, the user is able to select the virtual item with a spawned cursor representing the hand, while the camera is static. A menu was provided to manipulate the object, such as changing colour or size. Using a wearable device in this way, such as the Microsoft Band 2, enables the hands to be tracked regardless of the orientation of the HMD and allows the user to be less limited when performing the gestures.
2.6
Augmented Reality
2.6.1 Introduction
Augmented Reality (AR) is a technique which augments what the user can see, feel and hear. Usually, an AR application involves visualisation of virtual objects superimposed on the real world, to mainly supplement the real world with useful information [49, 50].
The occurrence of virtual objects, displayed in an AR application, together with their pose and position, can be based on camera-recorded fiducial markers which operate as positional reference points [51, 50]. A marker may consist of a picture or pattern, which is recognisable by the particular AR application, such as QR-codes (Quick Response Codes), BIDI codes (BIDimensional) or bar codes [52].
A virtual object’s appearance can also be based on marker-less tracking. For instance, external stationary cameras can be used for recognition and tracking of real-world objects. Based on detected objects in the video recording, the occurrence, position and pose of a virtual object can be determined [51]. The marker-less positioning of virtual objects can also be applied through the usage of
accelerometers, GPS-technique, compasses etc [50].
The possibility for user interactions, in an AR application, can be implemented in similar ways as with Virtual Reality (VR). Such as gesture tracking, voice commands, hand controllers, touch devices etc. [53].
2.6.2 Hardware and Software
In this section, modern AR-related hardware and different development frameworks will be presented. 2.6.2.1 Device Definitions
Head-Mounted Display (HMD) intended for AR, can display the real world, augmented with additional virtual objects for the user. There are both portable and tethered models for data transfer and/or power. An HMD can be classified as either an optical-see-through or video-see-through device [53].
2.6.2.2 Hardware
Microsoft HoloLens is an optical-see-through device with a visor, on which virtual objects may be projected in a maximum resolution of 1268 × 720 pixels [54]. By utilising four built-in spatial-mapping cameras and a depth camera, a 3D-model of the surrounding environment is created. The geometry of this 3D-model is used to determine the position and pose of virtual objects (also called holograms in HoloLens context) and for positional head tracking. Rotational head tracking can be achieved by using the built-in gyroscope, magnetometer and accelerometer. Both of these tracking techniques enables six Degrees of Freedom (DOF) movement. The horizontal Field of View (FOV), in which the 3D-objects may be projected, is 30° wide. Figure 2.6.1 depicts the HoloLens HMD.
Figure 2.6.1: Microsoft HoloLens Source: [55]
User interactions can be performed by looking in a particular direction while performing a certain finger gesture (recorded by the built-in camera) or through voice commands in Cortana, Microsoft Windows’ voice-activated assistant [56, 57]. Microsoft’s HoloLens is completely tetherless and probably the most known of currently few available AR HMDs. HoloLens can be bought in a
development kit, fit for an individual developer. For enterprises a similar kit can be purchased, but with a warranty and features like higher security etc. [58]. HoloToolkit, which is a set of APIs towards the HoloLens utilities, enables development of applications. These APIs can be integrated into Unity.
Meta 2 is similar to HoloLens; an optical-see-through device with common properties and a maximum resolution of 2560 × 1440 pixels distributed over a 90° wide FOV [59]. One difference is that Meta 2 is not tetherless when running and requires a connection to a workstation with Windows 8 or 10. The real world environment features are captured using a built-in computer-vision-camera. This camera enables virtual object positioning through a modified simultaneous localisation and mapping (SLAM) algorithm. The movement of the wearer is tracked by using a built-in inertial measurement unit. Totally this provides six DOF [60]. Figure 2.6.2 shows the Meta 2 HMD with the connection cable.
Figure 2.6.2: Meta Company’s Meta 2 Source: [61]
Google Glass is an optical-see-through HMD that appears like a regular pair of glasses which can provide an image of virtual objects, displayed on the built-in 640 × 360 pixel prism-projector screen just above the user’s natural line of sight (Figure 2.6.3) [62]. To the user, the built-in screen appears comparable to a 25-inch high definition screen viewed from a distance of 2.4 meters [63]. Similar to Google Cardboard, Google Glass captures video in the direction the user is facing with a built-in camera. The recorded video can then be used for object recognition, video conversations etc. User interactions can be performed by voice commands or through a built-in touchpad located on the side of the spectacle frame [53]. Using the Android API also enables rotational head tracking by utilising a built-in gyroscope, accelerometer and magnetometer [64].
Figure 2.6.3: Google Glass Source: [65]
Google Cardboard is a primitive video-see-through HMD (Figure 2.6.4) that requires an attached smartphone to be used for AR. Video of the real world can be captured by the camera of the
smartphone and displayed on the screen in real-time, augmented with additional virtual objects. The experienced horizontal FOV can alternate between 90° to 120°, depending on the attached smartphone [53].
Figure 2.6.4: Google Cardboard Source: [66]
Tablets and smartphones also offer the possibility to be used for AR applications. The user observes the screen which displays a video of the environment; captured by the built-in camera [67]. A tablet or smartphone device itself can be utilised for user interactions, which may be performed using the touchscreen or the microphone by voice commands, depending on the used AR software [53].
The Leap Motion controller can, similar to VR, be used for achieving natural user interactions in AR applications [24, 9].
2.6.2.3 Software and Development Frameworks
Vuforia is a proprietary AR Software Development Kit (SDK) that uses image recognition through computer vision technology for creating virtual objects. A Vuforia mobile app utilises the built-in camera to capture the environment. It can recognise a 2D-image (normally a photo, drawing, label etc.) or objects like cylinders and cuboids by reaching a cloud database, where predefined 2D- or 3D-data for recognition is stored. The program uses the 3D-data as a reference point to determine the position and pose of the virtual object that will be imposed on the real environment. Vuforia supports native development for iOS and Android but also enables integration with the Unity editor for targeting both platforms [50].
ARToolKit is a multi-platform, open-source (LGPLv3) library for development of marker-based AR applications. Similar to Vuforia, ARToolKit uses image recognition for determining occurrence, position and pose of virtual objects, but can only recognise 2D images, not 3D-objects. The library allows for integration into the Unity editor, providing a convenient way of developing AR applications for any platform [50].
D’Fusion is a proprietary, multi-platform AR engine established by Total Immersion. The engine supports both marker-less and marker-based tracking. One of the strengths is that the engine has well-developed human face recognition and tracking. To develop D’Fusion-based applications, the GUI-based tool D’Fusion Studio is used [68, 50].
2.6.3 Related Studies and Implementations
2.6.3.1 Usages in Assembly, Maintenance and Inspection Work
Aircraft maintenance training activities are closely related to maintenance and inspection work tasks in e g. the automotive or aviation industry. Ramakrishna et al. [53] propose a framework for inspection work using AR-technique. The framework consists of an HMD or tablet that streams video images to a back-end server, using UDP-connection. The server runs the images through a Convolutional Neural Network (CNN) to detect certain objects for inspection. When an object is recognised by the server, modelling data is sent to the HMD/tablet providing a guiding graphical overlay for the user.
An experimental study including this framework was performed, by the same authors, where three different AR platforms were used: Google Glass, Google Cardboard and an Android tablet. The experiment was carried through on 20 subjects (engineers and research staff from an industry research
lab). Their mission was to perform an inspection task on a 3D-printer. To create an inspection guiding graphic overlay, the user initially scanned a QR code, placed on the printer, which provided user manual data for the specific printer model. The user then had to inspect different parts of the printer and answer a set of questions through platform-specific interactions. The built-in touch-pad was used for Google Glass, voice commands for Google Cardboard and the touch screen for the Android tablet [69, 53].
Observations during this experiment show that stress was reduced when using AR with graphic overlays containing the needed information compared to looking through extensive paper- or PDF-manuals for completing complex tasks. The Android tablet performed best of the three platforms since the users found it the most intuitive. Google Cardboard, on the other hand, caused eye-sickness, gave the user a claustrophobic feeling and the low graphics resolution aggravated the user’s ability to read printed text on the printer parts. A low FOV gives the user a cramped vision which can reduce the awareness and be dangerous in rough industrial environments. The voice command interactions also work poorly in crowded/noisy areas. The Google Glass was hard to fit over regular safety glasses and the swipe patterns were difficult to remember. It is a more expensive platform than the others in this experiment but could be convenient for users in situations where both hands are needed for work tasks. In summary, the tablet was the best candidate in this experiment due to simplicity and measured inspection turnaround time [53].
Industrial assembly work, like previously mentioned inspection work, can be related to maintenance training. Loch et al. [70] analyse a training tool that provides step-by-step guidance for assembly tasks in an AR-setting. It consists of a computer screen, placed in front of the user, which displays a real-time camera recording of the workspace for the assembly task with an additional virtual graphic overlay for assembly guidance. The camera is mounted above the workspace, pointing downwards. A study was conducted to evaluate and compare AR-based to video-based guidance for assembly tasks. 17 undergraduate students from social science and technical disciplines, without any previous experience in AR-guidance, participated in the study. Before the experiment, preparation training was performed to become more familiar with the AR-guidance system. The experiment involved a practical part, where the participants were to assemble two different models using Lego blocks.
Measurements in the study showed that the average time to complete an assembly task was longer for the video-based instructions compared to the AR-guidance and the average number of errors per assembly task was over ten times(!) higher. Evidently, using AR-guidance increased the accuracy and task performance due to the lower number of occurred errors, compared to using the video-based instructions. The author draws the conclusion that, for assembly tasks, experts in a certain area are the most effective guides, but that AR-based solutions work better in several important aspects than video-based instructions or printed manuals [70].
Boeing Research and Technology (BR&T) performed several comprehensive studies that explore the usage of AR solutions in satellite and aircraft part manufacturing. For developing virtual work scenarios, BR&T used D’Fusion Studio. The choice of visualisation AR-platform fell upon high-end ruggedized PC tablets, snapped into hands-free holders to enable work with both hands simultaneously. A set of eight external Vicon RGB cameras mounted to the inner-roof of a dedicated room, were used for accurate marker-less tracking [50]. For passing the captured camera input to the D’Fusion application, a custom-made Lua script was developed. Microsoft Kinect was used for low-resolution 3D modelling of objects for tracking and recognition, but also allows existing CAD files to be utilised.
The system was tested by simulating an assemblage of an aircraft wing.
According to the BR&T studies, utilising AR techniques in ideal engineered environments can-provide more effective manufacturing processes in terms of task completion time and failure rate. Although, more development will be needed for the technology to be extensively applicable in mainline manufacturing industries. BR&T proposed an AR solution that an average manufacturing work
site today can suffer from camera line of sight problems when working in narrow spaces or behind occluding vehicle parts. Due to this problem, external cameras or sensors should not be preferred as the tracking solution. For display, tablet PCs works but does not give the user a deep level of immersion. Other more immersive display technologies, that can provide a wide FOV and high safety, would be preferred in manufacturing processes [51].
Caterpillar Inc. is the world’s leading mining equipment manufacturer at the time of this report. During 2015 Caterpillar, in cooperation with ScopeAR, developed an AR application for maintenance and inspection work assistance, using marker-less tracking; that can run on a smartphone, tablet or AR glasses [71]. The application provides a graphic overlay which guides the user through a certain maintenance or inspection task. The overlay consists of arrows, floating text and 3D models of machine parts involved in the task. Work instructions are given step-by-step and the application can evaluate if the user has completed a certain step in a correct way [72, 73, 74].
2.6.3.2 Educational Benefits
Ibáñez et al. [49] conducted an empirical study, including 40 ninth grade students, for the purpose of exploring student behaviours, learning effectiveness and motivation while using a marker-based AR-simulation tool called AR-SaBEr. The tool can be used for learning basic electrical principles and developed by using Vuforia and Unity [50]. The experiment used Android tablets and was divided into three topics: electric current, voltage and resistance. Each topic had one reading activity and two to three experimental activities where instructions were given step-by-step in a highly guided way called scaffolding. The students were given the task to build basic virtual electrical circuits from components like batteries, fans and switches which were visualised on marked physical blocks. By moving the blocks into the right place, while observing the augmented scene on the tablet screen, a circuit could be constructed. AR-SaBEr and similar tools can be useful as interactive learning environments and are especially important for achieving desirable results in given tasks. To help students keep the focus on their goal, it is necessary to provide free experimentation but also additional scaffolding for completing tasks.
Using the AR-SaBEr, Ibáñez et al. [67] also made a case study, investigating the attitude of learners towards education with AR as a technical device. The subjects in this study were physics undergraduate students with no previous experience in usage of AR. Their task was to understand and solve an
electromagnetic problem which included a visual 3D cube with a force to be determined and a simulation of the trajectory of a particle.
Two single-choice questions with four alternatives were then answered by the subjects. Before the experiments, the subjects were given some AR preparation training with a smartphone application which lets the user determine the position of a point in a 3D-space. The result from the study concluded that AR helped students to visualise the electromagnetic problem. The subjects also enjoyed it, which positively affects the attitude towards AR-learning that in turn can motivate students to perform learning activities.
2.6.3.3 Object Tracking Investigations
User interactions is a vital part of an educational AR application. Natural user interactions can bring immersion to the application and depth sensors can be used for achieving this. For detecting body or hand gestures with depth sensors, Microsoft Kinect and Leap Motion Controller are today’s most used devices.
Kim et al. [9] analyses a set of approaches for user interactions in a partially marker-based AR application. The implementation was done with the Vuforia SDK and Unity [50]. 20 persons, whereof 11 who had previous experience of augmented reality, participated in the experiment. A Samsung Galaxy Tab S 10.5 was used for running the application and a Leap Motion Controller to capture user interactions. Two different approaches were analysed without using the Leap Motion Controller. The first, vision-based approach records the user hands with the built-in camera, where the program interpreted the different hand gestures. The second approach uses the screen of the Android device for touch-based interactions. These cases were compared to an immersive approach where the Leap Motion Controller is used for tracking the hands of the user and virtually recreated as 3D hands in the application for natural user interactions.
Users performed three experimental tasks by using the mentioned interaction approaches: The first task consisted of transforming a virtual 3D bunny into a goal position and pose. The second task involved picking up and hanging a virtual 3D torus on a pin. The last task consisted of a simple assemblage of virtual engine parts; the user was to put a piston into a conrod, insert a lock pin into a hole and then to screw the pin into the hole.
The mean value of measured completion time for the first task was 45% longer for the touch and vision-based methods compared to using the immersive method. No measurable differences were observed while measuring failure rate.The second task did not produce any measurable differences in performance between the interaction methods, but a wide-angle lens attached to the camera helped to grant vision of the fiducial markers while performing the task. In the final task, the average measured completion time for the touch-based, vision-based and immersive methods were 225, 190 and 120 seconds and the mean failure rate were 55%, 40%, and 5% respectively.
The results show that the immersive implementation outperforms touch and vision-based methods. By using this or a similar method, the user can easily, in a natural and intuitive way, manipulate virtual 3D objects in six DOF.
Garon et al. [75] investigates the HoloLens and concludes that there are limitations in the device that can not be neglected in situations where high precision is important. It was found that shortage of high-quality depth data is one limitation that can result in poor tracking and thus, bad positioning of virtual objects. When developing for the HoloLens, a programmer is only permitted to use Microsoft’s own API and cannot reach the raw data from the built-in sensors. This prevents implementation of other, better-purposed tracking functions and algorithms. To bypass this restriction, Garon et. al. mounted an external Intel RealSense RGBD (Red-Green-Blue-Depth) camera on a HoloLens. The camera captured video that was processed on a connected stick PC and transferred via Wi-Fi, to the HoloLens. The mentioned processing on the stick PC consists of a program with a sophisticated object detection implementation that detects and estimates the pose and position of a scanned object. When testing this custom-built system, a great improvement in detection of objects measuring close to 20 × 20 × 20 cm could be observed.
2.7
Result
This section will contain comparisons between different system depending on the objectives. While Virtual Reality (VR) and Augmented Reality (AR) both can satisfy similar needs, there are both advantages and disadvantages with using techniques for both concepts.
From the given requirements, separate areas were extracted for evaluation. Since the requirements state that the teacher should be able to demonstrate a virtual model together with a group of students, using AR or VR technology to, one requirement would be to have a system which supports multiple users simultaneously. Both a teacher and a student should be able to interact with the virtual environment. Therefore making the manipulation of virtual models necessary, requiring support for user interactions. Being able to accurately observe a highly detailed virtual model, adequate graphics and performance are desirable for both VR and AR [6]. A high performing system for this matter should deliver low system delay, sufficiently high frame rate and stability in the placement of virtual objects [76]. These topics will present advantages and disadvantages of various VR and AR systems for the particular area.
2.7.1 Multi-user Capabilities
Multi-user in VR
There have been many VR systems utilising the Cave Automatic Virtual Environment (CAVE) platform. The main advantage is inherited by its multi-user capabilities and by offering an immersive and close-to-reality environment. Compared to VR Commercial off-the-shelf (COTS) systems, this platform can enable a wider Field of View (FOV) and allow multiple users to physically stand in the CAVE-area to feel present within the virtual environment. However, it only allows one user to be tracked at one time to control the simulation.
From the related studies and implementations, there is only one implementation which uses a COTS platform with added multi-user capabilities. DAssault Systèmes, used their 3DExperience platform, to create a multi-user virtual environment where an instructor guides trainees around a CAD-based 3D-model of a Falcon jet aircraft. Since this system is using Oculus Rift which is a tethered Head-Mounted Display (HMD), each user must have their own workstation. They use a network system, where the trainees join a session guided by an instructor. Since this is a novel implementation, there are no studies regarding how well the multi-user usability of this system works and if there are any possible drawbacks. However, there are videos [77, 78] which may give an observation of how the system works in general.
Though there are currently a limited amount of presented solutions for multi-user environments for VR, there are multiple Software Development Kit (SDK)’s and third-party applications being continuously developed. Using Unity 5 also easily enables the creation of multiplayer environments through using their networking API. Most SDK’s include some form of networking capabilities and are too many to mention all of them but most prominent ones are the built-in HLAPI of Unity and the Photon Engine [79] among others. The support is mainly for the more popular HMDs such as Oculus Rift and HTC Vive. The implementations are similar to how standard multiplayer networking works, for instance peer-to-peer, client-server, client-side prediction etc. [80].
