2009:108
M A S T E R ' S T H E S I S
Implementation and evaluation of a predictive mixed reality interface with an exocentric motion control
paradigm for time delayed teleoperation
Lakshminarasimhan Srinivasan
Luleå University of Technology Master Thesis, Continuation Courses
Space Science and Technology
Department of Space Science, Kiruna
Abstract
Teleoperation of mobile rovers used as explorers in outer space has one major draw- back - time delay. The requirement for human aid in this control chain stems from the basic need to utilize human cognition. The drawback with the early interfaces, particularly during teleoperation, was the huge workload placed on the user. With emphasis on easing workload in order to increase the efficiency of a human-robot team, newer techniques have been developed for visualization and control at the human end. A new paradigm for time-delayed teleoperation is being implemented and evaluated in this thesis - The use of predictive mixed reality interface with exo- centric motion control. Evaluation of this paradigm consists of testing three major areas - Navigation, Perception and Manipulation. System performance is measured in terms of navigation times and number of collisions. Operator performance is also measured using Situational Awareness Global Assessment Technique (SAGAT). The new paradigm is tested on users with different levels of prior experience. The major contributions of this work include
∙ A new motion control paradigm for teleoperation
∙ Experimental results when teleoperating with and without the new motion control paradigm
∙ A platform to test future augmented interfaces
∙ Connection to Helsinki’s GIMNET network and to J2B2 mobile rover
Keywords: Teleoperation, Time delayed teleoperation, Exocentric motion control,
Stereo projection, User training.
Contents
1. Introduction 1
1.1. Motivation . . . . 1
1.2. Description of the Task . . . . 2
1.3. Thesis Outline . . . . 2
2. Teleoperation 4 2.1. Human Machine (Robot) Interaction . . . . 4
2.1.1. Types of Interaction . . . . 5
2.1.2. Information Exchange . . . . 6
2.2. Teleoperation . . . . 7
2.2.1. Situation Awareness . . . . 7
2.2.2. Challenges with teleoperation . . . . 8
2.3. Graphic Displays . . . 18
2.3.1. Stereoscopic Displays . . . 19
2.3.2. The CAVE System . . . 20
2.3.3. The CAVE system at the University of W¨ urzburg . . . 21
3. Design of components and Implementation 22 3.1. Design of Graphical User Interface . . . 22
3.1.1. Scan Matching . . . 23
3.2. Egocentric and Exocentric Joystick navigation . . . 24
3.2.1. Navigation stages . . . 26
3.2.2. Egocentric Navigation . . . 26
3.2.3. Exocentric Navigation . . . 29
3.3. GIMnet . . . 33
3.4. Predictive Display . . . 35
3.4.1. The Open Dynamics Engine simulator . . . 35
3.5. Schematic of the experimental setup . . . 36
Contents
3.6. Implemented GUI . . . 37
4. Design of Experiments 42 4.1. Testing the connection to Helsinki’s GIMnet . . . 42
4.2. Comparing exocentric and egocentric motion control paradigms . . . 43
4.2.1. Test 1 . . . 43
4.2.2. Test 2 . . . 43
4.3. Testing the platform to develop future augmented interfaces . . . 45
4.3.1. Situation awareness testing . . . 45
5. Experimental Results and Analysis 48 5.1. Preliminary tests with J2B2 . . . 48
5.2. Navigation control paradigm testing . . . 50
5.2.1. Egocentric Joystick results analysis . . . 52
5.2.2. Exocentric Joystick results analysis . . . 53
5.2.3. Overload period math stimulant . . . 56
5.2.4. SAGAT . . . 59
5.2.5. Tests with time delay . . . 60
5.2.6. User interface questionnaire . . . 62
5.2.7. Control algorithm preference . . . 62
6. Conclusions and future work 64 6.1. Concluding remarks . . . 64
6.2. Future Work . . . 65
A. Experimental results in tabular format 66 B. ICP Solution 74 C. Physics behind polarisation 77 C.1. The Electromagnetic PlaneWave . . . 77
C.2. Polarised Light . . . 78
C.3. Polarising Filters . . . 79
Symbols and Abbreviations 84
Bibliography 84
Chapter 1
Introduction
1.1. Motivation
The field of robotics is gradually becoming more diverse, moving away from the traditional stronghold of industrial manipulators. The shift in emphasis to mobile robots is understandable with a look at the new avenues of interest. Examples in- clude mines where human deployment is prohibited on safety grounds, search and rescue operations from under rubble or fire [1], planetary exploration [2], and more recently, for exploration and reconnaissance [3]. More and more stress is being placed on mobile robots that can perform tasks independently. Such autonomy is quite realizable for repetitive tasks in known environmental conditions of operation.
The requirement for a human aid in this control chain stems from a basic need to utilize human cognition. Despite progress in perception technologies and emphasis on artificial intelligence and learning capabilities for robots, a human-machine in- terface can never be completely dispensed with. This interface cannot be related to the level of autonomy for the simple reason that the remote vehicle has to convey its progress and status back to the human operator. An increase in autonomy results in interfaces becoming less of control modules and more of troubleshooting guides [3].
Until human processing capabilities can be duplicated on a machine, the importance of effective human machine interaction can never be overstated. The drawback with the early interfaces, particularly during teleoperation, was the huge workload placed on the user. With emphasis on easing workload in order to increase the efficiency of a human-robot team, newer techniques have been developed for visualization at the human end. One of the major advancements has been the advent of ’augmentation’.
Traditional teleoperation consisted of just real components and simulators consist
of only virtual component. A combination of the two is called mixed reality. Specif-
ically, the presence of more real components is called Augmented Reality and the
Chapter 1. Introduction
presence of more virtual components is termed Augmented Virtuality. The USP
1of an augmented system is the creation of an integrated display that utilizes multiple data sources to depict an unambiguous interface. This greatly reduces attention levels required from the operator and provides intuitive hints to aid perception.
1.2. Description of the Task
The major problem in teleoperation arises out of time delay imposed by the speed- of-light limit on communications. Across larger distances, this is particularly accen- tuated. In case of an entirely human centric model of operation, the human operator has to follow the move-wait-move sequence in order to observe the results of the ear- lier commands making new operations. The wait stage cannot be dispensed with since otherwise an accumulation of errors takes place. Hence even the use of a pre- dictive simulator only helps in increasing the length of each step of the move-wait model and does not eliminate the waiting part. This thesis aims to explore the effect of a new motion control paradigm called exocentric motion control on teleoperation with and without time delay. A 3-sided CAVE
2will be used to create the interface for immersion and operation. The project is divided into two parts. Under Part I, the objective is to develop a mixed reality interface and implement exocentric motion control paradigm for teleoperation. A basic framework and implementation of connection to the GIMNET network at Helsinki for future cooperation for real teleoperation tests is also built as part of this thesis. The interface will be designed based on the principles of human machine interaction and the effect of various vir- tual components on the interaction will also be studied. Part II consists of testing this interface with USARSim and based on the results of the tests, identify some thresholds for human teleoperation.
1.3. Thesis Outline
The thesis can be broadly divided into three parts -
∙ Theory
Chapter 2 introduces the theory behind teleoperation, the existing technol- ogy and various techniques that have been implemented and published. It
1
Unique Selling Proposition-It is a term used casually to denote any feature of an object that differentiates it from its peers
2
Cave Automatic Virtual Environment - Rear projection screens are used and the user wears
special glasses for an immersive 3D effect.
Chapter 1. Introduction
contains the state of the art of the work from the allied fields of HRI, AR, Psychology and Teleoperation.
∙ Design and Implementation
Chapter 3 contains the design and algorithm of the exocentric model, connec- tivity to GIMnet and the user interface along with the components of the user interface and the prototyping interface.
Chapter 4 contains the design of experiments. Since the thesis aims to fulfill a number of disparate objectives, it is important to design a set of experiments that can individually throw light on the different aspects being studied.
∙ Experimental results and analysis
Chapter 5 describes the experimental results of using different navigation con- trol algorithms, user interface components and predictive display, both with and without time delay. All results are also compared and analyzed to arrive at informed conclusions.
Chapters 6 and 7 provide a summary of the results as conclusion and suggests
future research in this direction that might be carried out to further analyze
the theories of human robot interactions.
Chapter 2
Teleoperation
Teleoperation refers to the operation of a remote machine by a human operator who is spatially and/or temporally separated from the machine. Teleoperation involves three major tasks, listed below in no particular order:
1. Collection of data (position, velocity, obstacles, etc.) by the remote machine (rover)
2. Displaying this data to the remote operator 3. Sending commands to the remote machine.
As discussed in chapter 1, the major reason for teleoperation is to utilize the cognitive capabilities of the human teleoperator and control the remote machine.
This has one major advantage: The operating system is now a human-machine team, and the combined abilities are an improvement over the singletons. But it is clear that for this to succeed, the operator must be made to feel as though present in the remote site. This immersion of the operator is achievable through interfaces that contain a mixture of virtual and real objects and called mixed reality. The virtual objects are used to create a better understanding of the remote site and hence aid in the perception process. It is apparent that the effectiveness of any teleoperation system hinges on the success of the communication interface. This interface is referred to as the Human Machine/Robot Interface. The remaining part of this chapter is a synopsis of Human Machine Interaction that would help in designing a good interface for teleoperation.
2.1. Human Machine (Robot) Interaction
Human machine interaction, commonly referred to as human robot interaction (HRI)
in robotics refers to the interaction or transfer of information between humans and
robots. Any work on HRI is interdisciplinary, involving cognitive science, psychology,
Chapter 2. Teleoperation
engineering, mathematics, computer science and human factors engineering. These interactions are usually characterized by the
1. Type of interaction between the human and machine also called level of au- tonomy
2. Information exchange capabilities.
2.1.1. Types of Interaction
The interaction between the operator and the robot has been classified into multiple levels [4][5] ranging from direct and total control of the robot to an independent and completely autonomous control [6].
Figure 2.1.: Types of Human Robot Interactions [7]
The scale in figure 2.1 is described with a more combined or inclusive interaction
in mind. Such an interaction is termed as mixed initiative and defined as a ”flexible
interaction strategy in which each agent (human and [robot]) contributes what it is
best suited at the most appropriate time” [7]. The HRI issues on the direct control
level require well designed interfaces to reduce the cognitive workload on the operator
while for the extreme case of the completely autonomous peer to peer collaboration,
it is required to create human like cognitive skills on the robot. There may also be a
requirement to model social interactions. The concept of Neglect Tolerance [8] was
defined as a measure of the operator workload. ’Neglect time’ refers to the amount of
time the robot can be ignored before a drop in its performance to below a predefined
threshold and ’interaction time’ is the amount of time an operator requires to bring
back the robot to peak performance. Instantaneous performance of the robot is
defined as the ratio of instantaneous work done to the instantaneous capacity for
work. A combination of neglect time, interaction time and instantaneous work can
be used to define a new term called instantaneous autonomy. This is a step away
from the robot-intelligence based definition of autonomy and towards a task based
definition. Higher the autonomy (instantaneous), longer the robot can be neglected.
Chapter 2. Teleoperation
This definition assumes that the robots performance degrades with time and any interaction with the operator is always a value addition which need not be the case [3]. For example, this metric can be used during the teleoperation of an excavator.
The process of digging can be termed as semi-autonomous. When the depth of digging is constantly decreasing, the autonomy of the robot keeps dropping and at a particular threshold, requires human attention. Also, straight line movement with no obstacles within predefined vicinity can also be termed semi-autonomous. The advantage behind such a definition is that even during direct teleoperation, some tasks can be classified autonomous operations, with visual cues that indicate the same to reduce operator workload.
2.1.2. Information Exchange
There are two characteristics of any information exchange : Communication chan- nel and the format of communication. The different media for communication are typically characterized by the senses that are used to perceive them - seeing, hearing and touch. Classical examples that utilize each of these senses are
∙ Visual Displays like graphical user interfaces and immersive displays [9],[10],[11],[12],[13]
∙ Haptic devices that are used usually to give an immersive feeling to the tele operator like joystick, touchpad, etc. [14],[15]
∙ Gesture recognition (hands and facial recognition)[16], [17], [18], [19]
∙ Auditory or speech recognition for commanding and auditory responses [20],[21]
There are numerous metrics to define the efficiency of information exchange. The time required to send a command to the robot [8], the workload of the operator [22], the situation awareness of the interaction [23], the reaction time of the human-robot team to a stimulus are all different ways to measure this efficiency. According to [24], Interaction can be categorized as
1. Proximate Interaction - the robot and the operator are located at the same place
2. Remote Interaction - the robot and operator are located in different places.
They may also be temporally separated (Moon/Mars)
Chapter 2. Teleoperation
2.2. Teleoperation
Remote interaction with robots is known as teleoperation. Throughout this thesis document, teleoperation is a general word that is used to mean operation with and without (negligible) delay. There are two basic approaches to teleoperation [25]
1. Human Centered
Robot is a tool and operator is a supervisor continually (or with critical alarms) monitoring and commanding the robot. For example, a robot needs to move through a door. The supervisor divides this into a sequence of operations like align robot, give a set of directions to the door, cross in the middle, etc.
now, when a cat comes in between, the robot requires attention. The major drawback of such a control is the operator can become complacent or negligent during autonomous operation. Typical cases are discussed in by Endsley [26].
2. Collaborative Control
This is an approach that enables dialog between a human operator and robot.
A more equal status is assigned to the operator and the robot. The robot can adjust its method of operation based on its own perception of its surround- ings. It also uses human perception and cognition but does not depend on time critical response from the user. In either of the two approaches, a basic requirement is the knowledge and understanding of the existing environmental conditions surrounding the remotely operated rover. In robotic parlance, this is situation awareness.
2.2.1. Situation Awareness
Endsley [27] defined it as ”The perception of elements in the environment within a volume of time and space, the comprehension of their meaning and the projection of their status in the near future”. In simple terms, situation awareness can be defined as an opportunity to perceive and react to the surrounding environment.
The backbone of any teleoperation system would hence be the situation awareness capabilities. But the term by itself covers a huge gambit of information. Situation awareness needs to be broken down into smaller purviews that can be effectively modeled and implemented. Scholtz et al. [28] break down SA into three major components -
1. Vehicular Information, (involving current speed, position, orientation, motion
being executed )
Chapter 2. Teleoperation
2. Obstacle information (obstacles, other required environmental conditions) 3. Route information (in the form of maps, distance to the destination, etc) From the perspective of a user interface, it is necessary to have user configurable triggers for critical alarms and cautions.
Assessment of Situation Awareness The three levels of situational awareness are
1. Level I - Perception . This is the basic level of situational awareness and determines if there is enough information for the user to be able to do his/her task.
2. Level 2- Comprehension The operators must be able to understand and utilize all the data they receive
3. Level 3- Projection
This level determines if the operator will be able to predict the future course using the existing information. Typical questions to determine the three levels would be Level 1 Position of the rover, distance to destination
Level 2 - risks that the operator perceives
Level 3 - Is a particular action like making a left turn or moving straight ahead possible.
2.2.2. Challenges with teleoperation
Two of the biggest challenges [29] during remote machine operation are
∙ Time delay
∙ Interface effectiveness Time Delay
A very common problem in long distance teleoperation is the effect of time delay
on the performance of the entire system. Time delay is present in almost every
teleoperation system. Typically, delay can be classified as constant and variable
delay. A constant delay is one where, as the name suggests, the latency is constant.
Chapter 2. Teleoperation
Figure 2.2.: Situational Awareness Questionnaire
Typical example includes path induced delays like satellite based communications.
For underwater applications, the slow speed of acoustic communications generates huge time delays in the order of few seconds for just a few kilometers. For com- municating to a low earth orbit, the round trip time is around 0.4 seconds and for near-moon communications this is close to 3 seconds. On the other hand, the delay caused on an internet for example is a fluctuating delay due to varying traffic and routing conditions. The delay can often be much greater due to signal buffering and computer processing at communication nodes such as relay satellites. For the case of the Earth-orbiting Space Shuttle, round-trip delays can sometimes approach 6 seconds due to multiple up- and down-links (Earth-to-satellite and vice-versa) in the communication path [Sheridan, 1993]. Luck et al. [30] conclude that there are significant effects on control performance by varying latency, but very few for SA.
In a surprising result, more marking errors were made with short delays than long
delays. This could mean that the higher concentration levels required in long delays
actually helps the operators in maintaining situational awareness. So it now becomes
crucial to find new ways to maintain or improve situational awareness in relatively
shorter delay conditions. Also, variable delay was not found to have a big impact on
Chapter 2. Teleoperation
the SA front, though it causes a loss of smaller details. Latency conditions may also differ between commands sent from the user to the robot and feedback sent from the robot to the user (R2U). Perceived difficulty in maintaining SA was much larger when feedback was delayed than when control signals were delayed. Participants also made more navigation mistakes in the R2U condition, however no differences in marking mistakes were found. [30]
Interface Challenges
In order to receive information from the environment we are equipped with sense organs like eyes, ears and nose. Each organ is like a sensor in the overall system that receives inputs and transmits sensory information to the brain. The first step to an interface design is to comprehend the dynamics of perception. One of the most debated issues is the effect of stimulus on perception. One school of thought is that the process of perception depends on the perceiver’s prior knowledge and ex- pectations. This is an indirect approach to perception and was put forth by Gregory [31] and is also known as the ’top down’ approach. In stark contrast to this is the direct theory of perception proposed by Gibson [32] also known as the ’bottom up’
approach. Bottom-up processing is also known as data-driven processing, because perception begins with the stimulus itself. Processing is carried out in one direction in multiple stages for example from the retina to the visual cortex [33], with each successive stage helping to make a progressive analysis.
Top-down approach uses the contextual information in pattern recognition. A typical case in point is to understand a complex written document. It has been proven to be easier to read complete sentences than individual words because the nearby words help by providing a contextual understanding. For Gregory, percep- tion involves making inferences about what we see and trying to make a best guess.
According to him about 90% of the information that reaches the eye is lost by the
time it reaches the brain. Hence what we think we perceive is an active recon-
struction of reality by the brain aided by past experiences and knowledge. James
Gibson [32] argues that perception is direct, and not subject to hypotheses testing
as Gregory proposed. According to him, the environment has enough information
inbuilt for us to make sense of. He argues that sensation is perception: what you
see if what you get. There is no need for processing (interpretation) as the informa-
tion we receive about size, shape and distance etc. is sufficiently detailed for us to
interact directly with the environment. Affordances which are visual hints inherent
in the environment help perception. Typical affordances are:
Chapter 2. Teleoperation
OPTICAL ARRAY: The patterns of light that reach the eye from the environment.
If the patterns are converging, the object is moving away and if they are diverging, the object is moving closer.
RELATIVE BRIGHTNESS: Objects with brighter, clearer images are perceived as closer
TEXTURE GRADIENT: The grain of texture gets smaller as the object recedes.
Gives the impression of surfaces receding into the distance.
RELATIVE SIZE: When an object moves further away from the eye the image gets smaller. Objects with smaller images are seen as more distant.
SUPERIMPOSITION: If the image of one object blocks the image of another, the first object is seen as closer.
HEIGHT IN THE VISUAL FIELD: Objects further away are generally higher in the visual field.
In order to make teleoperation successful, it is crucial to positively transport all the visual cues from the remote location onto the operator’s visual interface such that the perceived and true affordances are the same.
Teleoperation Interfaces
There have been numerous studies of teleoperated robots and the effect of their interfaces. Tele-existence is an ideal method to perform remote operation of a robot/machine. Such systems, that attempt to immerse the operator into the re- mote site, cannot afford to assume ideal conditions to obtain and transmit sensory information. The design of teleoperation user interfaces has to be approached with a view to answering certain basic questions:
1. Two dimensional or 3 dimensional graphical display?
2. Augmented or virtual display or direct video streaming?
3. Position of camera (first person view or third person view)?
4. Multi windowed or single windowed?
Nielsen et al [13], have conclusively proved that 3D interfaces have a positive influ-
ence on the quality of tele operation with and without time delay. Their results show
that 3D interfaces improve robot control, map building, robustness in the presence
of delay and distracting sets of information, awareness of camera orientation on the
Chapter 2. Teleoperation
robot and ability to simultaneously navigate and perform search operations. Oper- ator workload was much lesser and the 3D displays were found to be very intuitive.
In [12], Nielsen and Goodrich have compared the use of maps and videos in tele operation interfaces and determined that the use of an integrated map and video information in a 3D interface was much more efficient in terms of time to completion and collision avoidance than a 2D combination of the same two cues. They conclude that the 3D interface supports the complimentary use of navigational information in a map and video. Another important conclusion from this study is that when navigational cues are not present in the video, it only serves create a distraction of the operator reducing the time to completion of the task (though there was no degradation in quality of navigation in terms of collisions).
(a) Improved Interfaces [9]. (b) Interface with video and map information [12].
s
(c) Effective Interface [10]. (d) PDA based interface [11].
Figure 2.3.: Graphical user interfaces
Sugimoto et al. [34] determined that the use of first person images was quite a
hindrance to teleoperation and designed a virtual exocentric view to better represent
the posture of the robot and its surroundings for easy cognition. For teleoperation
with a slightly large time delay, a form of supervisory control was proposed which
Chapter 2. Teleoperation
used a simulated model projected without delay for the operator to work on. Tachi et al. [35] developed a real world model in order to cope with difficult environments like a sudden blanket of fog. A virtual reality approach based on polygon models of a complete three dimensional world was used in the Mars exploration mission [36].
A new trend is to use mixed reality interfaces for teleoperation. This is an extension of the older virtual reality concept and works by adding virtual components onto a real world. Real image texture is augmented with polygon models, like simulated displays, that are used to aid human perception of the remote site. There is a growing belief in the robotics community that teleoperation can be dramatically improved through the use of mixed reality without requiring a complete model of the real world. A user interface design involves not just technical data presentation but also a deeper understanding into the psychological and cognitive processes of the human operator as discussed in the earlier section. With the human operator in mind, results from [37] show the need for
∙ A frame of reference for position relative to environment
∙ Indicators about robot status
∙ Information from several sensors displayed in an integrated fashion
∙ The ability for self-inspection and
∙ Automatic presentation of contextually-appropriate information.
Goodrich and Olsen [38] proposed seven principles for efficient HRI, based on ex- perimental evaluations. For information display systems, they suggest the use of natural cues, user determined display of information and design based on the principles of attention management. Techniques to improve situational awareness [39],[40],[41],[42],[43] include
∙ Using a Map
∙ Fusing sensor data,
∙ Minimizing use of multiple windows,
∙ Providing more spatial information to the operator.
Chapter 2. Teleoperation
Figure 2.4.: Ferrel’s experimental results for time delayed telemanipulation in 2- DOF tasks. [44]
Coping with time delay Different methods have been developed to overcome the effects of time delay. One of the earliest experiments performed is depicted in figure 2.4 [44]. The figure details the effects of different time delays in the communication channel on a teleoperation performing a remote manipulation task.
A predictive display model uses extrapolation techniques to calculate a new posi- tion of the machine from the last known position. This is a direct cause and effect scenario where each command is assumed to be standalone. Another type of sim- ulator called the preview-display can also be built where a set of commands can be previewed before ’committing’ into real world. Hence instantaneous results are displayed to the operator while at the same time transmitting the commands to the robot for delayed action. The predictor is a step forward to the ”move-and-wait”
approach, enabling the operator to commit to larger steps with confidence [36].
Time delay in force feedback is a more extreme problem than in visual feedback,
since the operator’s response to out-of-phase feedback can cause instability in the
Chapter 2. Teleoperation
control of the physical system [36]. The figure 2.5 shows the time required for a teleoperator in a master-slave setup to bring a disturbed system back close to normalcy.
Figure 2.5.: Ferrels experimental results time delay in force feedback. [44]
With visual delay, the operator has the possibility to use either a move-wait strategy or supervisory control in order to weave away from the inherent instability associated with unexpected disturbances due to delayed force feedback. The com- pletion time in supervisory control is a sum of the programming time and the time to monitor the task. Just like in real life, it was found that it was easier to do simple tasks by direct control rather than by teaching a machine to do them. As the slope of the complexity curve increases, the machine as logic dictates, becomes faster.
But for extremely complex cases, the programming and execution times increase at
Chapter 2. Teleoperation
a rate faster than by using direct control. This is shown in figure 2.6
Figure 2.6.: Supervisory to direct control time allocation. [36]
Delay-tolerant control laws have been developed which effectively spread the en- ergy out over time, providing robust control at the expense of performance [45],[46].
Techniques like passivation [47], have been developed to build force feedback sys- tems with constant and variable time delays. Some of the most common methods that have been implemented to counter the effects of time delay on tele operation are summarized below.
1. Feed forward control 2. Predictive Display [48]
3. Use of twin system : a complete replica of the remote system at operator site and simultaneous control commands to the twins [49]
4. Forward simulation -Wireframe simulated displays for real time change. smoother control. close task completion time as a no delay system
5. De-synchronization - Time clutch simply means that in the simulated display
there is no correlation with actual time delay. this is valid since there is no
requirement for this synchronization and the experiments have proved that this
method enables better performance in conjunction with the forward simulation
technique.[29]
Chapter 2. Teleoperation
6. Transformed control commands (scattering transformations can be applied to convert the input variables into suitable forms for delayed communications)[50],[47]
Graphical interface design guidelines
A graphical user interface (GUI) can be used as the basic monitor and control in- terface for robots. Information flows in both the directions of the interface. Usually the information to the remote robot (control signals) comprises of much less data compared to the information that flows from the robot (monitoring information).
So the GUI has more components and features dedicated to display of the in flowing data. The control communication and data transfer can be done through wireless Ethernet devices. The GUI is a rich environment for real -time two-way commu- nication between the remote unit and the user. There are three components that form the basis of the design approach:
∙ Visibility of system status
∙ Matching system and real world
∙ User control and freedom
This approach requires that a lot of information be displayed on the user interface.
But this is not practical, since with this amount of data on the screen the user is bound to get confused. So at any given time only a few details about the remote unit are to be supplied to the user. But the details should include critical information regarding the status for the control of robots. A user can interact with the robot through one of the three types of graphical user interfaces: observer, commander, and a super user. The observer can only observe the system. The commander can send commands to the robots. The super user has the ability to control and modify not only the robot but also the environment of the system [51].
Typical user interface components are
1. Navigation Components - Components like compass and GPS provide a nec- essary cue for travelling through any environment. GPS is used to provide the actual position and can easily be displayed with respect to a local frame of reference.
2. Trajectory view - an overhead map of the region to be crossed
3. Terrain view - a map of the environment, including terrain slopes, etc.
Chapter 2. Teleoperation
4. Camera Images
5. Obstacle view - Using sensor data to build an obstacle map 6. Critical alarms
2.3. Graphic Displays
The major part of human perception is through the sense of sight and it has been proven to occupy an extremely large processing bandwidth (106 bits/second as com- pared to 100 bit/second for tactile senses). For this reason, there are a number of display options available in the market. These graphic displays are computer inter- faces that provide real/virtual/mixed images to one or more users interacting with a virtual world. Graphic displays can be categorized according to the image resolution (number of pixels), field of view, type of image (monoscopic/stereoscopic), display technology (LCD/CRT),etc. A more generic classification is based on the intended audience - personalized displays and multi-user display. Personal graphics displays are those that output a scene to be viewed by only one user. The images can be monoscopic, stereoscopic, monocular or binocular. In contrast, displays that allow multiple users in close proximity to simultaneously view an image are called large volume displays. The different displays are listed in Table 2.1
Table 2.1.: Graphical Displays
Classification Types Examples
Personal Graph- ics Displays
Head Mounted Displays Olyumpus Eye-Trek, Pro View XL35
Hand Supported Displays Virtual Binoculars SX Floor Supported Displays Boom 3C, Window VR Autostereoscopic Displays DTI 2018XL Virtual Win-
dow, Ecomo 4D Large Volume
Displays
Monitor Based PV290, Panoram,
Projector Based Immersive Workbench, V- Desk6, CAVE, Panowall, V- Dome
Any display has to match the user’s ocular abilities. The feeling of immersion
depends heavily on depth perception, viewing area and quality of the image (reso-
lution). The field of view of the human vision is 180 degrees horizontally and 120
degrees vertically (when viewed with both eyes). There is a central portion where
Chapter 2. Teleoperation
both eyes register the same image which extends for 120 degrees. Human perception of depth is caused by the horizontal shift in image position viewed by the two eyes.
We never realize that the eyes register two unique images because the brain fuses the separate images into a single image in a process called stereoscopic fusion. Most common displays use monocular vision, like projectors and monitors, and yet they are quite effective in transferring depth cues. But two dimensional images come with the inherent disadvantage of illusions and depth ambiguities. This destroys the fun- damental requirement of tele operation - situational awareness. Hence, it has been generally accepted that stereoscopic displays provide better immersive capabilities.
2.3.1. Stereoscopic Displays
The basic requirement is to project, into each eye of a viewer, a unique image as would have been seen by the eyes in real time. This is generally accomplished by projecting two images onto a projection screen or displaying two images on a monitor, and using filters or optics to direct the correct image to the desired eye. A number of researchers [52],[53],[54],[55] have concluded that stereo displays definitely improve the tele operation, especially those tasks that require an interaction with the environment, obstacle avoidance and navigating unknown rugged areas.
However, there are a few challenges to be addressed with respect to stereo vision.
This works well as a depth cue as long as the image parallax is substantial. It was determined that at a distance of 10m from the viewer, the horizontal shift gets small enough to cause a major degradation of stereopsis and this technique becomes unusable at a distance of 135m [56]. For such larger distances, other visual cues such as linear perspective, shadows and occlusions, surface textures and object details have to be used for depth perception. One of the major drawbacks of stereo projection though is related to user fatigue. The main reason for fatigue is the fact that the brain focuses the eyes on the screen and simultaneously verging the eyes on an object in front or behind the screen. Objects in front of the screen require greater effort and cause more tiredness. This cannot be completely avoided in any stereoscopic display system. Despite this drawback, results conclusively prove that this is by far the most efficient tele operation display and hence is neglected for the remainder of this work.
The process of creating two different images is called stereo separation and there
are a number of techniques that can be used to create the stereo pairs. The most
common ones utilize the color, polarization, time and distance characteristics to
separate the two views. Since polarization of the views is a simple and inexpensive
Chapter 2. Teleoperation
technique and the setup is already available at the laboratory, this technique will be used to create stereo vision. In polarized projection, two polarized images are projected on to each screen and the users wear polarized glasses to view the image.
Both linear and circular polarization can be used. This also implies that there are two projectors required for each wall in order to display the stereo pairs.
The math behind stereoscopic displays using polarization is explained in Appendix C.
2.3.2. The CAVE System
Figure 2.7.: A CAVE model [57]
Before the advent of the CAVE system, VR was accompanied mainly by head
mounted displays. The CAVE is a system where stereoscopic images are projected
on walls using synchronised projectors. This addition of stereo effect creates an
illusion of immersion for the user. Users need special glasses that direct the polarized
image to the correct eye. These glasses made of liquid crystal are extremely light
and much more comfortable than the bulky head mounted displays that were used
earlier. Tracking systems are used to determe the position and orientation of the
users head in order to render the images in the correct [user’s] perspective. A typical
cave is shown in figure 2.7.
Chapter 2. Teleoperation
2.3.3. The CAVE system at the University of W¨ urzburg
Located at the Department of Robotics and Telematics in Julius-Maximilians-Universit¨at, W¨ urzburg, Germany, this system is proposed to be used for the immersive interface between GIMnet and the teleoperator in W¨ urzburg in order to perform tests with J2B2 and AVANT. This is based on a multi plane projection stereo system with 3 projection walls. Each wall is 2 meters wide and 1.6 meters high and arranged with 135 degrees between them. The hardware configuration is shown in figure 2.8 [58]
Figure 2.8.: Cave schematic [58]
For the projection, 6 beamers and a cluster composed of standard PCs are used.
The images for the left and right eyes are polarized orthogonally before being merge-
displayed on the screen. In order to gain stereo effect, the users will be wearing
glasses fitted with the same polarization filters that separate the images for the left
and right eyes. These glasses are extremely light weight and will have no effect on
the performance of the teleoperation task. the 6 client computers and an additional
control PC are connected to the same Ethernet network . The control PC provides
the common interface for the human interaction with the remote machine through
the immersive display. It manages changes in the three dimensional models and
broadcasts these changes dynamically to the clients.
Chapter 3
Design of components and Implementation
Design involves designing the architecture, user interface, and design of control algorithms. Design of architecture is the overall design of the project including connection to the simulators, robots and GIMnet. Interface design involves the various user interface components and control algorithms refer to egocentric and exocentric model of navigation. This chapter describes the design of these parts followed by their implementation.
3.1. Design of Graphical User Interface
As already discussed under section 2.2.2 in chapter 2, lot of research has gone into building user interfaces. Despite this, there are currently no defined metrics for building a new teleoperation interface. Traditional teleoperation consisted of streaming a live video from a camera on board the remote machine as shown in figure 3.1.
Figure 3.1.: Classical (here simulated) main element of a user interface - the camera
image from egocentric robot viewpoint
Chapter 3. Design of components and Implementation
Such a teleoperation lacks in performance and places a tremendous amount of stress on the operator. For this thesis, it was decided to build an augmented reality system, with live video along with certain virtual components that will maximize the performance of the operator. A 3D user interface is built using Java3D
TMcontaining the components listed below.
∙ VRML model of the remote robot
∙ Scan matched laser map
∙ Color coded obstacle map.
∙ Laser dataset displayed as boundaries/walls
The effects of each of these components is also tested during user teleoperation tests. The scan matching algorithm used in the interface is described in section 3.1.1.
3.1.1. Scan Matching
Any mobile robot in unknown surroundings has no objective frame of reference to determine its position. The most commonly used technique for robot localization in an entirely new environment is range scan matching - Despite lower accuracy than by using image data, this method scores due to its inherent ease on computational requirements.
In the scan matching algorithm implemented here, laser data is used as the source of 2D points. The range scan data is a sequence of points defining the locus of intersection of the laser beam and the robots environment. This sequence of points effectively describes a cross-section of the robots surrounding. If the initial position of the robot be 𝑃
𝑟𝑒𝑓and the corresponding laser scan at 𝑃
𝑟𝑒𝑓be 𝑆
𝑟𝑒𝑓. The robot makes an undetermined movement through the unknown environment and takes another scan 𝑆
𝑛𝑒𝑤. Scan matching aligns the two scans to determine the exact difference between 𝑃
𝑟𝑒𝑓and 𝑃
𝑛𝑒𝑤. Hence scan matching can be defined as the method of determining the rotation ℛ and translation 𝑇 which when applied to 𝑆
𝑛𝑒𝑤aligns it completely with 𝑆
𝑟𝑒𝑓.
Iterative Closest Point Method This is one of the most commonly used tech-
niques and was first proposed by Besl and McKay [59]. There are three basic itera-
tions in this model,
Chapter 3. Design of components and Implementation
1. Pair each point in 𝑆
𝑟𝑒𝑓to the closest point in 𝑆
𝑛𝑒𝑤2. Calculate the transformation such that the mean square error (MSE) between the paired points is minimized
3. Apply the transformation to 𝑆
𝑟𝑒𝑓to update the MSE
By giving an initial value for the maximum distance between any two points, outliers can be neglected which also speeds up the algorithm.
ℛ and 𝑇 are computed such that the squared euclidean error is minimized. This error denoted as 𝐸(ℛ, 𝑇 ) and defined by the equation below.
𝐸(ℛ, 𝑇 ) =
𝑁
∑
𝑖=1
[𝑆
𝑛𝑒𝑤− ℛ 𝑆
𝑟𝑒𝑓− 𝑇 ]
2(3.1)
where
𝑆
𝑛𝑒𝑤= ℛ(𝑆
𝑟𝑒𝑓) + 𝑇 (3.2a)
𝜉 = 𝑆
𝑛𝑒𝑤− ℛ (𝑆
𝑟𝑒𝑓) − 𝑇 (3.2b)
𝐸(ℛ, 𝑇 ) = ∑
𝜉
2(3.2c)
3.2. Egocentric and Exocentric Joystick navigation
The most used joystick algorithm in mobile robot teleoperation is called egocentric navigation in this thesis document. Egocentric joystick control is where displacing the joystick to the left or right is interpreted as command to turn/steer the robot to the left or right. Exocentric control interprets similar joystick displacement as a command to move the robot to the left or right, which would consist of an initial turn/steer to orient the robot before performing the actual movement. The symbols used in all graphical representations in this chapter are shown in Figure 3.2
Figure 3.3 shows the difference between ego- and exocentric methods of joystick
navigation. Figure 3.3a is the path the teleoperator visualizes in order to reach the
destination. Figure 3.3b shows how egocentric joystick navigation would take the
Chapter 3. Design of components and Implementation
Figure 3.2.: Symbols used in this chapter
(a) Path envisaged by the teleoperator (b) Egocentric navigation.
(c) Exocentric orientation. (d) Exocentric navigation.
Figure 3.3.: Egocentric and Exocentric navigation
Chapter 3. Design of components and Implementation
robot. Figures 3.3c and 3.3d correspond to how the newly defined exocentric nav- igation would look like as it traverses the path that the teleoperator has in mind.
In order to better understand the effects of the two different algorithms, a new term Navigation Stage is defined. Since the whole concept of remote operation is a human-machine interaction, a variance is always expected - In other words, teleop- eration, like any other human-influenced event, is not and can never have constant performance. Depending on a lot of subjective human factors like ability, fatigue, interest, etc., human performance varies. The different levels (in both learning and performance terms) in human teleoperation are called Navigation Stages.
3.2.1. Navigation stages
Navigation in general will have the three important stages -
∙ Learning curve
∙ Comfort Zone
∙ Overload Zone
Learning curve is the period that the operator requires before starting to feel completely comfortable with the navigation controls. Any experienced operator still requires a warm-up period to understand the controls, the responsiveness of the robot to controls and time delay if any. The comfort zone is the period immediately following the learning curve when the operator feels extremely comfortable using the controls and is at the peak of his efficiency. Overload period is the the final stage when teleoperation efficiency drops, the operator is physically and/or mentally fatigued and beyond a certain point, requires a break.
3.2.2. Egocentric Navigation
All joystick implementations so far have been programmed to behave like steering in a car. Figure 3.4 shows the egocentric navigation which is the same as in figure 3.3 with the different components of the movement labeled from A to F.
The first movement in order to bring the robot along the line is to keep the joystick
in the first quadrant. As the robot approaches point B, the teleoperator has to shift
the joystick position to the second quadrant in order to align the robot with the
previously thought out path. This is impossible to get precisely correct, and there
is always some overshoot as shown by the curved actual path between B and C in
Chapter 3. Design of components and Implementation
Figure 3.4.: Egocentric navigation
figure 3.4. From C to D, the joystick is in the first quadrant again and then reverts to the second quadrant for navigation from D to E, travels a straight path from E to F and then again in the second quadrant position for a slightly curved path to the final destination. The joystick positions for the navigation in figure 3.4 is shown in figure 3.5 [a-f] .
Effects of egocentric navigation
What was apparent when using the egocentric joystick model was that the the teleoperator had to not just think about the remote environment, but also constantly keep in mind the joystick behavior in order to move as desired. The major drawbacks with egocentric navigation are -
∙ Teleoperator has to perform mental 3D rotations before and during navigation.
∙ Operator needs to be aware of not just the planned path from start to desti- nation but also the path determined by the joystick behavior.
∙ Assumes that all teleoperators are aware of car-like steering
∙ Requires constant feedback from the remote site.
The ripple effects of the consequences listed above include
∙ Increased workload on the operator
Chapter 3. Design of components and Implementation
(a) From A to B (b) From B to C (c) From C to D
(d) From D to E (e) From E to F (f) From F to Destination
Figure 3.5.: Joystick positions for egocentric navigation
∙ Difficult for one operator to control multiple remote robots
∙ Higher probability of an accident/damage during teleoperation
∙ An unstable system during time-delayed teleoperation Navigation stage curve for egocentric navigation
The ’navigation-stages’ curve for egocentric navigation based on initial ideas is en- visaged in figure 3.6 The validity of this proposed theory will be tested and test results published in later chapters.
There appeared to be a real need to re-look the egocentric joystick model. In
terms of efficiency, safety and operator workload this model seems to have serious
drawbacks as discussed in the preceding sections. A direct result of this analysis
led to the formation of a new joystick navigation paradigm which will be called
Exocentric joystick model. Where the egocentric model is more concerned with the
robot itself, the exocentric model will differ by placing the required emphasis on the
Chapter 3. Design of components and Implementation
Figure 3.6.: Stages in egocentric navigation
environment. The joystick positions now correspond to directions on the remote site and how the operator wants to move in the environment rather than the operator first visualizing the path and then translating this path into joystick positions that the robot will follow according to the egocentric model.
3.2.3. Exocentric Navigation
In the last section, the drawbacks of the existing joystick navigation model were illustrated. One of the important contributions of this thesis, Exocentric navigation, uses the locus of all points in the environment that the user visualizes as the path to be followed for the remote vehicle. In simpler terms, the user moves the joystick in approximately the same curve that is intended as the final path. This is locally translated by the robot to orient itself in the correct direction and move. This type of navigation can also be termed intelligent navigation and will be illustrated in this section.
The exocentric algorithm can be described as
∙ Step I -Orient the robot along the direction pointed to by the joystick
∙ Step II -Move along the same direction at a speed proportional to the joystick displacement
⋄ An optimization of Step 1 is a ”‘move when orienting”’ process. While still
interpreting the joystick position as the locus of all desired intermediate posi-
Chapter 3. Design of components and Implementation
(a) Orientation in exocentric navigation (b) Actual movement in the direction of orientation
Figure 3.7.: Exocentric Navigation
tions, the orientation comes with a determined amount of translation, as the robot traverses an arc to turn rather than turning around a single point.
Figure 3.7 shows the exocentric navigation which is the same as in figure 3.3. The first step in this model is to orient the robot along the same direction as the joystick [figure 3.7a]. This action is triggered by every change in joystick direction. After this, it maintains its orientation and simply moves forward or backward along the pre-oriented path.
The preliminary instinct of the teleoperator is to point the joystick to the right to move the robot along the path conceived by her/him. This will first cause the robot to orient itself along the joystick direction and then start moving. Figure 3.9a] displays joystick positions. Each black dot corresponds to the times when the operator changed the joystick orientation, leading to a corresponding change in robot orientation and hence change in direction of movement of the remote vehicle.
Under normal conditions, the robot performs straight line motions between any two
joystick direction changes. This is the orient-move sequence. If the exocentric-
optimization described in section 3.2.3 is enabled, then the robot traces a curved
path between two direction changes of the joystick. In either case, the operator uses
Chapter 3. Design of components and Implementation
Figure 3.8.: Planned robot movement by the operator
(a) Actual joystick movement (b) Corresponding robot movement
Figure 3.9.: Joystick positions for exocentric navigation
the joystick to point at the direction in which the remote robot should move and the
robot determines this direction based on the joystick position of the operator and
moves ahead. The outcome of a sequence of such moves is shown on figure 3.9b.
Chapter 3. Design of components and Implementation
Design characteristics of exocentric navigation
The main idea behind exocentric navigation was to negate the drawbacks of egocen- tric navigation. As envisaged during the design stage, this navigation mechanism should have the following characteristics.
∙ Intuitive to the operator
∙ No complex mental calculations or rotations required
∙ No assumption with respect to end user’s prior knowledge.
∙ Ability to work without a constant feedback from the remote site.
∙ Reduced workload on the operator
∙ Ability of the operator to perform multiple tasks
∙ Higher safety of robot in the remote environment
∙ A reasonably stable system during time-delayed teleoperation Navigation stage curve for exocentric navigation
The ’navigation-stages’ curve for egocentric navigation based on initial ideas is il- lustrated in figure 3.10. This curve is a direct outcome of the design characteristics
Figure 3.10.: Stages in exocentric navigation
enumerated in section 3.2.3. A shorter learning curve and lesser operator workload
Chapter 3. Design of components and Implementation
are the salient features perceived for this navigation model. Actual test results are published in later chapters of this thesis.
3.3. GIMnet
GIMnet [60] is a communication tool that was designed and built for robotic ap- plications. It is built like a remote process communication channel and is used as a base architecture for any program interface. A very simple structure explaining the core functionality of the GIMnet components is shown in the skeleton below in figure 3.11.
Figure 3.11.: Major components of GIMnet
The backbone of the network consists of a set of hubs. GIMnet is designed such
that any number of hubs can be added or removed from the pool of interconnected
hubs which take the shape of a Virtual Private Network. With an open TCP port
on the hub being the only fixed requirement, the whole system is easily scaled up
or down.
Chapter 3. Design of components and Implementation
Figure 3.12.: GIMnet schematic [60]
The software modules, which are like clients of the hub, are essentially separate threads running on the network under a unique name and ID. The Generic Intel- ligent Machine Interface or the GIMI functions as an application layer providing a programmable API for module developers. The main features of GIMnet are:
∙ Unicast, multicast, broadcast
∙ Synchronized and unsynchronized data transmission
∙ Automatic hub-to-hub and client-to-hub reconnect
∙ Distributed name and ID service
∙ Service registration, subscription and listing
Chapter 3. Design of components and Implementation
3.4. Predictive Display
The simulator used as predictive display for J2B2 is discussed in this section.
3.4.1. The Open Dynamics Engine simulator
The simulator used is called FSRSim and provides all the interfaces available for the J2B2. This J2B2 Simulator has been widely used in several tests as the master robot of the real robot (teleoperator) J2B2. This means that the teleoperator is able to follow the master, which requires the following.
∙ The worlds (simulated and real) are correlated.
∙ The simulator cannot do something that the real machine is not capable of.
The figure 3.13 below shows the simulator that is used for this thesis.
Figure 3.13.: J2B2 Simulator
Chapter 3. Design of components and Implementation
3.5. Schematic of the experimental setup
The design of experiments is classified into two parts - experiments with J2B2 and experiments using USARSim. The teleoperation setup that was built in order to remotely control the J2B2 is shown below in figure 3.14.
J2B2/AVANT Tele Operator
Three sided CAVE system
Sends Motion Control Commands through Joystick/Keyboard/etc.
to the control PC
Multicast Listener
L R L R L R
PC1 PC2 PC3 PC4 PC5 PC6
Projection PCs
Left eye and Right eye image projectors for each wall
This is a client-server architecture where the listener
broadcasts the updated Java3D scene graph to the 6 synchronized projection PCs
Control PC
Odometry, Laser & Camera Image sent to the control PC
and simulator
Simulator
GIMnet Network in Helsinki/Tampere Würzburg Tele operation Interface
Motion control commands are simultaneously sent to the
simulator and AVANT
Motion commands Control PC creates the JAVA3d scene
graph using the data received from AVANT and Simulator.
Updates are sent via Multicast Connection
Simulated/Predicted values are sent to the control PC.
The operator/control PC connects to the GIMNET network using JNI-Wrapper for the GIMNET libraries
Motion commands can be routed through
the simulator to AVANT (to simulate delay)
