International Master’s Thesis
Inference of User’s Intentions in Adaptive
Autonomy
Giovanni Maioli
Computer Science
Studies from the Department of Technology at Örebro University Örebro 2020
Studies from the Department of Technology at Örebro
University
Giovanni Maioli
Inference of User’s Intentions in
Adaptive Autonomy
Supervisor: Andrey Kiselev Examiners: Todor Stoyanov
© Giovanni Maioli, 2020
Title: Inference of User’s Intentions in Adaptive Autonomy
Abstract
Human-Robot Interaction (HRI) is a leading topic in the field of robotics, its importance is manifest in those tasks where just teleoperating a robot can be not effective, due to time delay, or lack of contest provided by the robot’s sensors. On the other hand, a pre-programmed robot can be not efficient as well, since the environment could be unknown, and the robot could lack the ability of decision making.
For this reason, the aim of this project is to provide an efficient method for human-robot cooperation, in particular through the inference of the user’s intention while teleoperating a robotic arm.
The human agent interacts with the robot through a VR-based interface, which enables him/her with a better understanding of the surrounding en-vironment.
The interface provides an interaction proxy, allowing the user to drive the robot in the virtual reality, that reproduces the environment where the robotic arm actually is.
The algorithm here implemented is able to infer the intention of the hu-man agent, through a method named Head-Hand Coordination Based Inference. The positions and orientations of both the head of the user and the inter-action proxy are used in order to adjust the aim of the teleoperation, as explained in detail in Chapter 3.
Another result of this thesis is the capacity to detect the lack of attention of the user, and to evaluate his/her reliability through a real value parameter called confidence parameter.
The system is responsive to the trustworthiness in the user, and is able to slow down the robot if the human agent is distracted, as well as to move faster if the confidence is high.
In order to test the efficiency of the system, an experiment was conducted, that has shown that this method is safer than basic teleoperation, when the human is distracted.
The data collected also reported that the operator prefers teleoperation to the method here implemented. That comes with no surprises, since the aim
ii
of the system is to reduce the control of the user when the behavior is not safe.
In the future it would be interesting to develop a similar system from the robotic agent side, and test the responsiveness of the human agent in a similar HRI environment.
Acknowledgements
Contents
1 Introduction 1
1.1 Problem Statement . . . 1
1.2 Scenario . . . 2
1.3 Thesis Structure and contributions . . . 2
1.3.1 Overview of the experimental validation . . . 3
2 Related Works 5 2.1 General strategy of using Levels of Automation . . . 6
2.2 Types of LoA . . . 7
2.3 Shared Autonomy . . . 8
2.4 Supervised Autonomy . . . 8
2.5 Arbitration Model for Shared Control . . . 9
3 Methodology 13 3.1 Head-Hand Coordination Based Inference . . . 13
3.2 Differences and Motivations . . . 19
4 Implementation 21 4.1 VR-base Interface . . . 21
4.2 ROS Nodes and Messaging . . . 23
4.3 Simulated Robotic Arm . . . 23
5 Evaluation 27 5.1 Experiment Scenario . . . 27 5.2 Experimental Setup . . . 30 5.3 Results . . . 31 5.3.1 Subject . . . 31 5.3.2 Data Collection . . . 31
5.3.3 Task Completion Time . . . 32
5.3.4 Post Trial Questions . . . 33
5.3.5 Confidence Parameter . . . 34
vi CONTENTS 5.4 Other Observations . . . 34 6 Conclusions 37 6.1 Limitations . . . 39 6.2 Future Works . . . 39 References 41
List of Figures
1.1 Conceptual model of VR based teleoperation system . . . 3
3.1 Graphical representation of the method. The green line shows the head direction line, starting at the head position, that is the green circle. The interaction proxy direction line is repre-sented as the red line, and the proxy pose as the red circle. The distance vector is described by the blue line, and the blue hexagon shows the inferred point. . . 16 3.2 Graphic representation of the rubber target inference. The red
circle represents the interaction proxy, the blue one the in-ferred target, that is at distance di by the interaction proxy, and the yellow one the rubber target position, posed at dis-tance dr by the proxy. The big circle represents the maximum distance between the proxy and the rubber target, that is 0.5m. 17 3.3 3D plot of the confidence function. . . 18
4.1 Top level system architecture. A - VR-based teleoperation in-terface, B - robot control framework, C - robot and its task space. . . 22 4.2 First person view of the VR based interface. The interaction
proxy is represented by a sphere with a number of buttons that enable the user of a better control of the robot. The user can see also the pose of the target cup and of the gripper. . . . 22 4.3 ROS nodes and messages of the entire system. The VR
oper-ator interface is communicating with the rest of the system through the rosbridge_websocket node. The simulated environ-ment in the CoppeliaSim in connected via sim_ros_interface node. 24
5.1 3D model of the robotic arm, the dummies and the target object 30
viii LIST OF FIGURES
5.2 Box plot reporting the completion time results divided with respect to the conditions of the trials. Each box is centered around the mean value, and its size represents the standard deviation. The line represents the span between minimum and maximum value. . . 32 5.3 Box plot reporting the answers to the questionnaire, divided
with respect to the conditions of the trials. For each condi-tion the left box, with the blue line, represents the answer to question one, the right one reports the answers to the second question of the questionnaire. . . 33 5.4 Box plot reporting the average confidence parameter in the
List of Tables
List of Algorithms
Acronyms
AA Adaptive Autonomy
HRI Human-Robot Interaction
LoA Levels of Autonomy
ROS Robot Operating System
SA Shared Autonomy
SAS Semi-Autonomous System
SuA Supervised Autonomy
USAR Urban Search And Rescue
Chapter 1
Introduction
In today manufacturing facilities, teleoperation is widely used as a mean to control machines. This is due to the fact that teleoperation allows an expert operator to handle the work from a safe environment.
Teleoperation is by definition the full manipulation of a robot in real time, by an operator.
On the other hand, fully automated autonomous system are equally im-portant and exploited in the same facilities, for example in production lines. It does not require the presence and active operation of a human agent and it does not suffer from time delays or limited transmission lines. This kind of manipulation is very effective, but is based on the assumption that the robot is operating in a structured environment, since it is hard to pre-program protocols for every possible situation.
The goal of this project is to develop robot manipulation strategy that blends the two over mentioned, that keeps the best of both worlds: the au-tonomy and reliance of fully autonomous systems and the adaptability of teleoperation.
1.1
Problem Statement
In the last years, the interest in Human-Robot Interaction (HRI) has grown wider.
As stated earlier, both teleoperation and pre-programmed autonomy have their own strengths and weaknesses. As a result, the need to merge the two paradigms is apparent to achieve improved performance.
Several proposed merging methods are available in the literature, but due to the innovative nature of the field, some of the definitions remain fuzzy. For example the literature repeatedly interchanges the uses and definitions of the terms "semi-autonomy" and "adaptive autonomy".
In this thesis will be addressed the problem of Adaptive Autonomy (AA), defined as the ability of a Human-Robot system, to cooperate on the
2 CHAPTER 1. INTRODUCTION
plishment of a task by exchanging autonomy at different levels. In particular, in this project, the Levels of Autonomy are not well defined, but it is rather a smooth continuous transition from teleoperation and autonomy.
In the project AMICI, a form of collaborative system was implemented, using a robotic arm and a virtual reality based interface. In this implemen-tation the arm could have been either teleoperated by the operator through the VR interface, or perform pre-programmed operation in an autonomous way. The operator could switch between the two options manually.
The main goal of this thesis is to develop a method to infer the inten-tion of the human agent, while he/she is driving the robot to perform a certain task, and feed the system with this intent, in order to have a better performance.
Another goal of this project is the ability of the system to evaluate the trustworthiness of the operator, and react accordingly.
1.2
Scenario
In analogous systems there is a human agent that acts as a user, meaning that he/she has the duty to use the system to accomplish a certain task. Commonly, the user teleoperates the robotic agent, which means that the robot is moved step by step by the user is every movement, such as a puppet. In order to teleoperate safely and effectively, it is necessary to have a good view of where the robot is with respect to the environment around it.
The type of sensors used by the robot, the way these data are given to the human agent, and the way the user can interact with the robot is one of the main open question in this field of study.
The design of the interface is one of the main factor that affect the quality of the outcome of projects like this one. In this implementation, the interface is based on virtual reality, so that the user can have a view of the surrounding environment and of the robot’s position in it.
Figure 1.1 shows the conceptual model of the VR based teleoperation system. On the left side of the image, it is represented a robotic arm and the experimental environment. On the right side, the same robotic arm and environment are replicated in the Virtual Reality.
1.3
Thesis Structure and contributions
The main contributions of this thesis are: - Analysis of the state of the art (chapter 2).
- A mathematical formulation of the solution (chapter 3). - System implementation (chapter 4).
1.3. THESIS STRUCTURE AND CONTRIBUTIONS 3
Figure 1.1: Conceptual model of VR based teleoperation system. Left part: a slave robotic arm in the task space. Right part: virtual twin of the task space and the virtual master robotic arm for. Image courtesy of project AMICI, published with permission.
This thesis is organized as follows:
In chapter 2 an overview of the works that faced a similar problem has been provided, the most common strategies followed to solve the AA are described here, as well as a more thorough description of an inspiring article, fundamental for this project.
In chapter 3 a description of the proposed solution is provided, as well as the mathematical model used.
In chapter 4 the architecture of the system is described, also the ROS node and the VR-base interface are explained.
In chapter 5 the experiment scenario and setup are presented, together with the results achieved from the experiments themselves.
In chapter 6 conclusions on the project are drawn. Also the limitations of this implementation and insights on possible future developments have been provided.
1.3.1
Overview of the experimental validation
In order to test the developed system, experiments were conducted by the supervisor of the project, Andrey Kiselev, as explained in detail in 5.
The data collected were used to run pairwise t-tests, in order to detect any statistical significant difference between the experimental conditions re-ported later on.
The experimental validation has shown that a human agent which is not focused on the accomplishment of the task is detected by the system.
4 CHAPTER 1. INTRODUCTION
Also the method here implemented, that manages to infer the user’s in-tentions and react accordingly, proved to be safer than basic teleoperation, that is manually controlling the robot’s movements.
The operators’ feedback reported that in general the users preferred to teleoperate the robot, rather than to use the system here implemented. This comes expected, since the system developed does not aid the user in the accomplishment of the task, but only prevents the robot to be driven at high speed when the user is distracted.
Chapter 2
Related Works
In the latest years the topic of cooperation between human and machine, in particular in driving a robot, has been addressed in several ways.
Arkin et al. [10] provided a good taxonomy of the topic of Adaptive Au-tonomy (AA), that is a structure that helps to better frame the cooperation problem, answering to the questions: ’what is initiative?’(classifying it into: disjoint, slightly joint, mostly joint and significantly joint, according to in how many tasks both the operators were involved), ’when to seize initia-tive?’ (here the distinction is between reactive, deliberative and hybrid) and ’how to seize initiative?’ (the agents are required to perform initiative hand-off coordination in the form of one of the following: explicit coordination, implicit coordination, adaptive coordination or no coordination).
Dragan et al. [8] addressed the problem giving a more solid mathematical structure. The co-operation was implemented as policy blending, meaning that the system had to complete two tasks: infer the intention of the human operator, and try to assist the user, accordingly to his/her intentions. Two methods for the prediction were tested: one, simpler, called amnesic, that only kept into account the current position of the robot, the other, which proved to be more effective, that kept into account the trajectory drawn by the robot over time. The system then used the prediction to produce a planning, and blended it with the user’s commands, according to a confidence parameter that grew monotonically with the distance to the inferred target.
In [17] Tecucci et al. highlight seven aspects of mixed-initiative reasoning, namely: task, control, awareness, communication, personalization, architec-ture, and evaluation. Each one of these factors has to be kept in mind when designing any form of Human-Robot Interaction. The task issue regards the division of responsibility between the human operator and the robot system, in order to accomplish a task, the control issue, instead, regards the strategies for exchanging the control between the operator and the system. The aware-ness issue faces the problem of maintaining a shared understanding of the state of the solving process, by the human and the system. The
6 CHAPTER 2. RELATED WORKS
cation issue regards the exchanging of knowledge and information between the human agent and the robotic agent. The personalization issue concerns the adaptation of the system’s knowledge and behaviour to the user’s solv-ing strategies and attitude. The architecture issue faces the design principles, methodologies and technologies for different types of mixed-initiative roles and behaviors. The evaluation issue is related to the human and automated agent contribution to the system, and the overall performance with respect to classic control systems, such as teleoperation or fully autonomous control. In [1] the problem addressed by Allaban et al. is not only to design a co-operation of human and robotic agent, but to test the outcome of this cooper-ation in critical conditions. In this paper, a form of blended shared control is used, and tested with a time delay in receiving human commands, or with a drift in the rotational speed. These stressful conditions are designed in order to emulate possible problems that can affect the outcome of task completion in real time scenarios: where either the signal is not good enough to ensure fast communication between the agent and the robot, or the environment causes the robot to drift, thus affecting the odometry. The system was tested both in a controlled environment and in a more realistic scenario, on a good number of subjects, and showed a wide spectrum of feedback from the users, showing that this particular topic is important, and needs more time energy to be spent on it.
2.1
General strategy of using Levels of Automation
The common strategy is to define a fixed number of Levels of Autonomy (LoA), that are frameworks that describe how each operator (either human or robot) acts, and how much it is involved in solving the designed task.
Usually the system has the ability to change LoA according to which operator is best suited to solve a precise task. The choice of the LoA depends on multiple factors and it is not easy to formulate a criterion that is valid for every situation.
A possible strategy is to provide the human operator with the power to also switch LoA, in this case the system can be described as Human Initia-tive (HI). This method is not really efficient, though, for a certain number of reasons: the model would not have the same efficiency when used by differ-ent human operators (one could choose to give more freedom to the robot, while another could choose to teleoperate the robot for more time), or the human operator could not be able to switch LoA efficiently (for instance for some time delay, or some interference in the signal).
These reason show the importance of choosing a solid strategy to change LoA on the go, that is not barely leaving this power to the human operator. Different strategies to change LoA are still being explored: in [5], for ex-ample, Chiou et al. describe a model of Mixed Initiative variable autonomy, where both the robot and the human can change LoA. They implemented a
2.2. TYPES OF LOA 7
fuzzy Mixed Initiative Controller and use the knowledge of a third ’expert’ operator. Mixed Initiative proved to be a promising method to address the problem, even if it can create conflicts between the robot and the human agent.
In their work [7] Crandall and Goodrich showed a simple yet effective way to decide the actions of the robot, previous to the choice of LoA. They defined three behaviours: a goal-achieving, an obstacle-avoiding and a ve-toing behavior. The goal-achieving provides a vector that represents the de-sired direction of the robot called input vector, this is provided by the current LoA. This vector then is processed by the system in the obstacle-avoiding behaviour, that checks through the sonars around the input vector how safe the area is in that direction. Each of these seven sonars, that represents a direction for the robot, then provides a vote according to the measurement it received. These data are the elaborated in order to provide best direction that drives the robot towards the goal. The best direction, though, is not nec-essarily the safest, for this reason they implemented the vetoing behaviour that forbids dangerous direction.
2.2
Types of LoA
In every Semi-Autonomous System (SAS) there are two basic LoAs which model the fact that only one of the two agents can actually command the robot: Autonomy, where the robot moves itself and the human is just a spec-tator, and Teleoperation, where a human operator manually commands the robot.
Autonomy can be useful for instance when an important delay is present, or if some particular sequence of actions has been modelled in order to help the robot in a determined situation. For example in [2], Yanco et al. defined 2 modes where the robot could take the command regardless of the LoA in which the robot is operating: escape and pursuit. These modes are called behaviours and are triggered by particular conditions in the environment. The pursuit mode is the standard behaviour, where the robot tries to reach the target by moving towards it. The escape mode, instead, activates if the robot is stuck in a corner: it gives the control to the robot, that has a sequence of actions that will let him out of the corner, and then gives the control back to the previous LoA.
On the other hand, Teleoperation can be useful if the robot has some noise in some sensors, thus affecting its performance or even exposing the machine at risk to hit unseen objects. Teleoperation can also be used to guide the robot in decision making, this often happens in Urban Search And Rescue (USAR), which is a common task that is used to test Human-Robot Interaction (HRI) systems.
In [6], Chiou et al. explored the performance of these two LoA under different conditions. In particular, they tested four conditions by putting high
8 CHAPTER 2. RELATED WORKS
or low workload on both the human operator and the robot on a maze like arena.
2.3
Shared Autonomy
Other than these two, the most common LoA is Shared Autonomy (SA), that is a LoA that merges Teleoperation and Autonomy. Meaning that both the agents provide a command, and these two are then reworked into a single input for the system.
Nowadays the best way to merge the commands is still something that is being explored. In [12] Songpo et al., for example, used eye-hand coor-dination to help the system infer the desired target. The command to the robot is the combination of the human motion input and the robot’s input, weighted by a confidence parameter. They introduced two factors to con-sider: Helpfulness, that indicates how much the robot is moving towards the goal, and Friendliness, the agreement between the human command and the final motion command.
Bruemmer et al. in their work [4] provided a HRI system where the robot is not treated as an instrument of the human, but rather is treated more like a competent trusted peer. They designed four LoAs: tele mode (teleopera-tion), safe mode, shared mode and autonomous mode. Safe mode is very similar to teleoperation, but when the robot feels that the human is driv-ing right towards a danger, it takes the command and escapes the peril. In shared mode the robot uses reactive navigation to drive itself, thus setting the human agent free of some workload. In this mode there’s a constant dialogue between the two operators: the human operator can intervene at various levels, and the robot communicates with the human through a set of suggestions that rely on the environment.
2.4
Supervised Autonomy
Another LoA that is widely used is Supervised Autonomy (SuA), where one operator cannot interact directly with the robot’s movement, but can suggest an action to the other operator (who has the control), or can intervene in case of danger. This being a new topic, there is still no definition that is gen-erally accepted, and this can lead to some discrepancies in the terminology between papers.
A good example of this is provided by Basich et al. in [3]. In this paper they defined a Competence-Aware System, that is a model able to switch LoA on the go, in particular the LoAs they have chosen are: no autonomy (or teleoperation), verified autonomy ( where the human has to approve every single action before it is performed), supervised autonomy ( the human can take the command in case of danger) and unsupervised autonomy ( what we previously defined simply as autonomy). In this paper the human operator
2.5. ARBITRATION MODEL FOR SHARED CONTROL 9
is supposed to be expert and the conditions are controlled, avoiding time delays ore other kind of noise from the human, other than the amount of workload.
In [15] Sheridan shows 10 LoAs, in order to better describe the spectrum between autonomy and teleoperation, in particular giving different shades of Supervised Autonomy. Also Sheridan stressed the difference between Al-location Authority and Supervisory Control. The first one is the ability to change LoA (in HI this is a priviledge of the human agent, while in MI both the operators have this power). Supervisory Control, on the other hand, is the ability to change the parameters of the model on the go.
But still these are well defined, discrete levels and a jump between one level and the other is always required.
2.5
Arbitration Model for Shared Control
The solution provided in this thesis is inspired by the article "A General Ar-bitration Model for Robust Human-Robot Shared Control with Multi-Source Uncertainty Modeling " [12]. In this paper, Songpo et al. proposed a creative solution, that in some way, prescinds from the concept of LoA and gives a smoother version of Shared Control.
The major contribution of the paper are:
1. A model able to describe the multi-source uncertainties in the semi-autonomous system.
2. A system that handles the uncertainties from both the robotic and hu-man agent, in order to merge them in the better way.
3. Two metrics able to measure the system’s performance more in a qual-itative way, rather than just success rate and completion time, from the cooperation point of view.
The system is based on the shared control LoA, where the command from the human operator and the robot are merged in a single output command.
First of all, the paper provides a way to infer the target from the human agent. This is done by checking both where the eyes of the human agent are looking, and the path history given from the hand commands. The eye modality is given by collecting data and maximizing the posterior probabil-ity of the possible target position. The collected data are filtered in order to reduce the amount of noise. The hand modality, instead, is trajectory- based, and it maximizes the posterior probability given the history path. The two positions are then merged into one, which is called nominal target, this can be different from the actual target, due to noise in both the inferred modal-ities. Still, this is not a problem, since the purpose of this system is more to
10 CHAPTER 2. RELATED WORKS
minimize the probability of mistaking a target for another, rather than give the correct location of the true target.
The uncertainty on the inferred target is modelled through a 3D Gaus-sian, with the nominal target as the mean, and the variance is considered independent on each axis and it’s calculated using the history of the dis-tance between the nominal target and the actual one.
It is followed the principle for which the more the nominal target is close to a target, the more the target is the correct one; this allows to define a confidence measure that scales with the distance. A threshold is defined so that if the nominal target is too far away, the uncertainty is too big, and the system does not contribute to the task.
Intent uncertainty deals with the problem of which target to point to, au-tonomy uncertainty deals with the problem of how to get to that target. The classical sources of uncertainty in any robotic system are mainly the sources of autonomy uncertainty : sensor’s noises, mechanical issues or time delay, for example. The noises are modelled through a 3D Gaussian, again the dis-tribution is considered independent along each axis, but on the contrary with the inference confidence, the autonomy confidence drops the closest you get to the target.
The two confidence measurements are then merged in a parameter that is used to weight the two commands given from the human and the robotic agent.
α = confin(d)· confau(d)
mt= (1 − α)xt+ αyt
Where α is the combined confidence parameter, confinand confau are re-spectively the confidence values of the inferred target and the autonomy, both of them depend on the distance d. mt is the command vector at time t that merges the human command xtand the robotic yt.
This way of combining the confidence results in a bell-shaped policy that gives low level of autonomy to the robot in the starting and final part of the action.
In order to evaluate the results, it is important to find a good measure; this is actually an open topic in the field of semi-autonomous systems, since there’s no way to measure how the system is performing from the cooper-ation point of view, that is unanimously recognized and used. Success rate and time completion give an overview of the general performance, but do not give an evaluation to the system itself. For this reason in this paper two measure where used that try to address the evaluation problem: Helpfulness and Friendliness.
Helpfulness indicates how much the robotic system is helping in the task accomplishment. That is measured using the α-weighted projection of the robotic command towards the nominal target on the straight line between
2.5. ARBITRATION MODEL FOR SHARED CONTROL 11
the robotic hand and the true target. Obviously the more the nominal target is offset, the lower the helpfulness will be.
Friendliness, on the other hand, is the measure of how much the robot is following along with the human agent. It is calculated through the cosine of the angle between the human command xtand the final command the robot receives mt.
Chapter 3
Methodology
In [11] it was described eye-coordination, when completing a task: contrary on how one could expect, the eyes do not fix on the target point, but they follow a path of intermediate points on which the eyes anchor along the route to the target. In this paper it was shown also how peripheral view influences the fulfilment of the tasks tested.
As stated in the previous chapters, the solution of the problem is inspired by the article [12]. Similarly to the article from Songpo et al., the idea is to merge the data of where the human operator is looking to and where his/her hands are driving the robot, in order to infer which is the most probable target, and to calculate how much this inference is trusted.
In order to infer the target, though, different data from eye and hand are used. In fact the eye information are replaced by the head ones, this is possible thanks to a simulator of virtual reality, that informs the system of where the head is facing.
Instead of the hand information, the interaction proxy position is used.
3.1
Head-Hand Coordination Based Inference
Unlikely from the article [12], no eye-gaze data is available, and a different inference method is used for the hand/proxy modality. For this reason it proved necessary to use the head and the proxy poses to infer the target, since these are the data published by the VR interface.
The idea behind this inference method is to take the midpoint of the vector distance between the direction lines of the head and the proxy. This means to take the midpoint of the smallest size vector that is perpendicular to both the lines: the one that starts from the user’s head and goes along where he/she is facing, and the one drawn in a similar manner from the interaction proxy.
14 CHAPTER 3. METHODOLOGY
In order to do so each line is described in the parametric form, that allows to define a line by having a starting point, a three dimensional direction vector and a parameter.
For the thesis two parametric lines are needed: one that describes where the user is watching, and one that describes where the proxy is going to.
For the head line h the starting point H0 is obviously the head position, the direction vector vh is inferred by the head orientation, described as a quaternion.
Analogously, the proxy line p begins in the interaction proxy position P0 and the direction vector vpis derived by the quaternion.
The quaternion is a 4 dimensional vector, that describes a rotation in the 3D space, just like a 3 dimensional rotation matrix. It is possible to describe the rotation matrix, given the quaternion, in the following way:
x y z w →
1 − 2(y2+ z2) 2(xy − zw) 2(xz + yw) 2(xy + zw) 1 − 2(x2+ z2) 2(yz − xw) 2(xz − yw) 2(zy + xw) 1 − 2(y2+ x2)
Since the interest is in the forward direction, the direction vector is ob-tained by multiplying the rotation matrix with the vector (1, 0, 0)T, thus ob-taining the direction vector:
v = 1 − 2(y2+ z2) 2(xy + zw) 2(xz − yw)
It is possible now to describe the line h, for example, in its parametric form H(t):
h = H(t) = H0+ vh∗ t ∀t ∈R where t ∈R is the parameter.
In an analogous manner, the proxy direction line p is inferred in the parametric form:
p = P(u) = P0+ vp∗ u ∀u ∈R where again u ∈R is the parameter.
Given the two lines, there are three possibilities: either the lines belong to the same planes and are parallel, or belong to the same plane and intersect, or do not belong to the same plane.
Since the human agent does not operate behind the robot, but next to it, it is very unlikely that the two lines are parallel to each other.
If the two lines intersect in a point, the intersection is the inferred point. Lastly, if the two lines belong to different planes, the inferred point is the midpoint of the vector distance of the lines. The vector distance is the
3.1. HEAD-HAND COORDINATION BASED INFERENCE 15
smallest size vector, perpendicular to both the lines, that has each extreme that belongs to one of the two lines.
In order to calculate the midpoint, it is just necessary to have the two extremes of the distance vector: one point that belongs to the line p and one that belongs to the line h.
For any t, u ∈R the vector −−−−−−→
H(t)P(u) = P(u) − H(t) = P0− H0+ vp∗ u − vh∗ t (3.1) is the vector that goes from H(t) ∈ h to P(u) ∈ p. The distance vector is a vector−−−−−−→H(t)P(u)that is perpendicular to both the lines, that can be verified by imposing that the dot product with each direction vector equals zero:
−−−−−−→
H(t)P(u)· vh=0 −−−−−−→
H(t)P(u)· vp=0
By substituting the equation (3.1) into the above system it is obtained:
(P0− H0+ vp∗ u − vh∗ t) · vh=0 (P0− H0+ vp∗ u − vh∗ t) · vp=0
(vp∗ u − vh∗ t) · vh= (H0− P0)· vh (vp∗ u − vh∗ t) · vp= (H0− P0)· vp The above system can be written in a matricial form:
vp· vh −(vh· vh) vp· vp −(vp· vh) u t = (H0− P0)· vh (H0− P0)· vp
In order to ease the notation, the matrix will be signed as A ∈ R2x2, with component ai,j∈R and the vector used as the known terms will be signed as b ∈R2with component bi∈R.
That is a simple two dimensional linear system, that can easily be solved through the Cramer’s method.
u = (b1∗ a2,2− b2∗ a1,2) det(A) t = (b2∗ a1,1− b1∗ a2,1)
det(A)
where det(A) is the determinant of the matrix, that in a 2x2 matrix is calcu-lated as: det(A) = (a1,1∗ a2,2) − (a1,2∗ a2,1).
16 CHAPTER 3. METHODOLOGY
Figure 3.1:Graphical representation of the method. The green line shows the head direction line, starting at the head position, that is the green circle. The interaction proxy direction line is represented as the red line, and the proxy pose as the red circle. The distance vector is described by the blue line, and the blue hexagon shows the inferred point.
It is important to notice that det(A) = 0 if and only if either one of the two direction vector is the null vector (0, 0, 0), or the two lines belong to the same plane, but are parallel to each other.
The two parameters t and u identify two points, on their respective lines, that are the extremes of the distance vector between the two lines.
The midpoint of the vector is the inferred point, and is used to update a vector that keeps track of the latest inferred points. The vector is used to calculate the mean, that is used as the inferred target’s position, and standard deviation of the inferred points over the three axis. The arithmetic mean of the standard deviations over the three axis is calculated, and will be referred to from now on as nominal variance σn.
Figure 3.1 shows in a schematic way the Head-Hand Coordination Based Inference method. The head direction line h is represented in green, it starts from the head position H0 reported with a green circle, the direction vector vh, the black arrow starting from the head position, describes the orientation
3.1. HEAD-HAND COORDINATION BASED INFERENCE 17
Figure 3.2:Graphic representation of the rubber target inference. The red circle rep-resents the interaction proxy, the blue one the inferred target, that is at distance diby
the interaction proxy, and the yellow one the rubber target position, posed at distance drby the proxy. The big circle represents the maximum distance between the proxy
and the rubber target, that is 0.5m.
of the head. Analogously, the proxy direction line p is represented in red, starts from the proxy position P0, represented with a red circle, and is aligned with the direction vector vp described as a black arrow. The distance vector −−−−−−→
H(t)P(u) is shown as a blue line, that links the point H(t) in line h with the point P(u) in line p. The inferred point I is the midpoint of the distance vector, here represented with a blue hexagon.
The goal of this project is to correct the target’s inference, that should be given by the proxy pose. For this reason, the target pointed by the system is not the inferred one, but one called rubber target. This one belongs to the line that goes from the proxy position to the inferred target’s one, but it is closer to the proxy, according to the equation:
dr= atan(di)/π
where dr is the distance from the proxy and the rubber target and di is the distance of the inferred target from the proxy position. This formula ensures a maximum distance from the interaction proxy of 0.5 m, and also moving away from the proxy is harder, as the distance increases. The orien-tation of the rubber target is set to be the same as the inferred target’s one. The position of the rubber target will be referred as the rubber pose.
In Figure 3.2 it is shown in a schematic way the inference of the rubber target. The red circle represents the interaction proxy, and the blue one the
18 CHAPTER 3. METHODOLOGY
Figure 3.3:3D plot of the confidence function.
inferred target, the two are at distance di from each other. The yellow cir-cle represents the rubber target, that is aligned with the inferred target and the interaction proxy, but lies closer to the latter, at distance dr calculated according to the previous formula. The big dashed circle around the inter-action proxy represents the maximum value of dr, that was set to be 0.5 m in this project. The circle is more blurred far from the center, in order to rep-resent the behaviour of the rubber target: it gets harder to move away from the interaction proxy, as more as it gets distant from it.
A real value α ∈ [0, 1], called confidence parameter is needed, that indicates how much the human agent is reliable, and in this thesis is used to set the velocity of the robot’s movements.
In order to calculate the confidence parameter, two values have to be kept into consideration: the nominal variance σn and the distance d between the two lines over discussed, simply calculated as the norm of the distance vec-tor. The parameter α is calculated through the confidence function, as follows:
α = erf(1/d) ∗ erf(1/σn)
where erf(x) ∈ (−1, 1) is the error function. It is important to note that erf(x)∈ [0, 1) ∀x > 0.
In Figure 3.3 the plot of the confidence function is reported. As it can be seen from the plot, the confidence parameter α is close to one only if both the parameters have a low value.
3.2. DIFFERENCES AND MOTIVATIONS 19
The confidence function is designed in this way in order to model two sources of low reliance in the human agent.
In the first one, the one modelled by the variance, describes how much the operator is focused on the task: a low variance indicates that the inferred points are all close over time, meaning that the operator is focused on one point. On the other hand a high variance indicates that the inferences are quite different over the window of time, this can derive from one of the two lines, or both, most likely it is due to the operator looking around.
The second source depends on the distance d between the lines. A big distance indicates that the two lines are quite offset, most likely due to the human agent looking in a different direction, for instance if he/she is dis-tracted by another person. A small value of d, instead, is an indicator that the human is concentrating on the task.
In any case, if one of these values indicates that the operator is not to be trusted, it is important that the system penalizes it. For this reason the two contributions from the parameters are multiplied, so that if one has a low outcome, α will inevitably be set to a low value.
3.2
Differences and Motivations
The main difference between the method here designed and the one in [12] or in [8], for example, is that no previous knowledge of any possible target is needed. In fact Songpo et al., as well as Dragan et al. provided a system that infers the most likely target from a known set.
In [12] probability distributions were used in order to calculate the prob-ability for every element of the set to be the one aimed by the operator. In [8] two different inference methods were developed but both of them were based on the previous knowledge of a set of possible targets.
In this thesis, instead, no previous knowledge was provided, thus prob-ability distributions were not an option available. For this reason the Head-Hand Coordination Based Inference was developed, it allows to drive the robot in a free way, without being forced to move towards a target, enforcing the ability of decision making of the user, that otherwise would be forced to move in a fixed way towards the target.
Another great difference is that both Dragan and Songpo et al. also devel-oped a contribution from the robotic agent side, and some sort of confidence parameter was calculated for the autonomous system as well as for the hu-man agent. This allowed the system to exchange the control of the robot in a continuous way between the operator and the autonomous system. In this thesis only the contribution from the human agent was developed, but, as stated in the Future Works section, this form of cooperation is a goal for future project related to this. For the purpose of this project the confidence parameter was not used to exchange autonomy with a robotic agent, but to slow down the arm and move safely.
20 CHAPTER 3. METHODOLOGY
The confidence function designed in [12] was of great inspiration for the one used in this thesis. In the article by Songpo et al. the confidence function depends on the distance between the inferred target and the correct one as well as some sort of variance, and it uses the error function to manipulate these parameters. Both the contributions, from the human agent and from the autonomous system, are multiplied in order to merge them into a single parameter.
In [8] three different confidence functions are proposed, some of them are more sophisticated and keep into account the entropy of the system, mean-ing that the system is able to consider the surroundmean-ing environment in order to establish the confidence in the inferred target. All three of these confi-dence functions, on the other hand, are strongly based on the assumption that the target position is known and has to be inferred from a set of targets. In this thesis, instead, since no previous knowledge of the target is pro-vided, the confidence function is based on different parameters. The distance between the lines and the nominal variance are the values used to calculate the confidence, both of them are manipulated through the error function, and the contribution are merged through a multiplication, similarly to [12]. The environment is not used by the system in order to calculate the confidence since the system is not aware of it, but the user can react to the environment thanks to the VR based inference.
Chapter 4
Implementation
The system is structured in three main parts: VR based interface, simulated robot along with its environment, and finally robot control algorithms, which are the core contribution of this project.
The system is implemented using Robot Operating System (ROS) [13], to allow different components to communicate with each other. The sys-tem architecture is shown in Figure 4.1: the VR interface, based on Unity3D, publishes the head position and orientation, as well as the position and ori-entation of the interaction proxy. The simulation, designed through Cop-peliaSim [14], subscribes to those topics. Everything is handled by a ROS node, that subscribes to the topics published by the VR, and publishes the inferred target point, and the confidence parameter.
In order to link the various parts of this system, a VPN network was used.
4.1
VR-base Interface
The interface has been taken from project AMICI and adjusted accordingly to represent the actual task space on the robot side. It is based on Unity3D, and immerses the user in a VR environment analogous to the one in the simulation, with the robotic arm and the target.
The user can control the robot through an "interaction proxy", that can be moved by gesture tracking or controller. The proxy is represented by a sphere, equipped with a number of buttons that enable the user to activate or deactivate the tracking of the proxy, so that the human agent is free to position the proxy where he/she desires before the robotic arm tries to reach it. Furthermore the proxy is used to command the gripper of the robotic arm, so that the user can decide when it closes or opens.
Figure 4.2 represents a first person view of the VR based interface by the human agent. The user, through the help of a controller, is moving the interaction proxy, represented by a sphere. The proxy is equipped with a
22 CHAPTER 4. IMPLEMENTATION
Figure 4.1:Top level system architecture. A VRbased teleoperation interface, B -robot control framework, C - -robot and its task space.
Figure 4.2: First person view of the VR based interface. The interaction proxy is represented by a sphere with a number of buttons that enable the user of a better control of the robot. The user can see also the pose of the target cup and of the gripper.
4.2. ROS NODES AND MESSAGING 23
number of buttons that ease the control for the user. In order to give some knowledge of the surrounding environment, the target cup is represented as a cylindrical shape, and the gripper pose is reported as well.
Further details of the use of the interaction proxy are described in [16]. The simulation is based on Unity3D game engine, and is connected to ROS over WebSocket connection. It uses standard ROS messaging system, publishing and subscribing to topics in order to share data with other nodes.
4.2
ROS Nodes and Messaging
In figure 4.3 the conceptual map of the ROS nodes is displayed. The node /aa_auto_inferencepublishes two topics: a tf::Stamped<tf::Pose> message called /aa_auto_inference/auto_target and a std_msgs::Float64 called /aa_auto_inference/auto_alpha. Those are respectively the target pose in-ferred by the autonomous system and its confidence parameter.
Similarly, the node /aa_user_inference provides a tf::Stamped<tf::Pose> message called /aa_user_inference/user_target and a std_msgs::Float64 called /aa_user_inference/user_alpha, where the first one is the target pose inferred by the user, and the second one is the confidence parameter.
The node /aa_shared subscribes to all of these topics: both the poses and the confidence parameter from both the user and the autonomous sys-tem. The node realaborates the poses according to the confidence parame-ters, and publishes a third pose, /aa_shared/target that is the final target pose, that the system will point towards, and a new confidence parameter, /aa_shared/confidence, that indicates how much the target inference is to be trusted.
The simulated environment /sim_ros_interface subscribes to the top-ics published by /aa_shared: the robotic arm moves towards the target pose published, with a speed that depends on the confidence parameter. The sim-ulated environment publishes two pose topics through /rosbridge_websocket, to which the VR based interface subscribes: /sim_ros_interface/GripperPose, that is the pose of the gripper, and /sim_ros_interface/Cup1Pose, that is the physical target.
The user can see the poses in the VR based interface /vr_rig, and act con-sequently. The interface publishes two topics: /vr_rig/interaction_proxy_pose and /vr_rig/head_pose, that are used by the node /aa_user_inference in order to infer the target and the confidence parameter.
4.3
Simulated Robotic Arm
The simulation, designed through CoppeliaSim, comprehends the robotic arm, the target, represented by a cup, and a number of geometric shapes, called dummies used to indicate the positions of certain useful elements,
24 CHAPTER 4. IMPLEMENTATION Figure 4.3: ROS nodes and messages of the entir e system. The VR operator interface is communicating with the rest of the system thr ough the rosbridge_websocket node. The simulated envir onment in the CoppeliaSim in connected via sim_r os_interface node.
4.3. SIMULATED ROBOTIC ARM 25
which are: the head of the user, the interaction proxy, the inferred target and the rubber target.
The robotic arm is provided by CoppeliaSim itself, together with the grip-per, which are respectively: ’UR10’ robotic arm, that is a 6 degrees of freedom arm, and ’Mico Hand’ gripper’s 3D models.
The robot is controlled through the VR based interface, that provides the user with the ability to move an interaction proxy. If the interaction proxy is moved, the robot receives the new desired pose, and reaches it with a speed that depends on the confidence in the user.
In order to make the simulated arm move, the system uses the Inverse Kinematic, that links a small dummy, posed close to the gripper, to the de-sired dummy, so that the robot tries to grapple it.
Inverse Kinematics is the mathematical process that calculates the joint parameters needed so that the robotic end effector reaches a desired position and orientation relative to the start of the kinematic chain.
CoppeliaSim is linked to ROS through the ROS Interface plugin (libsimEx-tROSInterface.so). Thanks to this system, it is possible to write child scripts, which are short chunk of code, and attach them to elements in the simula-tion, these allow to receive and publish ROS messages.
This method was used to receive the pose message of the head’s, the grip-per’s, the inferred target’s and rubber target’s poses, so that the dummies in the simulation represent the relative objects in real time. At the same time, child scripts were used in order to publish the poses of the target, which is a cup in the simulation, and of the gripper.
As stated in the previous chapter, the confidence parameter is used to punish low confidence in the operator. That is done be setting a damping factor in the Inverse Kinematics, so that the robot moves slower, as the con-fidence decreases; high concon-fidence is rewarded with a small damping factor, instead.
The damping factor is a parameter used by the Inverse Kinematic in order to set the smoothness of the joints in the kinematic chain. Setting a high value damping factor results with harsh stiffness in the joints, thus the robot is harder to move. On the other hand, a low damping factor causes the joints to be smooth and easy to move, thus the robot moves easier.
If the system had to be switched from simulation to an actual UR10 robotic arm, the interface would not need to be changed. In that case it would prove necessary to solve the Inverse Kinematics problem in analytical way, for example. The setting of the Kinematic problem, though, would not change: the effector of the robot would still be the gripper, and the target to be reach by the end effector would be the rubber target.
That is because the simulation only provides the topics of the target pose and of the gripper pose, while the topics useful for the inference of the target, that are the head pose and the interaction proxy pose, are published by the VR based interface.
Chapter 5
Evaluation
The goal of experimental validation is dual:
1. Verify that the system developed is safer than standard teleoperation. 2. Verify that the confidence is according to the human behaviour.
5.1
Experiment Scenario
The task to be completed in the trial is to reach a target, represented by a cup, whose position is randomly generated every trial. The conditions for every trials are generated randomly, as explained further, and neither the user nor the supervisor know the conditions of the trial beforehand. After every trial, the user will have to answer a questionnaire, that will be discussed in details further on, in order to evaluate the performance of the trial.
Experimental Conditions
As stated earlier, the conditions for the experiment are generated randomly, and are the following:
0 The ideal good user is focused on the task for the whole trial. This means that the head and the proxy’s direction lines have a small dis-tance vector, and the variance of the target’s position (which is mostly determined by the head) is small. If the user is focused on the task, one expects to see a high confidence parameter for the whole time, mean-ing that the robotic arm would follow the rubber target with a high speed.
1 The bad user is distinguished either by a big distance vector between the head and interaction proxy’s direction lines or by a high variance throughout the trial. If the user is looking towards a different direction from the proxy’s or if the variance is too high, the confidence parameter
28 CHAPTER 5. EVALUATION
is set to be low. The low confidence can be visualized by the robotic arm that moves slowly towards the rubber target. In order to provoke this condition in the user, a distractor will be activated, so that the user will suffer of high workload.
D The user is directly teleoperating the robotic arm, regardless of where he/she is watching. In this kind of trials, the confidence parameter is not useful for the system, for this reason it’s not kept track of this data under this condition.
I The robot is set to track the rubber target, thus, the inferring system is tested, which is the core of this project. As explained earlier, the rubber target’s inference depends as much from the interaction proxy, as from the head of the user, for this reason this condition is the one that will provide an estimate of the user’s confidence parameter.
The previous notation indicates which are the conditions of the exper-iment:
– 0 means no distractor has been put in place during the trial.
– 1 means that the distraction took place in the trial.
– D indicates that the robot was set to be directly teleoperated.
– I means the robot tracked the inferred rubber target during the trial.
So the notation "0D", for example, indicates that the trials had no dis-tractor and the robot had been teleoperated.
It is important to have a comparison between the two latter conditions, in order to establish if the system designed in this project is actually a step forward in this field of study, or if it needs more work to be done on the subject.
Therefore, the resulting experiment scenario can be described as a 2X2 set of conditions that are tested in a number of trials.
Experiment Flow
The robot follows either the interaction proxy or the rubber target, this con-dition is decided randomly every trial, without the user knowing which one is the one being tracked.
In the same way, the distractor can be active or not every trial, this is chosen randomly, as well. The distractor is used to emulate the behaviour of a bad user, and it consists of the following: if the user hears a ringing bell he/she has to stop performing the task, and look for a sphere that is generated in a random place. The human agent has to look at the distractor
5.1. EXPERIMENT SCENARIO 29
and communicate the supervisor the colour of the sphere, which changes randomly every trial. The user is not aware if or when the distraction is going to take place, while the experiment is being performed.
After every trial, the subject had to answer a questionnaire asking two questions in order to receive feedback about the usability of the system. Both the questions can be answered through a Likert scale ranging from 1 to 5:
Q1. How easy was it to perform the task? (1 (very hard) - 5 (very easy)). Q2. How good is the system? How well does it respond to your
expecta-tion? (1 (very bad) - 5 (very good)).
The reason behind the choice of a questionnaire is that, since this is a Human-Robot Interaction (HRI) project, it is important to have feedback from the users. In particular, data such as completion time and average con-fidence parameter, describe the performance of the method, but do not mea-sure the cooperation with the human.
In order to measure the efficiency of a system from the cooperation point of view in an analytic way, no metric has been designed that is unanimously recognized as valid. For this reason subjective metrics such as questionnaire are used.
In [12] two metrics were designed in order to measure the cooperation: Helpfulness and Friendliness. Due to the fact that there is no actual contri-bution from a robotic agent in the choice of the target, in this thesis, these metrics were not used.
The questionnaire is aimed to measure the method as perceived by the human agent. The interaction between human and robot needs to be as smooth as possible, thus the feedback from the subjects is regarded as a fundamental data for the evaluation of the project.
In HRI it is crucial that the user perceives the system as helpful and friendly, in order to prevent him/her from getting frustrated. A policy that is designed poorly from the cooperation point of view is counterproductive to the purposes of accomplishing a task.
The user was not aware of the conditions above described of the trial just conducted, meaning that he/she was not aware if the system was operative, or the robot was being teleoperated.
In every trial it is measured the completion time and the average confi-dence parameter during the whole experiment.
If the system is set to follow the rubber target, we should be able to observe that the more the human operator is focused on the task, moving the interaction proxy consistently with the direction where the head is facing, the higher the confidence is, and the faster the robot moves.
30 CHAPTER 5. EVALUATION
Figure 5.1:3D model of the robotic arm, the dummies and the target object
On the other hand, if the user results not focused on the task, the con-fidence parameter is supposed to be low, and thus the robot should move slower, or even stay still if the confidence is zero.
If the arm is tracking the interaction proxy, instead, that is if the robot is teleoperated, we should simply observe the commands of the user being followed in real time by the robot arm.
5.2
Experimental Setup
The system was built using the simulator Coppelia Robotics, that provides a wide range of prebuilt robots.
Figure 5.1 shows the Experimental Setup, with the target and the robotic arm simulated. The robotic arm was modelled through the ’UR10’ 3D model, provided by CoppeliaSim, a gripper was attached on the ’hand’ link, in order to collect the items. For the gripper the ’Mico Hand’ 3D model was used, always provided by CoppeliaSim. In order to set up the Inverse Kinematic model, a target dummy was added that the hand is supposed to reach.
The green dummy represents the head of the operator, with its position and orientation, the yellow one represents the interaction proxy. The red dummy shows the pose of the inferred target, and the blue one the rubber target, which is the one used as a target for the Inverse Kinematic model.
5.3. RESULTS 31
The simulation was designed in order to publish Robot Operating System (ROS) pose messages of the target, /sim_ros_interface/Cup1Pose, and of the gripper pose, /sim_ros_interface/GripperPose.
The system received messages from a vr simulator, that published ROS
messages of the positions the human operator was looking towards, /vr_rig/head_pose, and the interaction proxy position, /vr_rig/interaction_proxy_pose.
The average ping under experiment was 140.9 ms (sigma 70.2).
Subjects were asked to sign consent form. The study has undergone eth-ical approval and registered with GDPR authority at Örebro University.
5.3
Results
The data collected during the experiments were used to test either or not there is statistical significant difference between the various conditions. In order to do so, t-tests were run pairwise for all the conditions: 0D-0I, 0I-1D, 1D-1I, 1I-0D, 0D-1D, 0I-1I.
T-tests were run, because the goal was to test two-tailed distribution. Data was assumed to be distributed according to a normal Gaussian distribution.
5.3.1
Subject
The subjects that tested the system were a set of six people, four female and two males. The mean age of the subjects was 33.17 and the standard devi-ation was 2.93. All the subjects have an educdevi-ation that regards technology, four of them have a master degree, and two are PhD students.
Three subjects out of six play video games at least 1-2 hours per week.
5.3.2
Data Collection
Each user performed twenty trials. The choice of the conditions for each trial was random, as already stated, and it was unknown to both the user and the experimenter before the start of the trial.
Due to the randomization, the trials were not ordered in any way. It hap-pened, for example, that a subject had to perform seven trials in a row under the condition "0I".
In total the number of trials for each pair of conditions are: • 30 trials "0D"
• 32 trials "0I" • 24 trials "1D" • 34 trials "1I"
32 CHAPTER 5. EVALUATION
Figure 5.2: Box plot reporting the completion time results divided with respect to the conditions of the trials. Each box is centered around the mean value, and its size represents the standard deviation. The line represents the span between minimum and maximum value.
It happened that a subject could not reach the target, even though the gripper was extremely close, this brought the user to get frustrated, and an unusual high completion time, as can be seen in the maximum completion time of condition "0I" reported in figure 5.2. This outlier was probably due to an inefficiency in the system, but it was kept as valuable data anyway.
Every subject received short instructions on the purpose of the experi-ment, the system and what they were supposed to do in order to interact with the robot. It was given the chance of testing the environment, try the VR based interface, and teleoperate the robotic arm. The experiment started when the subject was confident and gave signal to be ready.
The data collected are reported through the use of box plot images. Each box is centered in the mean of the trials for the corresponding condition, and the length of the box depends on the standard deviation. The line represents the maximum and minimum data collected.
5.3.3
Task Completion Time
Figure 5.2 shows the average completion times for each set of conditions. Since the comparison is between four conditions, instead of just two, the t-tests were run with a p value of p < 0.125, instead of the usual p < 0.5.
5.3. RESULTS 33
Figure 5.3:Box plot reporting the answers to the questionnaire, divided with respect to the conditions of the trials. For each condition the left box, with the blue line, represents the answer to question one, the right one reports the answers to the second question of the questionnaire.
Statistical significance was found on the pairs 1D-1I, 1I-0D, with such p value, this means that task completion time shows a relevant statistical difference between the trials in which the robot was being teleoperated (D) with respect to the ones where the robot was tracking the rubber target (I). Such a statistical difference was not noted for the pair 0-1, that is if the distraction was active or not.
5.3.4
Post Trial Questions
Figure 5.3 shows the answers of the subjects to the questionnaire after every trial, classified according to the classification of the trial itself. The left box, in each condition, represents the answers to the first question, the blue line rep-resents as always the maximum and minimum score. The right box, instead, reports the results for the second question, the span between maximum and minimum score is represented by the red line.
Since the answers to the questionnaire do not necessarily follow a Gaus-sian probability distribution, t-tests are not a valuable tool to evaluate this kind of data.
34 CHAPTER 5. EVALUATION
Figure 5.4: Box plot reporting the average confidence parameter in the trials with conditions 1I and 0I.
Still, the average results show some differences between the conditions of the trials. First of all, the trials were the robot was teleoperated by the user, scored higher in both the questions with respect to the ones where the system was inferring the target. That is: "0D" scored higher than "0I" and "1D" scored higher that "1I".
On the other hand, there is no evident difference in the trials where the distractor was active or not: "0D" and "1D" have a similar score, as well as "0I" and "1I".
5.3.5
Confidence Parameter
Figure 5.4 shows the average confidence of the trials with conditions "0I" and "1I". The trials were the robot was being teleoperated (condition D) are not reported in this table, since the confidence parameter data were not collected in these kind of trials.
The results show statistical significant difference with a p value of p < 0.5 between the trials with or without distractor, this means that the confidence is lower if the user is distracted by the system.
5.4
Other Observations
During the experiments it was noticed a behaviour of the users that can have had some effects on the outcomes of the trials: the user rarely changed his/her position with respect to the robot, even though it was not the best.
5.4. OTHER OBSERVATIONS 35
For example if the target was generated behind the user, the interaction proxy was moved through the human agent in the simulation. This made the completion of the task way more difficult than it would have been if the user was in a comfortable position with respect to the robot and the target.
Chapter 6
Conclusions
As stated in the previous chapters, there is a substantial difference between the scientific article from Songpo et al. and this thesis, that is the difference in the available data.
In [12] eye-gaze data was used, this allowed a clearer understanding of not only where the human operator was looking towards, but also what he/she was looking, making it easier to infer the target from the eye modal-ity.
For the hand modality inference, the path history was used, in order to infer the target which was most likely the one the robot was driven towards. In this thesis eye-gaze was not an available data, but the VR only pro-vided the position and the orientation of the user’s head. From this informa-tion, it was extracted the direction the human agent was looking towards, described as a parametric line.
In a similar manner, the VR provided the position and the orientation of the interaction proxy, and again from these data, the direction of the proxy was described as a line in parametric form.
The inferred point, was calculated as the mean of the midpoint of the distance vector between these two lines, calculated over a small window of time, and the norm of this vector was used, together with the nominal variance, to calculate the confidence parameter.
This way of inferring the target, may cause some problems, especially under certain conditions.
If the head and the proxy are close to each other, and they look towards a similar direction, it is very likely for the nominal variance to be quite high. That is because it is likely that at least one of the two lines does not intersect the target, and maybe intersect the other line in another point. In this situ-ation, small variation of the direction line (that are frequent from the head) can result in a big difference in the inferred points.
Another problematic scenario, would be when the two lines belong to the same plane but are parallel to each other, generating confusion on which
