Vision-Based In-Hand Manipulation with Limited Dexterity

(1)

Vision-Based In-Hand Manipulation with Limited Dexterity

SILVIA CRUCIANI

Doctoral Thesis Stockholm, Sweden 2019

(2)

TRITA-EECS-AVL-2019:74 ISBN 978-91-7873-332-3

Division of Robotics, Perception, and Learning School of Electrical Engineering and Computer Science KTH Royal Institute of Technology SE-100 44 Stockholm, Sweden Copyright © 2019 by Silvia Cruciani except where otherwise stated.

Tryck: Universitetsservice US-AB 2019

(3)

iii

Abstract

In-hand manipulation is an action that allows for changing the grasp on an object without the need for releasing it. This action is an important component in the manipulation process and helps solving many tasks. Hu- man hands are dexterous instruments suitable for moving an object inside the hand. However, it is not common for robots to be equipped with dexterous hands due to many challenges in control and mechanical design. In fact, robots are frequently equipped with simple parallel grippers, robust but lacking dexterity. This thesis focuses on achieving in-hand manipulation with limited dexterity. The proposed solutions are based only on visual input, without the need for additional sensing capabilities in the robot’s hand.

Extrinsic dexterity allows simple grippers to execute in-hand manipula- tion thanks to the exploitation of external supports. This thesis introduces new methods for solving in-hand manipulation using inertial forces, controlled friction and external pushes as additional supports to enhance the robot’s manipulation capabilities. Pivoting is seen as a possible solution for sim- ple grasp changes: two methods, which cope with inexact friction modeling, are reported, and pivoting is successfully integrated in an overall manipula- tion task. For large scale in-hand manipulation, the Dexterous Manipulation Graph is introduced as a novel representation of the object. This graph is a useful tool for planning how to change a certain grasp via in-hand manipulation. It can also be exploited to combine both in-hand manipulation and regrasping to augment the possibilities of adjusting the grasp. In addition, this method is extended to achieve in-hand manipulation even for objects with unknown shape. To execute the planned object motions within the gripper, dual-arm robots are exploited to enhance the poor dexterity of parallel grippers: the second arm is seen as an additional support that helps in pushing and holding the object to successfully adjust the grasp conﬁguration.

This thesis presents examples of successful executions of tasks where in- hand manipulation is a fundamental step in the manipulation process, showing how the proposed methods are a viable solution for achieving in-hand manipulation with limited dexterity.

(4)

iv

Sammanfattning

In-hand manipulation gör det möjligt att ändra fattningen om ett objekt utan att behöva släppa det. Detta är en viktig komponent och gör det möjligt att lösa många uppgifter. Den mänskliga händen är ett flexibelt instrument som är lämpligt för att flytta föremål inuti handen. Det är dock inte vanligt att robotar är utrustade med lika flexibla händer på grund av utmaningar inom reglerteknik och design av mekaniska system. I själva verket är robotar ofta utrustade med enkla parallel gripper, som är robusta men saknar finmo- torik. Denna avhandling fokuserar på att uppnå in-hand manipulation med begränsad finmotorik. De föreslagna lösningarna baseras endast på visuell perception, utan behov av ytterligare sensorer i robotens hand.

Extrinsic dexterity (extrinsisk ﬁnmotorik) gör att enkla robothänder kan utföra in-hand manipulation tack vare utnyttjandet av externa stöd. Denna avhandling introducerar nya metoder för att lösa in-hand manipulation med tröghetskrafter, kontrollerad friktion och yttre tryck som ytterligare stöd för att förbättra robotens manipuleringsförmåga. Pivoting ses som en möjlig lös- ning för enkla greppförändringar: två metoder som hanterar inexakt friktions- modellering presenteras samt som gungning är framgångsrikt integrerats i en fullständig manipuleringsuppgift. För storskalig in-hand manipulation intro- duceras Dexterous Manipulation Graph som en ny representation av objektet.

Denna graf är ett användbart verktyg för att planera ändring av grepp via in- hand manipulation. Det kan också utnyttjas för att kombinera både in-hand manipulation och regrasping för att öka möjligheterna att justera greppet.

Dessutom utvidgas denna metod för att uppnå in-hand manipulation även för föremål med okänd form. För att utföra de planerade objektrörelserna i robothanden utnyttjas dubbelarmade robotar för att förbättra den dåliga färdigheten hos parallel gripper: den andra armen ses som ett ytterligare stöd som hjälper till att skjuta och hålla objektet för att framgångsrikt justera greppkonﬁgurationen.

Denna avhandling presenterar exempel på framgångsrika utföranden av uppgifter där manuell manipulation är ett grundläggande steg i manipule- ringsprocessen och visar hur de föreslagna metoderna är en rimlig och eﬀektiv lösning för att uppnå handmanipulation med begränsad ﬁnmotorik.

(5)

v

Acknowledgments

This thesis is the product of four years of doctoral studies. Four years full of joy and desperation. Full of fun trips all over the world and endless nights trying to get things to work. Luckily for me, there were a lot of people I could share the moments of happiness with, and that supported me during dark times.

First of all, I would like to thank my main supervisor Danica for her guidance and wise advices. She helped me to look beyond the close problems and try to see the bigger picture. I would also like to thank my co-supervisor Christian for guiding me during my initial steps in the academic world.

A special thanks goes to Francisco, who will always have my gratitude. Without him the four years of PhD studies would not have been the good experience they were. Thank you for bringing good mood and friendliness wherever you go!

Thanks to the people of the Amazon Picking Challenge. That was probably the best and most fun experience I had during these four years. I have learned a lot from it and all of you (Raresh and Sergio deserve a special mention). Thanks to my first officemates in the lab, Michele, Alejandro and Ali, for the warm welcome and good first impression of RPL (CVAP back then). Many thanks to Judith, for putting a roof over my head when I first moved to Sweden. Joshua, thanks for bearing with me at the beginning, when I obviously did not know what I was doing. We traveled a lot together and having you around made those trips much better! Anastasiia, thanks for (officially!) being my friend. You are a rational voice in a world of nonsense. My dear friend João, so long and thanks for all the coffee. Hang, thank you for giving me the opportunity to go to Hong Kong! It was a great experience and working with you was a pleasure. Thanks to Rika! I have learned a lot from you. Akshaya, thanks for all the support and all the boardgames.

Speaking of which, thanks to the boardgames people for all the enjoyable games we played together, especially those I won.

I want to extend my thanks to all the people at RPL for the nice and friendly environment in the lab (and thanks to Patric, because he always tries to improve everything). Mia, Püren, Olga, Özer (thank you for the anime parties!), Vlad, the other Vlad, Michael, Nacho (pio pio), Isac, Ioanna, Irmak, the great Jonny Karaoke, Soﬁa, Marcus, Nils, Federico, Xi, Fernando, Yiannis, Iolanda, Jana, Daniel, Ludvig:

thank you all!

And then, in the end, thank you Diogo. You are probably the ﬁrst person I should have mentioned, but please appreciate the suspance.

I am sure I forgot someone, so if you are not in this list, fear not: I thank you too. Oh, right, many thanks to my family for all their support. Mamma, papà e Sonia. Visto? Non mi sono dimenticata di voi!

(6)

(7)

vii

List of Papers The thesis is based on the following papers:

Rika Antonova*, Silvia Cruciani*, Christian Smith and Danica Kragic.

Reinforcement Learning for Pivoting Task. In arXiv:1703.00472, 2017.

Silvia Cruciani and Christian Smith. Integrating Path Planning and Pivoting. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain, 2018.

Silvia Cruciani, Christian Smith, Danica Kragic and Kaiyu Hang.

Dexterous Manipulation Graphs. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain, 2018.

Silvia Cruciani, Hang Yin and Danica Kragic. In-Hand Manipulation of Objects with Unknown Shapes. Submitted to IEEE International Conference on Robotics and Automation (ICRA), 2020.

Silvia Cruciani*, Diogo Almeida*, Danica Kragic and Yiannis Karayiannidis. Discrete Bimanual Manipulation for Wrench Balancing.

Submitted to IEEE International Conference on Robotics and Automation (ICRA), 2020.

Some results reported in this thesis have been presented in the following papers:

Silvia Cruciani and Christian Smith. In-Hand Manipulation Using Three-Stages Open-Loop Pivoting. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) Vancouver, Canada, 2017.

Silvia Cruciani, Kaiyu Hang, Christian Smith and Danica Kragic.

Dual-Arm In-Hand Manipulation and Regrasping Using Dexterous Manipulation Graphs. In arXiv:1904.11382, 2019.

Joshua A. Haustein, Silvia Cruciani, Rizwan Asif, Kaiyu Hang and Danica Kragic. Placing Objects with Prior In-Hand Manipulation Using Dexterous Manipulation Graphs. In IEEE-RAS 19th International Conference on Humanoid Robots(Humanoids) Toronto, Canada, 2019.

*Equal contribution.

(8)

viii

Other Publications

The following additional works have been published during the author’s studies but are not included in this thesis:

Rakesh Krishnan, Silvia Cruciani, Elena Gutierrez-Farewik, Niclas Björsell and Christian Smith. Reliably Segmenting Motion Reversals of a Rigid-IMU Cluster Using Screw-Based Invariants. In IEEE-RAS 18th International Conference on Humanoid Robots (Humanoids), Beijing, China, 2018.

Judith Bütepage, Silvia Cruciani, Mia Kokic, Michael Welle and Danica Kragic. From Visual Understanding to Complex Object Manipulation. In Annual Review of Control, Robotics, and Autonomous Systems, 2019.

Silvia Cruciani, Kaiyu Hang, Christian Smith and Danica Kragic.

Dual-Arm In-Hand Manipulation Using Visual Feedback. In IEEE-RAS 19th International Conference on Humanoid Robots(Humanoids) Toronto,

Canada, 2019. Best Oral Paper Award Finalist.

(9)

1 Introduction . . . . C3 2 Related Work . . . . C5 3 Dexterous Manipulation Graph . . . . C6 4 Dual Arm Formulation . . . C14 5 Experiments . . . C17 6 Conclusions . . . C19 References . . . C20 D In-Hand Manipulation of Objects with Unknown Shapes D1 1 Introduction . . . . D3 2 Overview . . . . D5 3 Shape Reconstruction . . . . D6 4 Online Dexterous Manipulation Graph . . . . D7 5 Experiments . . . D13 6 Conclusions and Future Work . . . D16 References . . . D16 E Discrete Bimanual Manipulation for Wrench Balancing E1 1 Introduction . . . . E3 2 Problem Description . . . . E5 3 Proposed System . . . . E8

(11)

CONTENTS xi

4 Experiments . . . E12 5 Conclusions . . . E15 References . . . E16

(12)

(13)

Part I

Overview

(14)

(15)

Chapter 1

Introduction

1 Dexterous Manipulation in Robotics

Dexterity is a fundamental human skill that has guided the evolution of mankind.

For humans, it is an innate ability that can be easily exploited thanks to our ex- tremely dexterous hands. Our senses, such as touch and sight, also play a significant role in our manipulation capabilities. People can grasp objects and tools and use them to execute many different tasks. Most of these tasks require little or no training, and are intuitive and easy to accomplish. Fig. 1 shows examples of human hands executing different tasks thanks to their dexterity.

Since robots are machines meant to aid humans and improve our lives, we want them to be able to perform the same tasks as we can; if not better, at least as well as us. Hence, it is an obvious course of action to research how to equip robots with the skill of dexterous manipulation.

A first option to endow robots with such skill is to equip them with hands that resemble the human one [1]–[3]. Fig. 2 shows examples of this kind of robotic hands. The presence of many degrees-of-freedom, given by the several actuators in the mechanical design, allows the artificial hands to reconfigure objects within

(a) Shuﬄing cards. (b) Knitting. (c) Playing harp.

Figure 1: Examples of tasks that require dexterity.

3

(16)

4 CHAPTER 1. INTRODUCTION

(a) Pisa Soft Hand II. (b) DLR/HIT Hand II. (c) Shadow Dexterous Hand.

Figure 2: Examples of dexterous hands. Images from [1], [3], [6].

the hand itself by moving the fingers and sliding and rolling the contacts with the object’s surface. This ability of artificial hands is also referred to as intrinsic dexterity. The addition of high-quality and high-frequency sensory feedback enables dexterous hands to perform quick and precise readjustments of the grasp. For instance, in [4] the hand can toss the object in the air and catch it in the desired configuration thanks to the exploitation of a high-frame-rate camera. In [5] the artificial hand moves the fingers to react to changes in the object mass, perceived through tactile sensing, to achieve a more stable grasp.

These examples of dexterous manipulation are enabled by the mechanical complexity of the artificial hands. This complexity leads to many challenges in design, planning and control [7], [8]. Challenges in design result in hardware that cannot fully mimic the human hand, lacking in touch sensing and in actuation when compared to humans. Hence, reproducing human dexterity still remains an open problem [9]. Researchers often simplify the problem of dexterous manipulation and focus on a narrower scope, such as only moving a subset of the fingers, not moving the contacts between fingertips and object or planning for predefined simple motions [10]–[13]. Inspired by human motions to overcome the challenge of accu- rately modeling the task, the authors of [14] propose to interpolate between different pre-recorded human grasps to achieve versatile in-hand manipulation. In [15]

the challenges in control and modeling are overcome thanks to recent develop- ments in distributed Reinforcement Learning (RL) and significant computational resources. However, even in the most recent progresses towards general purpose dexterous manipulation, challenges in planning and control still arise in manipulat- ing novel objects. Moreover, frequent failures of the artificial hands after prolonged use demonstrate how complex it is to design a robust and reliable but still flexible instrument that can mimic the human hand.

(17)

1. DEXTEROUS MANIPULATION IN ROBOTICS 5

(a) Baxter. (b) Franka Emika. (c) Yumi. (d) PR2.

Figure 3: Examples of robots equipped with parallel grippers.

Due to the aforementioned challenges, the use of artificial dexterous hands is not widely diffused. In fact, most robots are equipped with simple parallel jaw grippers, with only two jaws (or fingers) that open and close. Fig. 3 shows some examples. The main advantage of parallel grippers over complex dexterous hands is that they are robust, reliable and easy to control, but these qualities emerge at the cost of the hand’s intrinsic dexterity.

Robots equipped with parallel grippers can reliably grasp objects [16], but grasping is just one step in most manipulation tasks. Any of the tasks in Fig. 1 requires further adjustments of the object within the hand. It is impossible to achieve the same level of dexterity by simply opening and closing the ﬁngers to execute a grasp: when tasks involve the use or handling of objects, dexterous manipulation is an essential step to ﬁnalize the overall manipulation task.

Regrasping [17] is a possible solution to change the current grasp conﬁguration.

This approach is often used in stable industrial settings, with a predeﬁned sequence of motions. However, the multiple pick-and-place sequences make this progress slow. Moreover, the task of placing an object on a given surface might itself require changing the current grasp [18]. Hence, it is desirable to provide robots with the ability to change the grasp on an object without the need for releasing it, despite the lack of degrees-of-freedom in simple parallel grippers.

Simple grippers can overcome their limitations by exploiting extrinsic dexter- ity [19], i.e. the enhancement of their poor dexterity by means of external supports, such as gravity, friction and inertial forces. Due to their structure, parallel grippers can only perform simple in-hand manipulation motions, but the combination of many of these simple motions allows the robot to successfully move the object inside the hand (or gripper) with a signiﬁcant change in the grasp conﬁguration.

Thanks to the use of extrinsic dexterity, it is possible to achieve in-hand manipulation despite limited intrinsic dexterity.

In addition to the lack of intrinsic dexterity, simple grippers also lack the sensing capabilities that characterize artiﬁcial hands that mimic the human hand. Tactile

(18)

sensors provide important information to execute contact rich tasks such as manip- ulating an object within the hand. However, most parallel grippers lack sensors, and the applied force is estimated from measured currents, providing noisy and unreliable data. Lacking accurate force or tactile sensors, the robot must rely on diﬀerent information types to properly execute in-hand manipulation. While most works that exploit extrinsic dexterity to manipulate objects with simple grippers depend on force and tactile sensing [20], as well as full observability and high ﬁdelity dynamic models [21], [22], visual feedback is often enough to obtain a successful in-hand manipulation execution [23]–[25]. In addition, visual information is important when reactive controllers based on tactile feedback might not be enough to successfully manipulate the object by accommodating for local shape uncertainties.

The research work discussed in this thesis exploits the concept of extrinsic dexterity and relies on visual perception to design solutions for in-hand manipulation of rigid objects executed with non-dexterous grippers. The full dexterity of a human hand cannot be reproduced, but it is still possible to achieve a large set of tasks without the need for complex hardware. It is also noticeable that human dexterity does not depend only on the redundancy of the hand, but also of the whole body: the tasks in Fig. 1 rely both on moving several ﬁngers and on the aid of a second hand. A similar behavior can also be designed for robots. However, despite dual-arm manipulation and dexterous manipulation are widely studied problems, their mutual exploitation has not received as much attention. This thesis explores diﬀerent solutions for in-hand manipulation, and relates them to the possibility of exploiting a second hand.

Providing dexterity to robots is a step toward their employment outside of repetitive tasks in industrial environments. While a perfect reproduction of human dexterity is still an open challenge, robots can nonetheless execute in-hand manipulation and achieve significant results. Since this skill can be extended to hardware with different levels of complexity, it gives flexibility and adaptability to many systems, even those that are currently in use only for simple tasks.

2 Thesis Contributions

This thesis’ focus is on achieving successful executions of in-hand manipulation tasks with simple grippers using visual feedback. The diﬀerent solutions, tasks and applications are explained in the next chapter and detailed in the included papers.

This section provides a quick overview and highlights the main contributions.

The manipulation process can be divided into three main components: sense, plan and act. In this research work these components are approached as follows.

Sense

The methods explained in this thesis exploit geometric information to plan and execute in-hand manipulation. Moreover, paper D reports a method for successfully planning and executing in-hand manipulation of objects with unknown shapes that

(19)

3. THESIS OUTLINE 7

relies only on the available visual input. In addition to geometric information, paper E exploits the combination of vision and force-torque sensing for changing grasp conﬁgurations while maintaining balance.

Plan

This thesis presents diﬀerent solutions for planning in-hand manipulation, depend- ing on the desired type of the object’s motion (e.g. simple rotation, large trans- lations, etc). Paper C proposes a new structure to represent an object and plan in-hand manipulation sequences to change the grasp conﬁguration. Planning the integration of in-hand manipulation with a full manipulation task is also addressed and reported with successful outcome in paper B.

Act

The exploitation of Reinforcement Learning is a ﬁrst solution to achieve successful executions, reported in paper A. This work demonstrates how properly training in simulation yields policies that are reasonably robust to the mismatch between simulation and reality (a problem that recently gained the attention of the research community as sim-to-real). Moreover, the works reported in papers C and E pro- pose the use of dual-arm robots as an addition to the concept of extrinsic dexterity, where the external support to enhance the poor gripper’s dexterity is external to the gripper itself, but not external to the whole system.

3 Thesis Outline

This thesis is composed of two parts. Part I provides an overview of the concepts and methodologies of in-hand manipulation with limited dexterity; Part II contains the included papers. The rest of Part I is structured as follows.

Chapter 2: Limited Dexterity in Robotic In-Hand Manipulation This chapter provides an overview over the main topic of this thesis. The problem of in-hand manipulation with limited dexterity is analyzed and motivated. The chapter also includes some methodologies and results from papers that have not been included in this thesis but whose outcome is a meaningful addition to the works attached in Part II.

Chapter 3: Summary of Included Papers

Part II includes five papers. These papers are summarized in this chapter, which briefly introduces their main concepts and reports the specific contributions of the author of this thesis for that line of work.

(20)

Chapter 4: Closure.

This chapter presents a discussion over the methodologies and results reported in this thesis, and it examines the possible directions for future work.

(21)

Chapter 2

Limited Dexterity in Robotic Manipulation

1 Changing Grasps

Changing grasps on an object is of fundamental importance to be able to use that object as desired. In fact, each object has specific locations and grasp configurations based on the task to execute, often referred to as grasp affordances [26]–[28]. For instance, a knife can be grasped by the handle for cutting, and by the blade for handing it over to someone. Under the constraint of limited dexterity, the robot must rely on additional supports to properly change the grasp on a given object without having to execute multiple sequences of pick-and-place.

Custom-made end-effectors [29]–[31] are a solution that enhances the intrinsic dexterity of the gripper just enough to robustly execute a given set of in-hand manipulation motions. The drawback, in this case, is that the robot must be equipped with special hardware for the given task, and cannot fully generalize to different in-hand motions. Alternatively, the robot can execute in-hand manipulation to change the grasp on an object by following the concept of extrinsic dexterity. Re- lying on external supports, even a simple parallel gripper can successfully move an object inside its fingers and obtain the desired grasp configuration. In fact, the exploitation of external constraints is a useful resource, and both robots and humans benefit from it during the grasping process [32].

External supports can be, for instance, additional contact surfaces. Prehensile pushing [33] is a method to execute in-hand manipulation within a parallel gripper by pushing the object against properly placed external fixtures. These pushes are controlled to manipulate the translational and rotational motion of the object between the gripper’s fingers and reach the goal grasp. The contact surface can also be exploited when the object is first grasped, so that grasping and in-hand manipulation are jointly executed [34], [35]. This line of work relies on sliding motions [36], and on the well studied mechanics of pushing [37].

9

(22)

10 CHAPTER 2. LIMITED DEXTERITY IN ROBOTIC MANIPULATION

The main drawback of exploiting additional contact surfaces are that they are not always available in the robot’s environment, and that their model, including friction properties, might not be known. In these situations, other external supports can be exploited to enhance the poor dexterity of the gripper: controlled friction, gravity and inertial forces [23], [38]–[40]. Methods that rely on these external supports only require models of the robot and object and not of the whole environment. However, the involved supports (e.g. sliding friction), still diﬃcult to model, and the perception required to execute the developed strategy (e.g. visual tracking) introduce additional challenges. Hence, the outcome of this kind of manipulation is often a limited in-hand motion. An example of such limited motion is pivoting.

2 Pivoting

Pivoting is the action of rotating an object between two ﬁngers to reorient it with a desired angle [41]. Solutions that do not rely on contact surfaces [35], [42], [43]

or custom made fingertips [44]–[46] exploit gravity, controlled friction and inertial forces. For instance, the adaptive controller proposed in [23] exploits gravity acceleration to initiate the object’s motion and adjusts the distance between the fingers to change the torsional friction at the pivoting point and control the rotation. This method was further improved in [20] with the addition of tactile sensing. The main drawback of using gravity to initiate the motion is that this motion can only be directed downwards. In [21], [40] the authors propose an energy-based control to pivot the object in a vertical pivoting plane, i.e. the plane in which the object rotates is parallel to the direction of gravity. Here, the object moves due to inertial forces generated by high acceleration of the robot arm, and the gripper’s fingers are used to exert dissipative torque. Unfortunately, the full observability, fast response time in controlling the gripper and fast visual tracking make this method infeasible with most commonly available systems.

This thesis presents in paper A a method for in-hand pivoting that lowers hardware and modeling needs, with a solution that relies on Deep Reinforcement Learn-

Figure 1: In-hand pivoting with inertial forces.

(23)

3. IN-HAND MANIPULATION BY PUSHING 11

Figure 2: Results of pivoting experiments. The green line shows the goal angle and the area around it represents the tolerance for a successful execution. The blue stars indicate the final angle obtained after the pivoting execution. The pivoting plane, which determines the influence of gravity on the motion, was different for different experiments.

ing. In addition, paper B presents an example of integration of pivoting in an overall robotic manipulation task. The pivoting method used in this case was ﬁrst proposed in [24]. It exploits inertial forces to move the object, without constraining the pivoting plane to be vertical, unlike previous works. In addition, it follows the approach of the gravity based pivoting in [23] to approximate the grasping force and change the torsional friction in lack of tactile feedback. Fig. 1 shows an example of such pivoting with inertial forces. Despite the approximate modeling and low hardware requirement, this pivoting method successfully moves the object to the desired angle, as seen in Fig. 2.

Pivoting is a fast method for in-hand manipulation that allows for reorientation around a single axis. While this reorientation might be enough for some tasks, it can also be combined with sliding actions to obtain a wider range of in-hand manipulation motions. Combination of motions to achieve large scale in-hand manipulation is subject to the same challenges in modeling and perception mentioned previously, but the effect of errors and uncertainties is magnified due to the larger scale. To reduce execution uncertainty and errors, many extrinsic dexterity methods that address more complex in-hand manipulation motions rely on pushing the object against an external contact. This contact provides an additional constraint on the object’s position, and allows for slower motions that can be modeled as quasi-static, reducing the influence of unknown dynamic coefficients on the execution.

3 In-Hand Manipulation by Pushing

Pushing against an external contact provides additional control and support for in-hand manipulation tasks with non dexterous grippers. The external contact

(24)

Figure 3: Changing the grasp by pushing the object against an external surface.

can be the surface on which the object lies before it is picked. This surface poses a constraint on the object that can be exploited to push it and move it inside the gripper while the grasp is executed or at a later stage to adjust it during task execution [34], [35], [42]. Fig. 3 shows an example in which a robot adjusts its grasp by pushing the object against a table. As an alternative, instead of relying on a single contact surface, the robot can exploit several fixtures suitably placed in the environment to enhance its in-hand manipulation capabilities. External fixtures are common in industrial environments, and the interaction with the object is naturally affected by grasping force and friction [47].

The act of pushing a grasped object against the environment to manipulate it is named prehensile pushing in [33]. Here, the authors define push primitives, straight sliding, pivoting and rolling, and provide a solution for each of them by pushing the object against external fixtures. This in-hand manipulation strategy is further improved in [22], [48], where sampling based planners provide the full sequence of pushes to execute for successfully obtaining a desired grasp configuration. The push primitives are combined to achieve a much wider range of possible changes in the initial grasp. Since the pushes against external fixtures are executed in an open loop fashion, an accurate model of all the involved dynamics, including the friction between object, gripper and fixtures, must be known a priori so that the outcome is successful.

Paper C in this thesis introduces a method, named Dexterous Manipulation Graph (DMG), where in-hand manipulation is executed by means of external pushes against the object. To remove the need for accurate friction modeling, this method exploits visual feedback at execution time to adjust the object pushing. The DMG is a graph structure that represents how the gripper’s ﬁngers can move on the object’s surface in terms of rotation and translation. The graph itself does not incorporate the assumption that the motions are executed by pushing the object.

This proves a useful property that makes DMG a versatile tool for planning in-hand manipulation also in diﬀerent situations, as reported in section 6.

Instead of relying on properly placed external ﬁxtures, the execution of the

(25)

4. EXPLOITING DUAL-ARM MANIPULATION 13

experiments in paper C exploits the redundancy of a dual-arm robot to enhance the poor dexterity of a parallel gripper: one gripper is holding the object and the second gripper is used as external pusher.

4 Exploiting Dual-Arm Manipulation

Dual-arm manipulation is a well studied problem in robotics. Several methods have been introduced for bimanual coordination, such as the Relative Jacobian [49], Cooperative Task Space (CTS) [50], [51] and Extended CTS [52], [53].

A common application of dual-arm robots is lifting heavy objects, thanks to the higher payload provided by the two robot arms. Since the object manipulation within the hands is not the goal, these works are developed under the rigid grasp assumption, and the main challenge is in controlling the forces so to reduce the internal stress on the object [54]. However, the possibility of changing the grasp is an important source of redundancy, and it can be exploited to ease diﬀerent manipulation tasks. For instance, sliding contacts can be represented as virtual joints and controlled as if they were part of the overall dual-arm manipulator structure [55].

Moreover, allowing a dual-arm system to switch contacts by releasing the object and regrasp it eases the problem of coordinated motion planning in proximity of singularities [56], [57].

Instead of considering two arms jointly grasping one large object [58], dual-arm robots can be used to change the grasp on smaller objects by means of multiple handovers between their two hands or grippers [59]–[61]. While it is not always true that using two arms for regrasping is better than placing the object back and picking it up again, dual-arm regrasping is more ﬂexible [62].

The second arm in a dual-arm system is also a useful support in the context of extrinsic dexterity: it can be used for pushing the object and achieve in-hand manipulation. This additional support given by the system’s redundancy makes task executions more eﬃcient and removes the need for complex strategies that require previously placed external ﬁxtures or full knowledge of the surrounding environment.

However, the second arm provides not only an additional support, but also an additional constraint. In fact, dual-arm regrasping cannot be executed when grasp poses overlap or lead to collisions between the two robot arms. This complica- tion benefits from the exploitation of in-hand manipulation in conjunction with the regrasping action. Consider the example in Fig. 4. The problem benefits from a solution that combines both in-hand manipulation and regrasping, so that the robot’s grippers reach the desired grasp configuration. Moreover, in-hand manipulation aids a regrasping strategy in case the desired grasp is not reachable by simply closing the fingers on the object; for instance, reaching for points inside a narrow concavity is easier or even only possible by sliding contacts on the object’s surface.

The disconnected graph structure of the DMG method reported in paper C can be exploited to plan combinations of regrasping and in-hand manipulation that lead

(26)

(a) The initial grasp of the gripper on the object is shown in red, on the left, and the desired one in green, on the right. It is not possible to reach the desired grasp by sliding or rotating the ﬁngers on the object’s surface because the ﬁngertip contacts lie on areas between which it is not possible to slide.

(b) Regrasping is needed. A second gripper, shown in blue, helps holding the object while the first gripper releases it. However, this second gripper prevents the first gripper from regrasping at the desired configuration.

(c) The ﬁrst gripper can execute a new grasp, shown in dark green, that is not in collision with the second gripper. Once the second gripper releases the object, the ﬁrst gripper can achieve the desired grasp thanks to in-hand manipulation.

Figure 4: A simple example of a desired grasp on an object that requires regrasping and in-hand manipulation.

Figure 5: Example of planning regrasping and in-hand manipulation with DMG. Fingers shown with color scheme same as above.

(a) Initial conﬁguration.

(b) Second gripper grasps.

(c) First gripper grasps.

(d) Push.

(e) Final grasp.

Figure 6: Changing the grasp conﬁguration with both regrasping and in-hand manipulation.

(27)

5. INCORPORATE UNCERTAINTY 15

the object toward the desired conﬁguration inside the robot’s gripper [63]. Fig. 5 shows a solution obtained exploiting DMG for changing the grasp on a concave object. This kind of solution can be executed by a dual-arm robot that exploits the second gripper as an aid to both hold and push the object, as in Fig. 6.

5 Incorporate Uncertainty

Robot operations are subject to many diﬀerent kinds of uncertainty that aﬀect the executed actions and must be taken into account to achieve the desired result.

A ﬁrst source of uncertainty is the robot itself. Approximations in modeling that do not consider nonlinear terms, friction at the joint level and delays in com- munication are examples of unknown factors that have an inﬂuence on the robot’s actuation and modify the behavior of the system.

Another signiﬁcant source of uncertainty comes from the geometric and dynamic properties of the objects involved in the manipulation task, such as estimating the ﬁngertip’s contact points and modeling the friction between gripper and object. In- hand manipulation solutions strongly depends on those quantities, e.g. the pivoting strategies through inertia and controlled friction. Hence, uncertainties in the robot’s actions and in the object’s properties must be taken into account to achieve a successful execution.

In addition, uncertainty on the surrounding environment also significantly in- creases the challenge in executing a given task, especially once the robot starts operating outside a closed industrial setting. For instance, in an industrial setting it is reasonable to assume perfect knowledge of locations and friction coefficients of all the external fixtures to use for in-hand manipulation by pushing, but the same knowledge cannot be assumed outside of that controlled environment. The exploitation of a dual-arm robot might help lifting some of the challenges in this particular case because the robot is used in place of external fixtures. However, in robotics it is always important to address the different kinds of uncertainty that might arise during the execution of a given task.

In the RL approach for in-hand pivoting (paper A), the uncertainty in robot’s actuation and dynamic coeﬃcients is considered when training the policy by including noise and randomized actuation errors during training. Paper B relies on a diﬀerent strategy for in-hand pivoting that still requires knowledge of the involved parameters. As reported in [24], these parameters do not have to be known a priori, but can be estimated as subsequent attempts to pivot the object are executed.

Moreover, the estimate does not have to match the real coeﬃcients, as long as it is suﬃcient to describe the behavior of the system so that the object can be reoriented with the desired angle.

Uncertainties in actuation and object properties can be addressed also with the use of feedback during the execution. For instance, in paper C the pushes performed by the dual-arm robot are adjusted to match the desired change in object pose by using sensor feedback. However, an additional source of uncertainty that should be

(28)

Figure 7: An example of object reconstruction with generative models for a ham- mer. The shape observed from the camera is red and the reconstruction is in blue.

addressed derives from the available sensors. In particular, in the context of vision- based solutions for planning in-hand manipulation, the used sensors are cameras.

In systems that exploit vision, a major source of uncertainty comes from the lack of full observability over the environment. For instance, the DMG method in paper C assumes that the object’s shape is fully known. However, in situations in which humans can give the robot tasks that involve manipulation of any object, the speciﬁc object’s shape cannot be known a priori. Paper D addresses this issue.

Here, the object is manipulated while only using a partial shape of the object, which is visible from the camera’s point of view. Generative models are exploited to reconstruct the missing part, as in the example in Fig. 7, and the DMG is modiﬁed to build and plan while taking into account the uncertainty in the object’s shape reconstruction.

6 Application Examples

Manipulation is a process, and in-hand manipulation is just one step in the overall execution [64]. All the main steps that compose this process present open research questions and are constantly studied on their own, but their combination is a chal-

(a) Initial grasp. (b) Final placement.

Figure 8: The robot has to place the object inside the cabinet, but to do so it must adjust its initial grasp.

(29)

6. APPLICATION EXAMPLES 17

Figure 9: Improvement of placement objective value over planning time. Chang- ing the grasp on the object via in-hand manipulation signiﬁcantly improves the objective value.

lenge in itself. The integration of in-hand manipulation with diﬀerent applications is one step forward for robots to execute complex tasks in uncertain environments.

Paper B in this thesis presents a simple example of integration of pivoting with an overall task. However, general purpose in-hand manipulation can be exploited and integrated in more complex systems to enhance the manipulation capabilities of a robot.

An example in which in-hand manipulation greatly improves the system’s per- formance is in object placing. Consider a robot that has to place an object in a desired location and in a desired pose. The current grasp on an object might hin- der not just the desired placement, but also any placement at all. Since a sequence of pick-and-place actions to change the grasp is obviously infeasible due to the placement issue, in-hand manipulation is of fundamental importance for the successful outcome of the execution. Fig. 8 shows an example of a situation in which in-hand manipulation is required for properly placing an object, and Fig. 9 shows how changing the grasp with in-hand manipulation improves the performances in that particular example. The method used for placement planning exploits DMG to select grasps reachable through in-hand manipulation, and it is detailed in [18].

Another situation in which the integration of in-hand manipulation beneﬁts the overall task is when the grasp on the object cannot be released due to constraints proper of the task itself. For instance, consider a dual-arm robot carrying a tray with both arms, a case studied in paper E of this thesis with the use of DMG. As objects are placed on the tray, the center of mass changes and the grippers should adapt their grasp to increase the stability of the carried objects, but releasing one of the grasps would lead to a signiﬁcant imbalance in the system. This imbalance weakens the hold on the tray, and can lead to the tray tilting and all of the objects on it being dropped. Hence, in-hand manipulation is a good solution to adjust the bimanual grasp on the tray, because both grippers keep the grasp on it at all times.

(30)

(31)

Chapter 3

Summary of Included Papers

A Reinforcement Learning for Pivoting Task

In arXiv:1703.00472, 2017.

Authors

Rika Antonova*, Silvia Cruciani*, Christian Smith and Danica Kragic.

Abstract

In this work we propose an approach to learn a robust policy for solving the pivoting task. Recently, several model-free continuous control algorithms were shown to learn successful policies without prior knowledge of the dynamics of the task.

However, obtaining successful policies required thousands to millions of training episodes, limiting the applicability of these approaches to real hardware. We developed a training procedure that allows us to use a simple custom simulator to learn policies robust to the mismatch of simulation vs robot. In our experiments, we demonstrate that the policy learned in the simulator is able to pivot the object to the desired target angle on the real robot. We also show generalization to an object with diﬀerent inertia, shape, mass and friction properties than those used during training. This result is a step towards making model-free reinforcement learning available for solving robotics tasks via pre-training in simulators that oﬀer only an imprecise match to the real-world dynamics.

Contribution by the author

Formulated and proposed the problem; run the robot experiments; wrote the paper together with R. Antonova.

19

(32)

20 CHAPTER 3. SUMMARY OF INCLUDED PAPERS

B Integrating Path Planning and Pivoting

In IEEE/RSJ International Conference on Intelligent Robots and Systems, Madrid, Spain, 2018.

Authors

Silvia Cruciani and Christian Smith.

Abstract

In this work we propose a method for integrating motion planning and in-hand manipulation. Commonly addressed as a separate step from the final execution, in-hand manipulation allows the robot to reorient an object within the end-effector for the successful outcome of the goal task. A joint achievement of repositioning the object and moving the manipulator towards its desired final pose saves time in the execution and introduces more flexibility in the system. We address this problem using a pivoting strategy (i.e. in-hand rotation) for repositioning the object and we integrate this strategy with a path planner for the execution of a complex task.

This method is applied on a Baxter robot and its eﬃcacy is shown by experimental results.

Formulated and proposed the problem; designed and implemented the proposed method; run the robot experiments; wrote the majority of the paper.

C Dexterous Manipulation Graphs

In IEEE/RSJ International Conference on Intelligent Robots and Systems, Madrid, Spain, 2018.

Authors

Silvia Cruciani, Christian Smith, Danica Kragic and Kaiyu Hang.

Abstract

We propose the Dexterous Manipulation Graph as a tool to address in-hand manipulation and reposition an object inside a robot’s end-eﬀector. This graph is used to plan a sequence of manipulation primitives so to bring the object to the desired end pose. This sequence of primitives is translated into motions of the robot to move the object held by the end-eﬀector. We use a dual arm robot with parallel grippers to test our method on a real system and show successful planning and execution of in-hand manipulation.

(33)

D. IN-HAND MANIPULATION OF OBJECTS WITH UNKNOWN SHAPES 21

Formulated and proposed the problem; designed and implemented the DMG method;

run the robot experiments; wrote the majority of the paper.

D In-Hand Manipulation of Objects with Unknown Shapes

Submitted to IEEE International Conference on Robotics and Automation, 2020.

Authors

Silvia Cruciani, Hang Yin and Danica Kragic.

Abstract

This work addresses the problem of changing grasp conﬁgurations on objects with an unknown shape through in-hand manipulation. Our approach leverages shape priors, learned as deep generative models, to infer novel object shapes from partial visual sensing. The Dexterous Manipulation Graph method is extended to build upon incremental data and account for estimation uncertainty in searching a sequence of manipulation actions. We show that our approach successfully solves in-hand manipulation tasks with unknown objects, and demonstrate the validity of these solutions with robot experiments.

Formulated and proposed the problem; designed and implemented most of the method for in-hand manipulation, with the exception of training deep generative models; run the robot experiments; wrote the majority of the paper.

E Discrete Bimanual Manipulation for Wrench Balancing

Submitted to IEEE International Conference on Robotics and Automation, 2020.

Authors

Silvia Cruciani*, Diogo Almeida*, Danica Kragic and Yiannis Karayiannidis.

Abstract

Dual-arm robots can overcome grasping force and payload limitations of a single arm by jointly grasping an object. However, if the distribution of mass of the grasped object is not even, each arm will experience diﬀerent wrenches that can

(34)

22 CHAPTER 3. SUMMARY OF INCLUDED PAPERS

exceed its payload limits. In this work, we consider the problem of balancing the wrenches experienced by a dual-arm robot grasping a rigid tray. The distribution of wrenches among the robot arms changes due to objects being placed on the tray. We present an approach to reduce the wrench imbalance among arms through discrete bimanual manipulation. Our approach is based on sequential sliding motions of the grasp points on the surface of the object, to attain a more balanced conﬁguration.

Proposed the problem; designed and implemented the in-hand manipulation planning and robot trajectory generation components of the system; modeled the problem, run the robot experiments and wrote the paper together with D. Almeida.

(35)

Chapter 4

Closure

1 Conclusions

This thesis addresses the problem of in-hand manipulation with limited dexterity.

This problem is challenging due to the lack of degrees-of-freedom available in the robot’s grippers. The system must rely on extrinsic dexterity, i.e. external supports, to be able to move an object within the hand. The methods proposed and reported in part II exploit visual input as a mean to plan and control the manipulation execution, showing that vision-based systems, even without additional sensors, are often enough to change the grasp on the object as desired.

For simple in-hand manipulation tasks, where the change in grasp does not require sliding contacts, pivoting is a viable solution. This thesis proposes two possible solutions for in-hand pivoting, one of which based on deep RL, and demonstrates the application of pivoting in a manipulation task, showing that successful integration between all of the system’s components is possible and that in-hand manipulation through pivoting can be a useful asset in the execution of an overall task. The two pivoting methods have been designed to cope with inexact friction modeling and with the lack of high-performing hardware tailored to the speciﬁc task.

While pivoting is a simple but flexible solution for in-hand manipulation, in many cases the sole rotation of the object between the fingers is not enough to obtain the desired grasp. To plan more complex changes in grasp through in-hand manipulation, this thesis presents the Dexterous Manipulation Graph. The DMG is a new representation of the object’s shape as a graph that contains information on how the gripper’s fingers can move along its surface, in terms of translation and rotation. This graph is used to plan a sequence of motions for the gripper’s fingers that lead the object to the desired grasp configuration. The motions can be executed by means of external pushes against the environment or against a second robot arm, but they can also exploit different external supports. The difference in execution does not affect the DMG structure, but it changes how the graph is searched while planning for a given task.

The DMG is also a useful tool to plan tasks that involve changing the grasp 23

(36)

24 CHAPTER 4. CLOSURE

on the object but are not limited to in-hand manipulation. By exploiting the graph structure, manipulation tasks can be solved as a combination of in-hand manipulation and regrasping. Furthermore, the DMG can be used also in the case of two grippers jointly grasping the object. In this case as well, the DMG structure is not altered, but the graph search to plan in-hand manipulation is modiﬁed to take into account the speciﬁc constraints of the task.

In addition, the DMG is extended into a method that removes the need for a full model of the object’s shape. The graph structure is generated from an estimate of the object’s shape obtained through a combination of visual sensing and generative models. The new graph includes information on what parts of the objects were visible, and what instead are only an uncertain reconstruction, and this information is used to plan in-hand manipulation solutions that minimize the inﬂuence of the invisible part of the object in the execution.

In the context of limited dexterity, dual-arm manipulation is introduced as a useful source of redundancy to compensate for the lack of degrees-of-freedom of simple parallel grippers. Dual-arm robots are used to execute in-hand manipulation, both non-prehensile, i.e. one gripper is simply pushing the object, and prehensile, i.e. both of the two grippers are grasping the object while executing in-hand manipulation.

This thesis presents many examples of manipulation tasks that can be solved with the proposed methods. Those application examples show that the step of changing the grasp on an object without releasing it is an important component of the manipulation process. A successful plan and execution of this step is of great importance to properly solve complex tasks, and it can be carried out without the need for complex or custom made hardware. Systems with limited dexterity are capable of executing in-hand manipulation and can successfully execute complex tasks when given the possibility of exploiting additional supports.

2 Future Work

The problem of in-hand manipulation presents several interesting challenges, and there are many possibilities to address them that can branch oﬀ from the works presented in this thesis.

Including vision as a more fundamental component in the system can be a first step forward. For instance, the DMG structure generated on the partially visible object can be adapted and modified as the object is manipulated. In fact, while the object moves inside the gripper, more parts of it become visible. Augmenting the graph structure while the manipulation task is executed improves the quality of the manipulation itself as it allows for corrections of errors due to wrong estimates of the object’s shape, and it provides a useful tool for subsequent manipulations with the same object. Having the possibility to adapt to the objects that are present in the environment is a useful tool for robots that have to be employed in homes or offices.

Vision-Based In-Hand Manipulation with Limited Dexterity

Contents

Overview

Introduction

Limited Dexterity in Robotic Manipulation

Summary of Included Papers

Closure