Report

Teaser, summary, work performed and final results

Periodic Reporting for period 1 - CLOTHILDE (CLOTH manIpulation Learning from DEmonstrations)

Teaser

Textile objects pervade human environments and their versatile manipulation by robots would open up a whole range of possibilities, from increasing the autonomy of elderly and disabled people, housekeeping and hospital logistics, to novel automation in the clothing internet...

Summary

Textile objects pervade human environments and their versatile manipulation by robots would open up a whole range of possibilities, from increasing the autonomy of elderly and disabled people, housekeeping and hospital logistics, to novel automation in the clothing internet business and upholstered product manufacturing. Although efficient procedures exist for the robotic handling of rigid objects and the virtual rendering of deformable objects, cloth manipulation in the real world has proven elusive, because the vast number of degrees of freedom involved in non-rigid deformations leads to unbearable uncertainties in perception and action outcomes.

This proposal aims at developing a theory of cloth manipulation and carrying it all the way down to prototype implementation in our Lab. By combining powerful recent tools from computational topology and machine learning, we plan to characterize the state of textile objects and their transformations under given actions in a compact operational way (i.e., encoding task-relevant topological changes), which would permit probabilistic planning of actions (first one handed, then bimanual) that ensure reaching a desired cloth configuration despite noisy perceptions
and inaccurate actions.

In our approach, the robot will learn manipulation skills from an initial human demonstration, subsequently refined through reinforcement learning, plus occasional requests for user advice. The skills will be encoded as parameterised dynamical systems, and safe interaction with humans will be guaranteed by using a predictive controller based on a model of the robot dynamics. Prototypes will be developed for 3 envisaged applications: recognizing and folding clothes, putting an elastic cover on a mattress or a car seat, and helping elderly and disabled people to dress. The broad Robotics and AI background of the PI and the project narrow focus on clothing seem most appropriate to obtain a breakthrough in this hard fundamental research topic.

Work performed

This initial period has been very productive, both in terms of setting up the conditions for carrying out the proposed research, as well as for the research results already obtained.

The main highlights are listed below:

- After a long negotiation with UPC, we have succeeded in setting up a new Perception and Manipulation Laboratory, within which we have installed an Assisted Living Facility with a kitchen-dining-living room and a bedroom, where we have started experimentation with such cloth manipulation tasks as putting a tablecloth, and grasping, folding and storing garments.

- As specified in the proposal, two Tiago robots have been acquired, which we have equipped with extra sensors and in-house software and applied to the mentioned experiments within the Assisted Living Facility.

- Four calls for postdocs have been issued and, although two of the selected researchers got better offers from their universities when they were about to sign, we managed to hire 3 excellent postdocs (Drs. Borràs, Colomé, and Strazzeri). Likewise, we hired two very good PhD students (F. Coltraro and J.A. Delgado).

- We have maintained a weekly CLOTHILDE seminar, in which we have discussed both state-of-the-art papers and own work by the four established subgroups within the project (Topology, Perception, Manipulation and Learning), with which the PI has held regular coordination meetings.

- Building on our previous work on dynamic movement primitives and reinforcement learning, important results regarding ways of reducing the dimensionality of high-dimensional configuration spaces have been attained and published in prestigious journals like IEEE Transactions on Robotics and IEEE Robotics and Automation Letters.

- Several results on cloth representation and simulation, as well as on perception, manipulation and planning of textile objects are underway. For example, an extension of ROSPlan to handle probabilistic planning has been devised, and an in-depth analysis of grippers and grasping primitives for manipulating clothing items is under review.

- Concerning Ethics, a book by the PI has been published by MIT Press together with online materials to teach a course on Ethics in Social Robotics and Artificial Intelligence. Several universities in the US and Spain are using the book in some master and undergraduate degrees.

- Three external advisors (Drs. Pokorny, Feix and Aksoy) have separately visited us, with whom we have initiated promising interchanges and collaborations. We have established contact with two companies in the textile sector to assess their needs in the manipulation of deformable materials. In particular, we have visited a large warehouse that one of these companies has near Barcelona.

- We have set up a CLOTHILDE website (https://clothilde.iri.upc.edu/), where extensive information on our team, publications and outreach activities (see specially “Media appearances” and “RoboEthics”) can be found.

Final results

The main six challenges put forward in the proposal are:

[TOP-1] Defining a task-oriented topological cloth representation tailored to robotic manipulation.

[ROB-1] Devising tools for estimating possibly tangled, folded, partially-occluded cloth configurations.

[TOP-2] Formulate the (learned) cloth manipulation skills in terms of the developed task-oriented topologic representation and cast them as planning operators (with preconditions, action effects and associated probabilities) amenable to being used in a probabilistic planner.

[ROB-2] Diagnose failures in a way that permits providing guidance to the user as to what knowledge (cause-effects, actions) is missing to complete the plan.

[ROB-3] Developing an integrated system for teaching basic cloth manipulation skills (first one-handed, then bimanual) to a robot.

[TOP-3] Exploiting recent computational topology methods to extract task-relevant motion features from the high-dimensional manifolds obtained by learning from demonstration.

The TOP challenges are mostly theoretical and are being addressed using computational topology and geometry procedures, whereas the ROB challenges entail developing hardware tools and algorithms for robot perception, manipulation and learning to exploit the mathematical concepts and procedures being developed.

In this initial period, steady progress has been made in relation to challenges TOP-1, ROB-3 and TOP-3, as pointed out in the highlights 5 and 6 in the preceding section. At this stage, we think we are in a good standpoint to attain all the objectives put forward in the proposal to address the aforementioned six challenges.

Website & more info

More info: http://www.clothilde-project.eu.