Learning Dexterous Manipulation Skills from Imperfect Simulations

Elvis Hsieh*, Wen-Han Hsieh*, Yen-Jen Wang*, Toru Lin, Jitendra Malik, Koushil Sreenath†, Haozhi Qi†
* Equal contribution (listed in alphabetical order). † Equal advising.
UC Berkeley
pdf arxiv code (sim-to-real) code (skill-teleop)
TLDR:

We learn rotational motion in a simplified simulator, use it for skill-based teleoperation to collect multisensory data, and train a policy that performs screwdriving and nut-bolt fastening in the real world.

teaser

Stage 1: Simulation

Stage 1

We train a transferable rotational motion policy in simulator.

Simulation objects
Simulation objects
Simulation Training

Stage 2: Teleoperation

Stage 2

We use the trained rotational motion policy for skill-based teleoperation to collect multisensory data.

Screwdriver
Bolt-nut

Stage 3: Multisensory Policy

Stage 3

We distill the multisensory data into single policy for screwdriving and nut-bolt fastening.

Tactile-Aware
Non-Tactile
Tactile-Aware
Non-Tactile

Ablation: Tactile Feedback with Temporal Context

We compare the performance of the tactile-aware and non-tactile policies without temporal context below. Tactile feedback is best to interpreted with temporal context.

Tactile-Aware
Non-Tactile
Tactile-Aware
Non-Tactile

Tactile Visualization

Screwdriver
Bolt-nut

Acknowledgements

This work is supported in part by the program "Design of Robustly Implementable Autonomous and Intelligent Machines (TIAMAT)", Defense Advanced Research Projects Agency award number HR00112490425. We thank Mengda Xu for their valuable feedback.

Website inspired by PRoPE