InHandWriting

Tactile-Driven Dexterous In-Hand Writing
via Extrinsic Contact Sensing

Overall pipeline

For more details about this study, please refer to the appendices.

Abstract

Dexterous in-hand manipulation, especially involving interactions between grasped objects and external environments, remains a formidable challenge in robotics. This study tackles the complexities of in-hand manipulation under extrinsic contact through a representative three-finger handwriting task. We propose a hybrid arm-hand coordination framework that combines reinforcement learning with compliance control, offering both flexibility and robustness. Leveraging tactile sensors embedded in each finger, our tactile-driven estimation model dynamically predicts in-hand object pose and external contact, eliminating the need for fixed contact states. The proposed framework is first validated in simulation, where it successfully executes diverse writing tasks with accurate contact sensing. Sim-to-Real transfer is achieved through systematic calibration of finger joints and tactile sensors, supported by domain randomization. Realworld experiments further demonstrate the system’s adaptability to writing tools with varying physical properties—such as radius, length, mass, and friction—while maintaining stability across different trajectories. This work advances robotic manipulation capabilities in unstructured environments.

Videos

The following videos demonstrate the performance and generalization capabilities of the proposed framework for tactile-driven dexterous in-hand writing.

Real-World Experiments

This video covering:

  1. Experiment Setup: overview of the hardware, including the three-finger robotic hand and the ABB IRB1100 robotic arm.

  2. Generalization Tests: evaluation across various writing tools, writing surfaces, and writing trajectories.

  3. Ablation Study: comparison of different policy variations, analyzing the impact of compliant control and extrinsic contact perception on task performance.

Trained Trajectories in Simulation

This video showcases the trained trajectories in simulation, including circles, straight lines, heart shapes, figure eights, and a spiral. The blue lines represent the actual trajectories generated by the system, while the red lines indicate the target trajectories, highlighting the system's accuracy in following diverse patterns.

Generalization abilities in Simulation

This video presents the framework’s generalization to unseen trajectories in simulation, specifically showcasing various digits.