FlashDexRetarget: Accelerating Dexterous Manipulation Data Generation through Multi-Motion Retargeting

Retarget a Whole Collection with One Policy

Physics-based retargeting makes a human demonstration executable by a robot hand: the robot must reproduce the demonstrated object motion through its own contacts, under its own dynamics.

Human demonstration

Reference motion

Retargeted robot trajectory

Existing sampling-based and RL-based retargeting methods optimize each demonstration independently, so their cost grows linearly with the number of demonstrations. FlashDexRetarget instead studies multi-reference tracking: a single reference-conditioned policy is trained jointly across all reference motions, amortizing training over the whole dataset, and every successful rollout is stored as robot data.

FlashDexRetarget at a glance: N reference motions, one policy, retargeted data, and success rate versus GPU hours — **FlashDexRetarget at a glance.** Left and right hand policies are jointly trained across reference motions, avoiding repeated optimization for individual demonstrations, and successful rollouts are stored as robot data. The right plot shows cumulative conversion success rate versus total GPU hours on a logarithmic scale.

Method

Reference trajectories are sampled from the dataset and assigned to parallel simulation environments. At every step the policy sees the current simulated state together with the next reference state and predicts the robot action.

Interaction-aware observations

Observation

Object point clouds

One policy has to handle many objects, so it observes their shape instead of memorizing one: 128 uniformly sampled surface points for the current simulated pose and for the next reference pose, expressed in the wrist-local frame.

Current cloudsimulated object

Reference cloudfrom the demonstration

Observation

Hand–object distance features

To describe the local hand–object relationship, including before contact, we query a signed distance field for the distance and surface normal from each fingertip and the wrist to the object, in the wrist frame.

Fingertip-to-surface vectorsduring a rollout

Observation

Future reference conditioning

A single next frame cannot tell where the motion is going. We encode the reference hand and object states from $t+1$ to $t+K$ ($K = 10$), relative to the current wrist frame, into a 128-d latent with a temporal encoder.

Next 10 reference framesreference hand & object poses

Geometric tracking rewards

Reward

Object point-cloud matching

Instead of weighting translation and rotation errors by hand, we compare corresponding surface points and average the $k = 3$ largest errors, a single metric that captures both. For a uniform reward signal across object sizes, the object-frame surface points are first rescaled to a common radius $\rho = 0.058$ m, so that a 30° rotation yields a 3 cm error, matching the success thresholds:

$$e^{\text{obj}}_t = \frac{1}{k}\sum_{m \in \mathcal{I}^{(k)}_t} \big\lVert \tilde{\mathbf{P}}^{\text{cur}}_{t+1,m} - \tilde{\mathbf{P}}^{\text{ref}}_{i,t+1,m} \big\rVert_2, \qquad r^{\text{obj}}_t = \exp\!\big(-e^{\text{obj}}_t / \sigma_{\text{obj}}\big), \qquad \tilde{\mathbf{p}}_m = \rho\,\frac{\mathbf{p}_m}{\max_n \lVert \mathbf{p}_n \rVert_2}$$

with $\sigma_{\text{obj}} = 0.1$ m.

Worst-3 point pairscurrent vs. reference

During a rolloutpoint-cloud matching

Reward

Reference distance matching

Reconstructed contacts are noisy: fingers float above the object or sink into it, so binary contact labels are unreliable. Instead, each robot fingertip should be at least as close to the object as the human fingertip was:

$$\varepsilon^{(f)}_t = \max\!\big(0,\; d^{(f)}_t - d^{\text{ref},(f)}_t\big), \qquad r^{\text{int}}_t = \frac{1}{5}\sum_{f=1}^{5} \exp\!\big(-\varepsilon^{(f)}_t / \sigma_{\text{int}}\big)$$

with $\sigma_{\text{int}} = 0.01$ m. A hand-tracking term and regularization complete the per-hand reward.

Reference distanceshuman fingertips

Robot distancesrobot fingertips (SDF)

Learning

Architecture

Per-hand actor–critic

With one shared critic, the rewards of both hands collapse into a single scalar. Each hand therefore gets its own actor and critic, trained with its hand-specific reward. Both actors observe the full bimanual state but control only their own hand, staying physically coupled through the simulation.

RL algorithm

Scaled off-policy learning

Many references share one policy, so experience is too valuable to discard. We adopt the off-policy algorithm FlashSAC and scale it to the broader multi-reference state distribution.

Replay buffer 10M → 50M Critic width 256 → 1024

Results

Higher success at a fraction of the compute

We compare against Do as I Do, DexMachina, and CHORD on 50 reference motions (25 single-object from HOT3D and 25 two-object from TACO and OakInk2), retargeted to XHand and Sharpa Wave Hand. All methods are evaluated in IsaacSim on NVIDIA RTX 3090 GPUs. Each single-reference run stops once a successful rollout is found, and GPU-hours are summed over all motions, including the full cost of unsuccessful runs; FlashDexRetarget trains one policy for 300M environment steps.

Method	Motion	SR_SPIDER ↑	SR_MT-Obj ↑	SR_MT ↑	MOPE (mm) ↓	MORE (rad) ↓	GPU-h ↓
(a) XHand
Do as I Do	Single	0.36	0.04	0.00	47.32	0.28	66
DexMachina	Single	0.48	0.32	0.12	37.49	0.29	447
CHORD	Single	0.46	0.02	0.00	55.70	0.16	2,847
FlashDexRetarget^†	Multi	0.90	0.86	0.86	10.95	0.25	32
(b) Sharpa Wave Hand
Do as I Do	Single	0.64	0.10	0.10	58.33	0.24	62
CHORD	Single	0.50	0.22	0.00	60.07	0.19	3,314
FlashDexRetarget	Multi	0.72	0.60	0.60	17.28	0.29	36

SR_SPIDER: object position / rotation error within 10 cm / 0.5 rad, averaged over the objects of the active hands. SR_MT-Obj: object within 3 cm / 30°. SR_MT: additionally fingertip and hand-joint position errors within 6 cm and 8 cm. MOPE / MORE: mean object position / rotation error over trajectories that satisfy SR_SPIDER.
^†Uses object-size normalization in the point-cloud matching reward. It is applied only to the XHand result; the Sharpa Wave Hand, ablation, and scaling results do not yet use it.

What makes multi-reference retargeting work

Ablation curves: observation, reward, architecture, RL algorithm — Cumulative number of successfully converted reference motions (out of 50) over 300M environment steps; a reference counts once a rollout satisfies SR_MT-Obj.

Observation. Encoding future reference frames accumulates successful trajectories faster and converts more references than next-frame conditioning alone.
Reward. Removing both point-cloud matching and the interaction reward (with their observations) hurts most; either one helps, and both together is best.
Architecture. Separate left- and right-hand actor–critic pairs improve both learning speed and final success over a unified actor–critic.
RL algorithm. PPO stays near zero within 300M steps while default FlashSAC converts about 32%; a larger buffer or a wider critic each help, and combining both is best.

Scaling to larger reference collections

Cumulative successful clips for 200, 500 and 1,000 training motions

When one shared policy is trained on collections of 200, 500, and 1,000 motions (600M environment steps each), larger collections yield more successfully retargeted motions within the same environment-interaction budget.

Sharing training across more references therefore increases the data-generation yield of a single training run: more demonstrations in, more robot data out.

FlashDexRetarget

Video

Retarget a Whole Collection with One Policy

Method

Interaction-aware observations

Object point clouds

Hand–object distance features

Future reference conditioning

Geometric tracking rewards

Object point-cloud matching

Reference distance matching

Learning

Per-hand actor–critic

Scaled off-policy learning

Results

Higher success at a fraction of the compute

What makes multi-reference retargeting work

Scaling to larger reference collections

Real-World Replay

BibTeX