Abstract
A long-horizon manipulation task consists of primitive skills and the transitions connecting them. Learning skills that can be executed in sequence requires supervision for both, which is costly to collect on a robot. Human videos offer a scalable source of demonstrations, but transferring them to robots requires extracting robot-compatible observations and actions together with their skill structure. We present DexStitch, a framework that cuts long-horizon human videos into skills and transitions, converts them into aligned robot observation–action–skill data, and stitches the learned skills into a requested sequence. A skill classifier first segments the videos into manipulation skills and transitions. DexStitch then converts the human demonstrations into robot observations and actions while retaining their corresponding skill labels. During conversion, per-finger human–object contacts guide dexterous hand retargeting and robotic inpainting, preserving contact geometry and finger–object occlusion ordering. To stitch the learned skills into a sequence, a shared skill-conditioned policy is pretrained on the resulting synthetic data and finetuned on a mixture of synthetic data and a small set of single-skill teleoperation episodes. Conditioning on the current and next skills guides each transition toward the following skill, while predicted skill progression enables closed-loop switching. On a held-out session, the classifier agrees with manual annotations on 92.4% of frames. Across four household skills, DexStitch achieves 80.0% single-skill success and 77.5% sequence success.
Skill and action supervision from human video
Long-horizon human videos become robot-domain observation–action pairs that keep their frame-level skill annotations, supervising both the manipulation skills and the transitions between them — from RGB video alone, with no object meshes and no simulator.
Contact-aware human-to-robot conversion
Per-finger human–object contact is a shared cue for both retargeting and visual compositing, preserving multi-finger grasp geometry while resolving finger–object occlusions during robot rendering.
Sequential composition of reusable skills
The converted data yields an explicit, reusable repertoire of robot-executable skills, transitions included. The policy conditions on the current and next skills for target-aware transitions, and predicts skill completion for closed-loop composition.
Video
static/videos/dexstitch.mp4
Contact-Aware Human–Robot Data Conversion
DexStitch converts egocentric human video into robot-domain observation–action pairs in three stages: estimating hand motion and contact, retargeting that motion to the robot, and rendering the robot into the frames in place of the human. Because conversion is frame-wise, every converted frame keeps the skill label of its source frame.
1. Hand pose and contact estimation
HaWoR fits the MANO hand model in camera coordinates, giving hand keypoints and the wrist pose per frame. HaCo then predicts a contact probability over the MANO vertices; vertex probabilities are pooled per finger and thresholded with hysteresis, yielding temporally stable per-finger contact states and the 3D contact points of each contacting finger.
2. Contact-aware retargeting
DexPilot initializes the robot hand by matching wrist-to-fingertip vectors from the MANO keypoints. Because that initialization need not preserve the demonstrated contacts, each contacting finger is refined by minimizing the distance from every human contact point to the nearest robot fingertip surface point with an aligned normal, with a quadratic regularizer toward the initialization and L-BFGS-B as the solver. For the arm, a fixed camera-to-base transform is calibrated once with EasyHeC, inverse kinematics on the wrist pose gives the arm joints, and the stacked per-frame configurations are smoothed with a median filter followed by a Savitzky–Golay filter.
3. Inpainting and compositing
SAM2 tracks masks for the human hand and arm and for the manipulated objects. ProPainter inpaints the human mask while protecting the object mask, and the robot is rendered at the retargeted configuration using the calibrated extrinsics and the original camera intrinsics. Overlaying the robot naively would place grasping fingers in front of the object, so for wrap grasps the thumb is assumed in front and the contacting non-thumb fingers behind: their 3D contact points are projected into the image and dilated into a connected region, and wherever that region overlaps the object mask, object pixels from the source frame are composited over the rendered robot.
Skill Learning and Composition
The skill set contains both manipulation behaviors and the transition behavior connecting them, with transition treated as an ordinary skill label. A classifier recovers the skill sequence from human video; a single skill-conditioned policy executes those skills and decides when to advance.
Skill classifier
A pretrained V-JEPA 2 video encoder stays frozen and a small MLP head reads its representation. The encoder emits one token per pair of frames sampled at 4 FPS, so the head sees the video at 2 Hz; it reads a causal window of the eight most recent tokens — four seconds of context — and predicts the skill at the latest one. Freezing the encoder leaves only 296K trainable parameters, which is what makes the classifier fittable from the handful of annotated sessions available. Frame-wise predictions are smoothed with temporal filters and consolidated into an ordered sequence of skill segments.
Skill-conditioned policy
Given the current robot observation, current skill, and next skill, a shared policy predicts an action chunk of H steps together with its skill progression. The current skill specifies the behavior to execute; the next skill provides context for how that behavior should terminate and connect to what follows. The policy is instantiated with Diffusion Policy over the robot camera observation and the current arm and hand configuration. It is first trained exclusively on the synthetic robot data — which already covers the complete demonstrated sequence, so transitions are learned with no teleoperated demonstration of them — then trained further on a mixture of synthetic data and a small set of teleoperated episodes that grounds the behaviors in the robot's own execution.
Compositional inference
Given an ordered skill sequence, whether from a human prompt video via the classifier or from a script, the policy executes the entries in order, conditioned on the current and next pair. Execution advances through the sequence by the policy's own skill predictions rather than a fixed duration: when it consistently predicts the next skill, the current skill is considered complete and the conditioning advances. Predictions are aggregated over consecutive time steps so that isolated predictions cannot trigger premature switches.
Results
Evaluation runs on an RB5-850E arm with an xHand and a D435i camera across four household skills. Roughly 240 long-horizon human demonstrations are converted into robot trajectories, and about 8 teleoperated demonstrations per skill (~32 total) are collected for finetuning. The Diffusion Policy consumes RGB, proprioception, and current/next skill labels; it is pretrained on converted data and finetuned on mixed data with a 90% teleoperation ratio unless noted otherwise.
| Method | Single-skill | Sequence | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Green | Red | Cup | Stack | Avg. | Green | Red | Cup | Stack | Total | |
| UniSkill | 80 | 20 | 20 | 60 | 45.0 | 0 | 0 | 0 | 0 | 0.0 |
| Ours (DexStitch) | 100 | 100 | 40 | 80 | 80.0 | 80 | 70 | 80 | 80 | 77.5 |
UniSkill reaches 45.0% on individual skills but 0% on sequences, as it has no explicit mechanism for transitions between skills. Both methods use the same synthetic robot data, the same 120k-step pretraining, and the same teleoperated finetuning, isolating the effect of the skill representation — in particular, transition modeling.
| Method | Acc. ↑ | Macro-F1 ↑ | Edit ↑ | F1@50 ↑ |
|---|---|---|---|---|
| Qwen2.5-VL-7B (zero-shot) | 65.8 | 66.0 | 70.3 | 41.0 |
| InternVL3-8B (zero-shot) | 57.8 | 56.7 | 47.5 | 28.5 |
| V-JEPA 2 + MLP (Ours) | 92.4 | 92.7 | 99.6 | 99.0 |
The baselines get over half of the frame labels right, but their predictions fragment: the roughly 30-point accuracy gap grows to a two-to-threefold gap in F1@50. DexStitch recovers 242 segments against 241 ground-truth segments and matches the skill order in all but one episode. The largest gap is on transitions (F1: 0.89 for DexStitch, 0.41 for Qwen, 0.18 for InternVL), suggesting the baselines struggle more with temporal state changes than with recognizing manipulation skills.
static/images/confusion_matrix.png
| Variant | Success |
|---|---|
| Training data | |
| w/o Human data | 35.0 |
| w/o Teleop data | 5.0 |
| Skill conditioning | |
| w/o Transition | 10.0 |
| w/o Next Skill | 7.5 |
| Finetuning ratio | |
| 50% teleop | 10.0 |
| 75% teleop | 20.0 |
| 100% teleop | 50.0 |
| Ours (90% teleop) | 77.5 |
Neither data source suffices alone: the synthetic robot data provides broad skill coverage, while the teleoperated data grounds the behaviors in the robot's own execution. Removing the explicit transition skill or the next-skill conditioning degrades most at transitions, where the same transition must lead to different subsequent behaviors depending on which skill follows. Task success peaks at a 90% teleoperation ratio and drops to 50.0% at 100%, showing that retaining a small fraction of synthetic robot data during finetuning beats finetuning on teleoperated data alone.
Conversion quality
Robot rollouts
static/videos/rollout_*.mp4
Limitations
- Grasp taxonomy. Compositing assumes a wrap grasp, with the thumb in front of the object and the other fingers curled behind it, so pinch and top-down grasps fall outside that rule. Covering a wider grasp taxonomy is left to future work.
- Fixed evaluation ordering. Composition is evaluated on a single fixed four-skill order. Evaluating orderings held out from training would test the compositional claim directly.