3D HAMSTER: Bridging Planning and Control in Hierarchical Vision Language Action Models through 3D Trajectory Guidance

Hierarchical Vision-Language-Action (VLA) models split manipulation into a high-level planner and a low-level policy. Today's strongest low-level policies operate on 3D point clouds, yet existing planners predict only 2D pixel trajectories — forcing each waypoint to inherit the depth of whatever surface lies beneath it.

3D HAMSTER removes that mismatch. We build:

1. A depth-aware VLM planner — augmented with a geometry encoder and a dense depth-reconstruction objective — that predicts metric 3D end-effector trajectories. Visualized in §02.
2. A 3D-trajectory-conditioned point-cloud policy (rectified-flow, 3DFA) trained to roll out the planner's predictions on real hardware. Real-world rollouts in §03.

Our depth-aware VLM predicts metric 3D end-effector trajectories directly — every waypoint a real point in the scene, not a pixel whose depth must be inferred at execution. The two views below let you scrutinize that geometry yourself: paired renders that lock the same trajectory to two camera angles, and free-orbit point clouds you can explore from any viewpoint. Across DroidSpatial-Bench and our real-world benchmark, our planner stays tight on ground truth where strong general-purpose VLMs — RoboBrain 2.5 and Gemini 3.0 Pro — visibly drift in depth.

The planner takes a single RGB-D observation and a language instruction, and directly outputs a metric 3D end-effector trajectory — a real path in the scene. Rotating it makes clear the prediction lives in 3D, not on the image plane.

How it is trained. We build on Qwen3-VL-8B and add a dedicated depth encoder plus a dense depth-reconstruction loss that keeps the model's internal geometry metrically faithful. Training runs in two stages — Stage 1 aligns the depth features with the VLM space (encoders frozen), Stage 2 fine-tunes for trajectory prediction with LoRA. The planner learns from a mixture of 3D-capability data (RGB-D trajectories and spatial reasoning) and 2D preservation data (RGB-only pointing, detection, and VQA) that keeps its general vision-language ability intact.

Three real-world tasks, three different scenes — each rendered with the ground-truth trajectory and the predictions from our VLM, RoboBrain 2.5, and Gemini 3.0 Pro overlaid on the same point cloud. Our planner threads the correct 3D path in every scene, while the baselines drift in depth or miss the target surface.

The same comparison on DroidSpatial-Bench — three benchmark scenes where our planner threads the gripper onto the correct target surface, while RoboBrain 2.5 and Gemini 3.0 Pro drift in depth.

Don’t take our word for it — orbit freely and check the geometry from any angle. Each viewer shows our 3D HAMSTER prediction (a red→blue gradient trajectory) on the scene point cloud. From every viewpoint, our trajectory stays glued to the correct object surface.

Drag to orbit, scroll to zoom, right-click to pan. Use ← / → (or the buttons) to cycle scenes within a carousel.

We roll out 3D HAMSTER on a real Franka Panda across three task families and the five condition shifts from our quantitative protocol (§05): In-distribution, Language, Spatial, Visual, and Multiple. Pick a task to scan all five conditions at a glance, then select one to inspect its rollout next to the predicted 3D trajectory in an interactive point cloud.

The same real-robot scenes run head-to-head against strong baselines. 3D HAMSTER’s metrically grounded guidance holds where surface-level 2D guidance drifts.

Same four methods on the real Franka Panda, evaluated under shifted lighting conditions. 3D depth cues remain stable across illumination changes while surface-level 2D guidance degrades.

On the real Franka Panda, the target object is placed inside a visually crowded scene populated with distractor objects. The instruction uses an unseen referring expression, further stressing language grounding. 3D HAMSTER's metrically grounded trajectory guidance keeps the policy on-task while 2D baselines drift onto visually salient distractors.

HAMSTER predicts 2D pixel trajectories that cling to the scene surface (the “graffiti effect”), so execution drifts off the target. 3D HAMSTER predicts metric 3D trajectories that stay geometrically grounded — turning the same failures into successes.

Our depth-encoder-augmented VLM produces more accurate 3D trajectories than strong baselines including proprietary VLMs and open-source models.

Per-variation success rates averaged across 11 tasks on Colosseum, which stress-tests policies under 14 perturbation axes. 3D guidance provides both appearance invariance and geometric robustness.

Success rates (%) on real Franka Panda arm across three task families. 3D HAMSTER achieves the highest average success across all tasks, with the largest gains under visual and spatial distribution shifts.