Procedural Assistance Memory for Proactive Robot

Proactive robot assistance requires acting on user needs before explicit requests, but zero-shot vision-language models struggle in this setting: lacking knowledge of the user's personal routine, they treat every situation as general and uncertain, defaulting to inaction rather than risking an incorrect intervention. We observe that everyday routines are highly repetitive, so even a few interactions reveal stable user-specific patterns that can ground proactive decisions. Based on this, we propose a System 1–2 framework in which a VLM task predictor reads a Procedural Assistance Memory (PAM) alongside the current observation to decide when and how to assist, and a VLA executor realizes the predicted task as low-level robot action. PAM is structured as a finite-state machine over activity phases, per-state assistance rules, and event-driven transitions; a reflection VLM revises PAM from each day's interaction trace without weight updates or retraining. We evaluate on a multi-day desk-assistance scenario with six pick-and-place tasks. Memory-free baselines remain conservative across all five evaluation days, yielding near-zero recall on proactive tasks. Our system improves consistently with accumulated experience, achieving the highest precision, recall, and F1 and reducing first-fire delay from approximately 5 seconds on Day 1 to 1 second by Day 5. Ablations further show that the FSM structure, rather than memory alone, is the primary driver of these gains.

A person sits down with an empty bowl. Oatmeal? Cereal? Yogurt? Nothing in the scene settles it — the answer lives in the user's habits, not in the pixels. A zero-shot VLM, unable to tell which of several plausible interventions is the right one, does the safe thing and does nothing. That conservatism runs deep: LIT and Event-Driven rarely fire on any proactive task, at near-zero recall throughout the five days. The Zero-shot VLM does fire more often, but it plateaus after Day 1 — five days of repeated exposure to the same routine never sharpens it further.

The missing information is learnable, and cheaply. Everyday routines repeat, so a handful of days of watching one person is enough to fix what "a bowl on the desk" means for this person. PAM accumulates exactly that, at deployment time, with no weight updates.

System 1–2, split. Once per second, a Qwen3.5-27B task predictor (thinking off) reads the PAM memory together with the last W = 5 frames and emits either a language subtask — "bring milk" — or NO_ACT. Anything that is not NO_ACT goes to a π_0.5 executor fine-tuned on the six atomic skills, which turns it into robot action. Neither model has to do the other's job, which is exactly the failure mode of a monolithic VLA under limited robot data.

The memory writes itself. At the end of each day, a reflector — the same backbone, thinking on — watches the day's recording and rewrites PAM: adding the assistive opportunities the robot missed, broadening triggers that fired too late, introducing new states, and revising rules that no longer match what the user does. No gradients, no retraining, no human writing rules.

In the desk routine, an empty bowl is a strong hint. So we built a routine that offers almost none: the user walks in empty-handed, takes a pill with water, then drinks a Coke while working. Nothing on the desk says what comes next — which is precisely the knowledge a memory is supposed to supply.