hud controls

Direct Control

The Direct Control interface allows the agents to control the robots directly through the defined perception and control primitives by us (RoboDawn). And this makes it clear that this interface is bottlenecked by the available primitives to the agent and the intelligence of the agent itself.

The implementation for direct control looks like:

Direct Control architecture: the agent gets camera observations, robot state and information about the environment and task (the prompt, e.g. pick up this cube); it calls primitives such as move_to() on the env; env feedback and updated observations return to the agent.

hud-python#778 works towards adding primitives for sensor transforms by giving the agent full pose of the camera, live transform for moving cameras like on wrists and the their intrinsics to let agent estimate the object distance from the robot and in the world. This gave huge gains in the success rate and efficiency as compared to direct control without this information.

Direct Control (Sonnet)

task	camera poses	success	calls / run	sim time	first-move miss	cost / run
lift a cube	none	8/10	11.2	30.0 s	130 mm	$0.094
lift a cube	yes	10/10	5.0	13.2 s	12 mm	$0.039
pick and place a cube	none	8/10	15.0	43.9 s	37 mm	$0.113
pick and place a cube	yes	10/10	9.5	25.7 s	8 mm	$0.067

10 runs per row. first-move miss is how far the agent's first reach landed from the cube.

pick and place a cube, camera poses given: 9 calls.

pick and place a cube, no camera poses: 30 calls.

Direct Control, Live Wrist Pose (Haiku)

task	camera poses	success	calls / run	first-move miss
pick and place 2 cubes	none	0/5	30.2	95 mm
pick and place 2 cubes	fixed cameras	3/5	27.2	57 mm
pick and place 2 cubes	fixed + live wrist	5/5	22.6	8 mm

pick and place 2 cubes, no camera poses: 30 calls, neither placed.

same layout, fixed and live wrist poses: 15 calls, both placed.

Direct Control differs from Code as Policy as agent in direct control executes one command at a time and takes in feedback, whereas, in CaP it writes the whole sequence at once and executes it and then acts upon it. This might lead to direct control being slower and inefficient than CaP.

Direct Control vs Code as Policy (Sonnet)

task	Direct Control			Code as Policy
task	success	calls / run	sim time	success	calls / run	sim time
lift a cube	5/5	5.0	13.1 s	5/5	3.0	12.3 s
pick and place a cube	5/5	8.2	21.7 s	5/5	2.6	21.9 s
pick and place 2 cubes	4/5	16.8	49.4 s	5/5	3.2	48.8 s

pick and place 2 cubes, direct control (sonnet): 14 calls, one cube placed.

same layout, code as policy (sonnet): 2 calls, both placed.

Policy Supervision

The Policy Supervision interface lets the agent guide the VLA/WAM model by breaking down tasks in subtasks on which the model was trained.

VLAs alone are usually bad at long horizon tasks and to any perturbations in the scene or the prompt (RoboCerebra). A higher level orchestrator/supervisor (System 2) is needed to cover these limitations for it and that is where we put our agent. (Orchestrating Robot Policies, VoLoAgent)

LIBERO

turn on the stove → push the plate to the front of the stove → put the bowl on the plate → put the cream cheese in the bowl → put the wine bottle on the rack.

policy	start layout	policy alone	sonnet supervising	opus supervising
smolvla	1	0/5	1.3/5	1/5
smolvla	2	0/5	2/5	3/5
pi0.5	1	2/5	2/5	4/5

smolvla alone: no step completed.

opus supervising smolvla: stove, bowl on plate, wine bottle on rack.

The agent acts as the orchestrator and breaks down the task into subtasks. While the low level model is working on the given task, the agent still supervises it and looks out for any failure. If any, the agent takes over the control of the robot. Now, I've noticed there are several conditions on which researchers have arrived for as to when should the agent be invoked and take over the robot. One is to invoke it every few seconds (Hi Robot, Orchestrating Robot Policies) or is to set expected execution time for a task (Orchestrating Robot Policies).

Policy Alone vs Supervised

task	policy	policy alone	sonnet supervising	opus supervising
lift a cube	smolvla, fine-tuned on this scene	1/3	2/3	3/3
pick and place a cube	smolvla, fine-tuned on this scene	0/3	3/3	3/3

pick and place a cube, policy alone: never picks it up.

same layout, opus supervising: placed in 10 calls.

In this interface, the supervisor decides if the task has been completed or not and it can be a bit challenging to conclude that a task has been completed from ~2 medium quality images from sim. That's why, a grounding agent is used that uses Grounding DINO or SAM3 to detect objects, which agent can use to guide the VLA or to take any further decision. (VoLoAgent, FIT-MAC)

My implementation for the Policy Supervision interface is something like:

Policy Supervision architecture: the agent gets camera observations, robot state and information about the environment and task; it gives the VLA/WAM subtasks and uses primitives such as depth() and detect(); the VLA/WAM sends actions to the env; the agent can directly interfere and take control; env feedback returns to the agent.

Here, the agent has access to grounding models (DINO) and agent is being run at 1Hz.

What I'm noticing is that most of the research in this topic is converging towards grounded features + VLA controlled by agent which is what the above implementation mostly looks like.

Code as Policy

The Code as Policy interface lets the agent write code that will be executed on the robot.

The simplest version of this interface will be agent writing code using the existing primitives for perception and control (Code as Policies, RoboCodeX). But this is bottlenecked by the primitives that are written by us. The agent will be able to use only the capabilities provided by the primitives and will be forced to try to complete the task using them. There's also no way for the agent to recover from a condition if it's just not able to figure out the solution, so it depends on the intelligence of the agent as well.

pick and place 2 cubes, code as policy (sonnet): 2 calls, both placed.

LIBERO open the drawer, code as policy (sonnet): 56 calls, never opened.

A solution to this limitation is adding skill memories (ASPIRE, RATs). Skills are memories that can be referenced by the agent when it wants to recover from a failure. These memories are created where a fix was validated in re-runs. This increases the chances of the agent successfully recovering from a failure.

To make it efficient, we can maintain code memories (RATs) which store the logic and the primitives used to complete a particular task so that when the agent encounters the same/similar task in the future, it can take that as the reference and code correctly in few or one attempt.

But after all these additions, the agent might still fail at contact-rich tasks. In contact-rich tasks, agent has very few ways to figure if it is manipulating correctly, whereas, VLA/WAM models are trained on contact-rich data which make them the strong candidate for these type of tasks. So, we can add a policy primitive which the agent can call in the code (HarnessPAI). This helps in taking advantage of agent's long horizon planning and semantic understanding along with model's contact-rich manipulation experience.

The agent can fix the environment and work on tasks which are not in the training distribution of the model. The data collected through this can be used to fine tune the model on tasks or scenes it has never encountered and is proven to make them better (HarnessPAI)!

Code as Policy Ablation (Haiku)

task	code only (≈ CaP-Agent0)	+ memory (≈ RATs)	+ policy (≈ HarnessPAI's policy primitive)	+ memory + policy (≈ HarnessPAI)
lift a cube	5/5	5/5	5/5	5/5
pick and place 2 cubes	5/5	5/5	3/5	5/5
LIBERO: open the drawer	0/3	2/3	2/3	2/3
LIBERO: put the bowl on the plate	0/3	1/3	2/3	2/3

LIBERO open the drawer, code only (haiku): 30 calls, never opened.

same layout, code calling the policy (haiku): 2 calls, opened.

LIBERO-PRO: Objects Swapped (Haiku)

task	smolvla alone	code + memory + policy
LIBERO: open the drawer	0/3	1/3
LIBERO: put the bowl on the plate	0/3	3/3

objects swapped, code + memory + policy (haiku): reports the middle drawer pulled out about 12 cm.

objects swapped, code + memory + policy (haiku): the bowl ends on the plate, off its centre.

This is the implementation of the interface that I think serves the best under CaP:

Code as Policy architecture: the agent gets camera observations, robot state, information about the environment and task, and optional code and skill memories; it writes code that calls move_to() and run_policy(); the code executes in the env and the env feedback returns to the agent.

I believe the future interface for using robotics with agents will be something like:

System 2:
Agents: Responsible for long horizon planning, creating sub tasks, sometimes control via code

System 1:
VLA/WAM: Model trained on diverse and contact-rich manipulation data which completes the sub tasks given by the agents (system 2)

This seems to be embodiment agnostic as well!