2.3 Part 2 · Spatial representations, language and planning

LLM planners and VoxPoser

In lesson 2.2, given the command “open the top drawer”, PerAct picked the next gripper pose. The request “make me a coffee with milk” is longer: it takes about ten skills in the right order, and demonstrations of individual skills say nothing about that order. Large language models know such procedures from text, but they know neither the robot nor its kitchen. We'll see how SayCan multiplies “useful” from the language model by “feasible right now” from the robot, how Inner Monologue feeds the outcome of every step back to the model, and how Code as Policies and VoxPoser ask it to write code: first a program of calls to the robot's functions, then maps over which a classical planner builds the gripper path without any training.

≈ 45 minutes3 labs, an interactive and a quizmissions with a goal

PLANCompare three planners in the kitchen: the language model alone, the “feasible right now” score alone, and SayCan; then change the kitchen

FEEDBACKFind the step where the plan parted ways with the kitchen, and feed the outcome of every step back to the model

CODEStep through a program the model wrote and watch it write the missing functions

MAPBuild attraction and avoidance maps for “open the top drawer and don't hit the vase” and move the vase on the fly

Start the lesson ↓

1 Knowledge

What a language model knows about the kitchen

In lesson 2.2 PerAct received a single command, such as “open the top drawer”, and picked the next gripper keypose. A long request works differently. To make a coffee with milk, you have to take a cup, put it under the coffee maker, brew the coffee, pour in the milk and bring it over, and in exactly that order. That order is not in the demonstrations of individual skills, and recording demonstrations of every long task end to end is too expensive: in the SayCan work we are about to discuss, training the skills alone took 68,000 demonstrations recorded over 11 months on 10 robots.

Knowledge of the order of steps lives in large language models (LLMs). A model trained on text from the internet has read thousands of recipes and manuals and will continue the phrase “to make coffee, first…” without a single demonstration. But it has never seen this robot or this kitchen. The SayCan paper gives an example: asked “I spilled my drink, can you help?”, the model suggests a vacuum cleaner. Sensible advice, but useless if there is no vacuum or the robot can't use one. And asked how a robot would bring an apple, the model answers that the robot could go to the nearest store and buy one.

So the model isn't asked to write the plan as free text. The robot has a set of skills, each with a short description: “take the cup”, “open the cabinet”. A language model can do more than continue text: it can also score ready-made options, estimating how likely this particular description is to come next after the request and the steps already taken. The model scores every skill, and the plan is assembled only from what the robot can do. The model's prompt is a dialogue with examples:

Human: How would you bring me an orange?

Robot: 1. find an orange, 2. pick up the orange, 3. bring it to you, 4. put down the orange, 5. done.

… 16 more examples in the same format …

Human: Ada, make me a coffee with milk.

Robot: 1. which skill?

The examples before the request set the format of the answer; the paper uses 17 of them. The model scores which skill description most likely continues the line “Robot: 1.”, then the chosen step is appended and the model is asked about step 2.

Ada is the home humanoid robot the course labs are built around. More about her in lesson 0.1.

Choosing from a list takes the vacuum cleaner out of the plan, but it doesn't give the model eyes. If the cabinet is closed, the model will still start with “take the cup from the cabinet”: that's how thousands of recipes begin, and it can't see the closed door. How do you tell it what the robot can do right now?

🔬 Lab 1

SayCan: useful and feasible

The robot itself knows what it can do right now. In SayCan the skills have value functions trained with reinforcement learning: from the camera image, the function estimates how likely the skill is to succeed from the current state. If the cabinet is closed, “take the cup from the cabinet” gets a low score and “open the cabinet” a high one. Such a score is called an affordance: what the environment allows you to do.

SayCan (Ahn et al., Google, 2022) multiplies the two scores. The language model says how useful a step is for the request, the value function says how feasible it is right now. The robot executes the skill with the highest product, appends it to the plan and asks both models again, until the language model picks “done”. The authors call the robot the language model's “hands and eyes”: the model knows the task, the robot knows its kitchen.

In the paper the robot has 551 skills in seven families over 17 objects: pick, place, rearrange, open and close a drawer, go to a location. Ada's kitchen has 11 skills. The language model's scores here are set in advance: they illustrate the principle rather than the answers of a real model. The affordances are also set by simple rules based on the state of the kitchen. Three planners get the same request and work in the same kitchen: the language model alone, the affordance alone, and SayCan.

Cabinet

Milk

Cup

Under the hood: the selection rule, the value function and our kitchen

Let $i$ be the request, $\ell_\pi$ the description of skill $\pi$ , $s_n$ the state before step $n$ , and $c_\pi$ the event “the skill completed successfully”. At step $n$ SayCan picks

\begin{aligned} u_\pi &= p_{\text{LLM}}\bigl(\ell_\pi \mid i,\ \ell_{\pi_0}, \dots, \ell_{\pi_{n-1}}\bigr), \\ a_\pi &= p_{\text{value}}\bigl(c_\pi \mid s_n,\ \ell_\pi\bigr), \\ \pi_n &= \arg\max_{\pi \in \Pi}\ u_\pi \cdot a_\pi. \end{aligned}

The product follows from a simple assumption: a skill that succeeds advances the task with probability $u_\pi$ , the probability that it is the right step, and a skill that fails doesn't advance it at all. Then $p(c_i \mid i, s, \ell_\pi) \propto a_\pi\, u_\pi$ . The chosen skill is executed, its description is appended to the prompt, and everything repeats until “done” is chosen.

The skills themselves were trained both by imitation (BC-Z) and with reinforcement learning (MT-Opt), while the value functions were always trained with reinforcement learning on a sparse reward: 1 at the end of the episode if the skill succeeded, 0 otherwise, with no discounting. The expected sum of rewards in such a task is exactly the probability of success, which is why the value serves as an affordance. The raw scores are calibrated, with parameters picked empirically: for “pick”, $a = \operatorname{clamp}\bigl(\tfrac{v - 0.2}{0.5 - 0.2},\ 0,\ 1\bigr)$ ; for “go to”, by the distance to the location; “place” is always considered feasible (the language model puts it after “pick” anyway); and “done” gets 0.1 so that the plan ends when no feasible steps are left. A skill that has already been done, such as going to the table the robot is already standing at, is not chosen again.

We have 11 skills. The affordance is set by rules: “take the cup from the cabinet” gets 0.04 with the cabinet closed and 0.88 with it open, and a skill that is already done gets 0. The language model's scores are preset shares: the model knows the recipe “cup → coffee maker → coffee → milk → pour → bring” and continues it after the last step in the history. It treats the steps in the history as done and almost never repeats them. In the paper $u_\pi$ is the probability of the description text as a continuation of the prompt under the 540-billion-parameter PaLM; here it is shares that sum to 1.

In the paper, on 101 requests in a mock kitchen, PaLM-SayCan produced a correct plan 84% of the time and carried the task through to the end 74% of the time. In a real kitchen: 81% and 60%. Without value functions, when the language model alone picks the step, 67% of the plans are correct. Of all errors, 65% came from the language model and 35% from the affordances. Long requests were the hardest: the model finished too early, for example bringing one item out of two. And with the smaller FLAN language model the plan was correct 70% of the time and execution succeeded 61% of the time: a better language model makes a better robot.

But both scores look only at the current step. If a skill fails even though the value function promised success, the language model never finds out: in its history the step is already done. The authors explicitly call this a limitation: the system struggles to react when an individual skill fails despite a high score. What happens to the plan then?

🔬 Lab 2

Inner Monologue: the step's outcome in words

If a skill fails, SayCan learns about the kitchen only through the affordances of the next step. The language model sees only the request and the list of chosen steps and treats each of them as done, so the failed step stays in its history as completed.

Inner Monologue (Huang et al., Google, 2022) writes the outcome into the same prompt, in words. After a step, the text gets a line from the success detector (“Detector: success — no”), from a scene description (“I see: coke, water, chocolate bar”), or a human's answer to the robot's question. The language model is not fine-tuned: it reads the feedback as a continuation of the dialogue and decides on its own whether to repeat the step, choose another one or ask the human.

Take the same kitchen and SayCan with the same scores, but on the first attempt the cup will slip out of the gripper. The left side of the log is the prompt the language model sees; the right side is what actually happened in the kitchen.

Feedback

Detector

Mishap

Under the hood: kinds of feedback and where they fail

The success detector answers whether the last skill succeeded: “Detector: success — yes” or “Detector: success — no”. In simulation it looks at the true state; in a real kitchen it is a classifier trained on images.
A passive scene description arrives after every step without being asked for: a list of visible objects (“I see: …”) or which subgoals have already been reached.
An active scene description is the answer to a question the model itself asked: “Is the drawer open?” — “No, it's closed.” In the paper a human answered the questions, including ones about their own preferences: “Which drink should I bring?” — “Something with caffeine.”

The language models in the paper are InstructGPT on a tabletop with blocks and PaLM in the kitchen, both without fine-tuning, with a few examples in the prompt. On a real tabletop, where the robot had to finish a tower of three blocks and sort food and condiments onto different plates, the variant that sees the object list only at the start succeeded in 20% of attempts on average, and the one with the object list and the success detector after every step, in 90%. Without the detector the system didn't retry a failed grasp; without the object list it didn't know about objects that were at first hidden behind others.

Feedback brings new failure modes too. If the detector errs toward “no”, the robot needlessly repeats a step that succeeded. If it errs toward “yes”, the model considers done what isn't, and the plan parts ways with the kitchen. The authors also note that the model sometimes ignores the feedback and proposes skills with objects that aren't in the scene.

Our detector tells the truth, and in “faulty” mode it answers “yes” once, on a failed step. The model repeats a step after “Detector: success — no”, and after two failures in a row it gives up on it and ends the plan.

In the real kitchen from the SayCan paper, the authors interfered with the robot: a human stepped in while a skill was running. In tasks that involved moving around the kitchen, SayCan under such disturbances didn't complete a single attempt, while Inner Monologue with the success detector and the object list completed 75%. Across all 120 trials, with and without disturbances, SayCan succeeded in 30.8% and Inner Monologue in 60.4%. On top of that, the model with feedback changed the plan by itself when the human changed their mind mid-task, and proposed taking a different block when the first one turned out to be too heavy, even though the prompt examples had nothing like that.

But the model still chooses only from ready-made skills. The request “move the cup a little to the left” needs a skill “move a little to the left”, and you can't train a skill for every such phrasing. Can a language model be given more than a choice from a list?

2 Code

Code as Policies: the model writes a program

A list of skills limits the model. To move the cup “a little to the left” or to push a block “while it's left of the bowl”, you need a skill with that number or that condition built in, and training a separate skill for every phrasing means collecting data all over again.

Code as Policies (Liang et al., Google, 2022) asks a language model trained on code to write a program. The prompt holds a few examples of “command in a comment → code” and the robot's functions: perception, such as finding objects and their coordinates, and control, such as picking up an object and placing it on another. The model answers with Python code. The program calls the robot's functions, computes coordinates with NumPy and makes decisions with conditions and loops. The model picks the numbers itself: in the paper's examples, “a little to the right” turns into adding 0.1 to the x coordinate, and “5 cm at a time” into a step of 0.05.

If the code uses a function that isn't on the list, the same model writes it in a separate query, and the program is built up from the top down. The authors call this hierarchical code generation.

Let's walk through the program from Figure 1 of the paper: “stack the blocks in the empty bowl”. The code is “written” in advance and follows the paper almost verbatim, while the perception and control functions really run on our table.

🧩 Interactive

Command

Lemon

Block

Under the hood: how the model's code is run and what the experiments showed

Unless stated otherwise, all experiments in the paper use the Codex model code-davinci-002 at temperature 0, that is, greedy decoding with no randomness. The prompt has two parts: import lines that tell the model which functions are available, and examples of “command in a comment → code”. New commands and answers can be appended to the same prompt, and then the model understands “undo the last action”.

Before running, the code is checked: it must not contain import, names that start with two underscores, or calls to exec and eval. Then the code is run through exec with two dictionaries: the first holds all the functions the program may call, and variables and new functions appear in the second. Hierarchical generation parses the code's syntax tree, finds functions that are not in scope and asks the model to write each one, then does the same inside the functions it wrote, depth first.

Hierarchical generation helps on ordinary programming tasks too: on HumanEval it solves 39.8% of the problems on the first try versus 34.9% without it, and on the 37 problems of the RoboCodeGen robotics benchmark, 95% versus 81%. In simulation, on a tabletop with blocks and bowls, on new commands with new objects, Code as Policies handled long-horizon tasks 80% of the time, a natural-language planner 64%, and CLIPort trained on 30,000 demonstrations 0%.

The paper doesn't give the body of is_empty; ours is one way to write it. The robot's functions: detect_objects returns the names of the objects found, get_pos returns coordinates in meters, and pick_place puts the first object on the second one or at a point.

Code makes the plan flexible: the program computes coordinates itself, checks conditions and repeats an action in a loop until perception says the goal is reached. But the robot still moves with ready-made primitives like “pick and place”. The authors write that the programs are limited by what the perception functions can describe and which control primitives are available, and that you can't tell in advance whether the model's answer is correct. A trajectory that brings the gripper to a drawer handle while going around a vase can't be assembled from “pick and place”. How do you get the motion itself out of a language model?

🔬 Lab 3

VoxPoser: code builds the map, the planner builds the path

A language model can't write out a trajectory as text: control runs at a high rate and along many axes at once. But it is good at saying what to reach for and what to avoid. For the request “open the top drawer and don't hit the vase” that means three conclusions: grab the handle of the top drawer, pull it toward you, and keep away from the vase.

VoxPoser (Huang et al., Stanford, 2023) turns these conclusions into maps. The GPT-4 language model writes code, as in Code as Policies. The code calls vision models: one finds an object in the frame by name (OWL-ViT), a second traces its outline precisely (Segment Anything), a third tracks that outline from frame to frame (XMem), and a depth camera gives the object's 3D points. Then the code fills in a 3D grid of 100 × 100 × 100 voxels. On the affordance map, values are high where the gripper should reach, at the handle; on the avoidance map, where it should keep away, around the vase. There are also maps for the gripper rotation, the velocity, and when to close the gripper.

The path is built by a classical planner. The cost of a cell is a weighted sum of the normalized maps with weights 2 and 1, signed so that attraction to the handle lowers it and closeness to the vase raises it. Greedy search leads the gripper through the cells with the lowest cost, and while the robot moves, the path is recomputed 5 times per second from a fresh image. No part is trained for the task: not the language model, not the detectors, not the planner.

The scene is a top-down view: a slice of the map at the height of the top handle, 60 × 40 cells of 1.5 cm. The code is “written” in advance, modeled on Figure 2 of the paper and its functions, while the maps, the cost and the path are really computed here. You can drag the vase.

Map

Avoidance weight1

Under the hood: the planner's objective, the cost map and our simplifications

Let $V$ be the value map and $p_1, \dots, p_N$ the cells the gripper passes through. VoxPoser writes the task objective as the value accumulated along the path, with a minus sign:

F_{\text{task}} = -\sum_{j=1}^{N} V(p_j).

The maps are smoothed: a Euclidean distance transform is applied to the affordance map, so the value rises smoothly toward the target, and the avoidance map is blurred with a Gaussian filter. The cost for the planner is the normalized maps with weights $w_a = 2$ and $w_o = 1$ :

\begin{gathered} c(p) = -\,w_a\,\hat A(p) + w_o\,\hat O(p), \\ \hat A,\ \hat O \in [0, 1]. \end{gathered}

Greedy search finds a collision-free sequence of gripper positions, then the other maps set the rotation, velocity and gripper state at every point. The robot executes the first point, and the path is recomputed 5 times per second, as in MPC. The language model doesn't slow this loop down: the subtask code doesn't change, so the model's answer is cached, and every re-plan runs the same code with fresh detector outputs. In the paper the code is written by several programs: one called planner splits the request into subtasks, and one called composer, for each subtask, calls the programs that build the individual maps. Each prompt has 5 to 20 examples.

Here everything is merged into one program, and the map is a flat 60 × 40 slice. The avoidance map is a disk of radius 8 cm around the vase, blurred with a Gaussian of σ = 3 cm. The path is not found by greedy search; it is the path with the smallest sum of cell costs (Dijkstra's algorithm, as in lesson 0.4): in a flat slice, greedy search easily gets stuck in a dead end between the vase and the dresser, while in the volume the gripper has more ways around. The gripper hits the vase if it comes closer than 6.75 cm to its center: the vase radius of 4.5 cm plus half the gripper width.

In the paper, VoxPoser on a Franka Panda arm performed five tasks with 10 attempts each: move the gripper to an object while keeping away from another, set the table for pasta, close a drawer, unscrew a bottle cap, and sweep paper trash into a dustpan. 88% of the attempts succeeded, and 70% with disturbances, when the arm was pushed, objects were moved, and a drawer the robot had closed was pulled back open. A Code as Policies variant with simple primitives like “move to a point” managed 24% without disturbances and 0% with them. On the task with an object to keep away from: 9 and 8 out of 10, without and with disturbances, versus 0 out of 10. In simulation, on 13 tasks with 2,766 different commands, on commands and parameters that weren't in the prompts, VoxPoser completed 65.0% of the tasks that require acting on objects and 76.7% of the tasks with spatial constraints. Code with primitives: 17.5% and 25.0%; a cost map output by a trained U-Net: 0%.

Most errors on the real robot came from perception: the detector is sensitive to how an object is posed and is worse at finding parts of objects, such as handles. The authors name other limitations too: external perception modules, manual prompt engineering, and the fact that the planner guides only the gripper, not the whole arm. VoxPoser is assembled from separate parts (the language model, the detectors, the planner), and each can let the others down. Which of these ideas have survived in the systems of 2025–2026?

3 Today

What's left of these ideas

The split into “what to do” and “how to move” has stayed. The two-level models of 2025 that came up in lesson 0.1 — Helix from Figure, GR00T N1 from NVIDIA and Gemini Robotics from Google DeepMind — work like this: a slow model thinks, a fast one moves. In Helix a 7-billion-parameter vision-language model runs at 7–9 Hz and an 80-million-parameter policy at 200 Hz; in GR00T N1 it's 10 and 120 Hz (lesson 1.6). We'll look at how they work inside in lesson 3.7. The difference from SayCan is what the levels exchange: not the text of a step but feature vectors.

The pairing “a big model picks the goal, classical methods build the path” is alive too. In lesson 0.4, Mobility VLA from Google DeepMind took a request and picked the frame of a video tour that shows the goal, and the path was built by Dijkstra's algorithm and MPC. Large vision-language models have also learned to mark points, grasps and trajectories on a frame themselves: in lesson 2.1 that was done by Gemini Robotics-ER, a 2025 model from Google DeepMind (lesson 3.7), while VoxPoser, for a similar task, assembled a map in code from the detector's outputs.

But a patchwork of separate models (the language model, value functions, detectors and the planner) remains a weak spot: each part can let the others down, like the detector that said “success” when the cup had slipped. Can you train a single model that goes straight from pixels and words to actions? That's what Part 3 is about.

4 Check yourself

Eight common misconceptions

5 Takeaways

Lesson takeaways

A language model knows the order of steps from text, but it doesn't know the robot or its kitchen. If it scores the descriptions of the robot's skills instead of writing the plan as free text, the plan is assembled only from what the robot can do.
SayCan picks a step by the product of “useful” from the language model and “feasible right now” from the skill's value function. A step has to be both.
Inner Monologue appends the outcome of every step to the prompt: success, what is visible in the scene, the human's answers. Without any training the model retries a failed step and changes the plan on the fly, but it trusts the feedback even when the detector is wrong.
Code as Policies asks the model to write code: the program calls perception and control functions, computes coordinates and repeats an action in a loop, and the model writes the missing functions itself.
VoxPoser writes code that builds attraction and avoidance maps. A classical planner builds the path over them and recomputes it on the fly, and no part is trained for the task.

Open questions

The value function promised 0.9, but the skill failed anyway. How can you notice that without a separate success detector?

Ask the same value functions after the step. If the cup is in the gripper, “put the cup under the coffee maker” becomes feasible; if that skill is still infeasible while “take the cup” is feasible again, the step most likely failed. That's a success detector too, just one assembled from functions that are already trained, and it errs together with them. A separate classifier trained on successful and failed attempts is more reliable; that's what ran in the kitchen in Inner Monologue.

VoxPoser guides only the gripper over the map. Why can the arm hit the vase even if the gripper path goes around it?

The arm's links sweep through the volume between the base and the gripper, while the cost map checks only the points the gripper passes through. The authors call this a limitation and write that planning for the whole arm would probably be better. You could check every link for collisions at each step of the path, or plan in joint space with collision checking, like the planners in lesson 0.4.

Paper guide

M. Ahn et al., “Do As I Can, Not As I Say: Grounding Language in Robotic Affordances”, CoRL 2022.

Read closelySection 3 and Algorithm 1: where the product comes from and how the plan is assembled step by step. Figure 6 and Table 2: what both models propose at every step and what is lost without value functions.
Safe to skipAppendix C on the architectures and training of the skills (BC-Z, MT-Opt) and the full list of 101 requests.
Question to keep in mindWhy does the value with a reward of 1 for success and 0 for failure equal the probability of success? What changes if a skill half-succeeds, say the cup is picked up but crooked?

W. Huang et al., “Inner Monologue: Embodied Reasoning through Planning with Language Models”, CoRL 2022 — feedback in words: the success detector, the scene description, the human's answers.

J. Liang et al., “Code as Policies: Language Model Programs for Embodied Control”, ICRA 2023 — programs instead of skill lists, and hierarchical code generation.

W. Huang et al., “VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models”, CoRL 2023 — code builds 3D maps, the planner builds the path.

Resources

SayCanProject page: videos from the kitchen and open code for the tabletop version
Inner MonologueProject page: videos with feedback on the tabletop and in the kitchen
Code as PoliciesProject page: videos, prompts and Colab notebooks
VoxPoserProject page: videos and the full prompts of all the programs
VoxPoser codeAn open implementation on the RLBench simulator

✅

Check yourself on all of Part 2. Questions from lessons 2.1–2.3, shuffled: that shows what has stuck. At the end, a score per lesson and the sections worth revisiting.

Part 2 check →

➡️

Next: Part 3, foundation models. In this lesson the robot is assembled from separate parts: the language model plans, the skills and value functions are trained separately, detectors see the scene, and a classical planner guides the gripper. Each part can let the others down. Part 3 is about models that go straight from pixels and words to actions: RT-1 and RT-2 from Google (2022–2023, lessons 3.1 and 3.3) and open vision-language-action models (VLA, lessons 3.5–3.6). Course program →

Was the lesson clear?

Found a mistake or inaccuracy? Report it on GitHub

Found this lesson useful? Say thanks — that is how the author knows the course is being read.