2.2 Part 2 · Spatial representations, language and planning · deep dive

PerAct: 3D representations and language

The pixel map of the table from lesson 2.1 gives a spot on the plane and a rotation about the vertical. To open a drawer or put a cup on the upper shelf you need a gripper pose in the volume. PerAct (2022) gets it with three moves. It compresses a long demonstration down to a few keyposes, and a motion planner drives the arm between them. It looks for the next pose as the most valuable voxel, a small cube of space, in a 100 × 100 × 100 grid. A million voxels plus a command in words go through the Perceiver transformer, which first squeezes them into a small set of 2,048 internal vectors. One such model learned 18 tasks in simulation and 7 tasks on a real arm.

≈ 40 minutes3 labs, an interactive and a quizmissions with a goal

KEYPOSESTune the stop threshold and compress a 325-frame demonstration down to six poses the planner uses to put the cup on the shelf

VOXELSCompare coordinate regression with picking a voxel when the demonstrations put the cup on different shelves

FIELD OF VIEWFind how much of the scene the model has to see to tell the upper shelf from the lower one

LATENTSWork out how much memory attention over a million voxels takes and how the Perceiver packs it in

Start the lesson ↓

1 Volume

From a point on the table to a pose in the volume

In lesson 2.1 an action was a point on the top-down view of the table. Transporter scores every pixel and takes the best one, and searches for the rotation among 36 angles about the vertical. But try putting a cup on the upper shelf: from above, the upper and lower shelves are the same pixel. To open a drawer, the gripper grabs the handle from the side and pulls toward itself. To turn a tap, the gripper rotates about a horizontal axis. A table map cannot describe such actions.

These tasks need a gripper pose in the volume: three coordinates, three angles and the gripper state, open or closed. Such an action is called 6-DoF, after the number of degrees of freedom. Agents that map an image straight to 6-DoF actions learn slowly: the PerAct authors cite examples where data for many tasks took weeks and months to collect.

PerAct (Perceiver-Actor; Shridhar, Manuelli, Fox, CoRL 2022, by the authors of CLIPort) carries the action map into the volume. A cubic meter above the table is split into a grid of 100 × 100 × 100 voxels. A voxel is a volumetric pixel, a small cube of space, here 1 cm on a side. The action is found as the most valuable voxel. To make that work, the authors solve three problems, and the next three sections are devoted to them: how to avoid making a decision on every frame, how to pick a pose in the volume by classification, and how to run a million voxels through a transformer.

Let's start with the first one. A demonstration is hundreds of frames. How many of them should the policy predict?

2 Keyposes

Keyposes instead of a trajectory

The policies of Part 1 decide on every frame. A ten-second demonstration at 30 frames per second is 300 decisions, and every mistake takes the arm where the operator has never been. In lesson 1.1 such mistakes piled up as $\varepsilon T^2$ , where $\varepsilon$ is the probability of erring on one step and $T$ is the number of steps; in lesson 1.7 ACT cut the number of decisions with action chunks.

Following C2FARM (James et al., 2022), PerAct keeps only a few keyposes (keyframes) from a demonstration. These are gripper poses the task necessarily passes through: in front of the cup, on the cup, above the obstacle, in front of the shelf, on the shelf. The authors call predicting continuous actions directly inefficient and noisy. The policy outputs only the next keypose, and the path to it is built by a motion planner, a classical algorithm like those in lesson 0.4.

Keyposes are found with a simple heuristic: an action is a keypose if the joint velocities are near zero and the gripper state has not changed. The arm stops where the operator aims or waits for the gripper to close. Every frame of the demonstration becomes a training example: given this observation and the command, predict the next keypose.

Ada is the home humanoid robot the course labs are built around. More about her in lesson 0.1.

Let's check on Ada. The operator drives her arm with a joystick and moves the cup from the table to the upper shelf, around the kettle. The recording is 325 frames: the gripper pose, its state and the velocities of the three arm joints. In the paper the demonstrations are cleaner: an expert sets intermediate poses, a planner drives the arm between them, and at the stops it stands perfectly still. A human hand trembles, so the “near zero velocity” threshold has to be tuned.

🔬 Lab 1

Side view: Ada and her arm on the left, a shelf unit with two shelves on the right.

Horizontal axis: demonstration time, 30 frames per second. Vertical axis: velocity of the fastest joint, rad/s, log scale. The strip at the bottom is the gripper: light means open, dark means closed. Drag the dashed threshold line with the mouse or a finger.

Velocity threshold0.04 rad/s

gripper condition

Under the hood: the heuristic, the planner and the price of every decision

Let $\dot q_j(t)$ be the velocity of joint $j$ in frame $t$ , $g(t)$ the gripper state, $\varepsilon$ the threshold. A frame is a keypose if the arm is still and the gripper is not changing state. Every frame $t$ of the demonstration turns into an example “observation and command → next keypose”:

\begin{gathered} \max_j |\dot q_j(t)| < \varepsilon, \qquad g(t') = g(t) \ \ \text{for}\ \ |t' - t| \le w, \\ (o_t,\, l) \;\mapsto\; a_{k(t)}, \qquad k(t) = \min\{k \in \mathcal K : k > t\}. \end{gathered}

The paper does not name the threshold or the window. Here the velocity is the fastest of the three arm joints, smoothed over 5 frames; the gripper “is not changing” if its state is the same in the 12 frames before and after; a gap shorter than 4 frames inside a stop counts as tremor and is merged; the keypose is the first frame of a stop; a stop before the motion starts does not count. The arm has three joints with lengths of 45, 42 and 11 cm, the gripper is always horizontal, and the joints are found by inverse kinematics.

Our planner drives the gripper in a straight line from pose to pose and does not avoid obstacles: that way you immediately see which pose went missing. In the paper the planner searches with random samples, like RRT from lesson 0.4 (in the real-arm experiments, RRT-Connect from MoveIt), and a collision flag in every action tells it whether to avoid everything in the grid or nothing. The cup must be approached from the side: coming from above knocks it over.

Why are fewer decisions better? Suppose every decision leads to a failure with probability $\varepsilon$ , after which the task is lost. Then an episode of $T$ decisions succeeds with probability $(1 - \varepsilon)^T$ . At $\varepsilon = 1\%$ , 3.8% of episodes get through 325 decisions without a mistake, and 94% get through six. It is a crude model: it ignores that deciding on a distant pose is harder than a one-frame step.

Keyposes by stops catch everything that matters: the side approach to the cup, the grasp, the lift over the kettle, the spot in front of the shelf, the release and the retreat. The policy is left with six decisions instead of 325; the planner does the rest. Equal intervals give the same number of poses, but not where they are needed. In the paper, keyposes at equal intervals or at random frames give zero success on the drawer task. On the RLBench tasks PerAct has 2 to 17 keyposes per episode.

There are few decisions now, but each one is a pose in the volume: where the gripper ends up, how it is rotated and whether it is closed. How does a network pick such a pose?

3 Voxels

The next best voxel

There are few keyposes, but each one sets the gripper in the volume. The direct route is regression: the network outputs three coordinates and three angles as numbers. From lessons 1.3 and 2.1 we know what happens when the demonstrations hold several correct answers: regression with a mean squared error outputs their average.

PerAct does to the volume what Transporter does to the table. Point clouds from calibrated cameras are fused into a grid of 100 × 100 × 100 voxels above the table, each 1 cm in size. A voxel holds 10 numbers: color, the point's coordinates, whether it is occupied, and its index in the grid. The network scores every voxel: how good it would be for the gripper center to land here. It is trained as a classifier: the correct answer is the voxel where the gripper was in the next keypose, and the loss is cross-entropy, as for any classifier. The action is the most valuable voxel. The authors call this finding the next best voxel.

The rest of the action is classification too. Rotation is split into 72 bins of 5° for each of the three axes. The gripper state and the collision flag have two options each. The flag tells the planner whether to avoid everything in the grid or nothing: the gripper has to reach the drawer handle without touching anything, but pulling the drawer open means touching it.

Let's check in miniature. The grid is 20 × 20 × 20 voxels of 5 cm. Ada's arm holds a cup in front of a shelf unit with two shelves: in the scene the gray voxels are the gripper, the teal ones are the cup. The command says where to put it. We train two models on 24 demonstrations: coordinate regression and a voxel classifier. The regression even gets more than the classifier: the exact position of the shelf unit. The classifier only sees which voxels are occupied, and not the whole scene but a window: a column of ±8 voxels above and below the spot it is scoring. How much of the scene it really needs, we check in mission 3.

🔬 Lab 2

Coordinate regression

Voxel classifier

Window height±8 voxels Sceneexample

Under the hood: PerAct's Q-functions and our classifier

From the voxel grid $v$ and the command $l$ the network outputs four Q-functions. The action is their maxima: the voxel $(x, y, z)$ , the Euler angles $(\psi, \theta, \phi)$ by bin, the gripper state $\omega$ and the collision flag $\kappa$ :

\begin{aligned} T_{\text{trans}} &= \arg\max_{(x, y, z)} Q_{\text{trans}}\bigl((x, y, z) \mid v, l\bigr), \\ T_{\text{rot}} &= \arg\max_{(\psi, \theta, \phi)} Q_{\text{rot}}\bigl((\psi, \theta, \phi) \mid v, l\bigr), \\ T_{\text{open}} &= \arg\max_{\omega} Q_{\text{open}}(\omega \mid v, l), \\ T_{\text{collide}} &= \arg\max_{\kappa} Q_{\text{collide}}(\kappa \mid v, l). \end{aligned}

The targets are one-hot: the keypose voxel in the $H \times W \times D$ grid, one bin per rotation axis, two options each for the gripper and the flag. Training is a sum of cross-entropies, as for a classifier:

\begin{aligned} \mathcal L = &-\mathbb E_{Y_{\text{trans}}}[\log V_{\text{trans}}] - \mathbb E_{Y_{\text{rot}}}[\log V_{\text{rot}}] \\ &- \mathbb E_{Y_{\text{open}}}[\log V_{\text{open}}] - \mathbb E_{Y_{\text{collide}}}[\log V_{\text{collide}}], \\ V_\ast = &\operatorname{softmax}(Q_\ast). \end{aligned}

The authors checked the resolution like this: they replayed the demonstrations with every action rounded to a voxel and to rotation bins, and confirmed that a 100³ grid is enough. In the sensitivity analysis some tasks get by with a 32³ grid, while precise tasks need the full 100³.

Our grid is 20³ with a 5 cm voxel. The voxel features are hand-made: whether it is free, whether the voxels in the column $K$ above and below it are occupied, its four neighbors at the same height and one layer down. The score is linear: a shared weight vector plus a command vector, one per each of the three commands. Training is 120 Adam steps of cross-entropy over all 8,000 voxels. The regression is least squares from the exact shelf position and the command to the target coordinates. In PerAct a network learns the voxel features, and the command is encoded by CLIP's text encoder.

The regression put the cup between the shelves: the mean of two correct answers lands in empty space. The classifier gave two peaks and picked one of them. The command moves the peak to the named shelf. Without a command the model does not know which task to solve: in the paper, PerAct without language performs at chance level. Look at the scene from above: both shelves are one column of voxels, and the table map from lesson 2.1 could not tell them apart.

But the classifier only managed when it saw a column 45 cm tall. With a smaller window the shelves and the table look the same to it: support below and emptiness above. A real network has to see the whole scene, a million voxels. How do you do that?

4 Latents

A million voxels and 2,048 latents

In Lab 2 the classifier told the upper shelf from the lower one only when its window reached down to the table and up to the next board. In the paper the same happens with drawers. A chest has three identical handles, and the command “open the middle drawer” requires seeing the whole chest. C2FARM-BC, a 3D convolutional network that first looks at the scene coarsely and then zooms in on part of it, performed at chance level, around 33%. PerAct: more than 70% of attempts.

Seeing the whole scene at once is what attention from lesson 1.6 does: every token looks at every other. But the number of pairs grows as the square of the number of tokens, and a million voxels give $10^{12}$ pairs per layer. So PerAct first cuts the grid into 5 × 5 × 5 patches, the way ViT cuts an image. That leaves $(100/5)^3 = 8000$ tokens plus 77 text tokens. But even 8,000 tokens, the authors say, are hard for an ordinary transformer to fit into the memory of an ordinary GPU.

Latents are the way out. Perceiver IO (Jaegle et al., ICLR 2022; the PerAct paper just calls it Perceiver) keeps a small set of learnable vectors, 2,048 vectors of length 512 in PerAct. Cross-attention (from lesson 1.6) pulls what is needed from the input into them: the queries come from the latents, the keys and values from the 8,077 tokens. Then six self-attention layers work on the latents alone. At the end one more cross-attention hands the result back: every patch asks the latents what is known about the scene. That is why the cost grows linearly in the number of tokens.

Let's count how much memory the attention scores alone take during training.

🔬 Lab 3

Grid100³

Outputs of the rotation head

Under the hood: cross-attention to the latents and our arithmetic

Let the input be $N$ tokens $X \in \mathbb R^{N \times d_x}$ and the latents $Z \in \mathbb R^{L \times d}$ . In cross-attention the queries come from the latents, so the score matrix is $L \times N$ rather than $N \times N$ :

\begin{gathered} Z' = \operatorname{softmax}\Bigl(\frac{(Z W_Q)(X W_K)^\top}{\sqrt{d}}\Bigr)\, X W_V, \\ Z \in \mathbb R^{2048 \times 512}, \quad X \in \mathbb R^{8077 \times 128}. \end{gathered}

The reverse cross-attention is the mirror image: the queries come from the input tokens, the keys and values from the latents. The number of scores per example, with six layers in the network:

\begin{gathered} N = \Bigl(\frac{G}{p}\Bigr)^3 + 77, \\ \text{all-to-all: } 6N^2, \\ \text{Perceiver: } 2NL + 6L^2. \end{gathered}

Here $G$ is the grid side and $p$ the patch side. Memory is the number of scores × 16 examples per batch (what the authors use) × 4 bytes. It is a lower bound: a single attention head, and weights, feed-forward activations and gradients are not counted. The 16 GB limit is just for illustration. The authors trained PerAct on 8 V100 GPUs for 16 days, 600 thousand steps.

PerAct's grid is 100³ × 10 channels. A 1 × 1 × 1 convolution raises the channels to 64, a convolution with kernel and stride 5 gives $20^3 \times 64$ patches. Proprioception features are glued onto the patches: the gripper state, the finger positions and the step number, giving $8000 \times 128$ . The text is 77 CLIP tokens of dimension 512, squeezed by a linear layer down to 128. The Perceiver input is $8077 \times 128$ plus learnable positional vectors.

All-to-all attention over a million voxels takes hundreds of terabytes even by the lower bound; over 8,077 tokens, more than 16 GB; the Perceiver with 2,048 latents, about 3.7 GB. The authors also tried 512 latents: on their tasks it is no worse, and sometimes better. Rotation is predicted per axis separately: 216 outputs instead of 373,248.

Let's assemble the whole model. The input is the voxel grid and the command, the output is four heads: translation, rotation, gripper and collision flag. Along the way a million voxels are squeezed down to 8,000 patches and 2,048 latents, and then unfolded back into a million.

The data path for one PerAct step. On the left, a table with a chest of drawers: four depth cameras capture it, the blue dots are the point cloud. Blue is scene patches, orange is command words, purple is latents. On the right, the voxel scores: valuable ones glow orange, the black cube is the chosen gripper pose. Three more heads output the rotation about three axes, whether the gripper is open and whether to avoid obstacles. The numbers in the diagram are illustrative: only a few patches and latents are shown. Tap the diagram to pause it.

🧩 Task

What shape the tensor has at each step

A tensor is a table of numbers with several axes: 100 × 100 × 100 × 10 means 10 numbers in each of a million voxels. Max-pool takes, for every feature, the maximum over the whole grid; trilinear upsampling enlarges the grid again, smoothly filling the gaps. All the sizes are in the paragraphs above and in the “Under the hood” block. This task is for those who want to take the model apart layer by layer; you can skip it.

Sort the PerAct steps by the shape of their output tensor. Drag a card, or tap it and then a bin.

Only the translation head looks at every voxel: it outputs 100³ scores, one per voxel; the authors call them a Q-function. Rotation, gripper and the collision flag are computed from the features of the whole scene after max-pool. Does such a model work on many tasks at once?

5 Results

One model for 18 tasks

The authors trained one model on 18 RLBench tasks with 249 variations: open one of three drawers, stack blocks of the right color, screw in a light bulb, put a bottle on a rack. The variations are the position, color, shape, size and category of objects. There are four cameras, RGB-D at 128 × 128 pixels. Per task, 10 or 100 demonstrations; evaluation is 25 episodes per task, at most 25 actions per episode.

They compared against the image-to-action agents Image-BC on a CNN and on a ViT, and against C2FARM-BC trained on the same demonstrations. PerAct beats the former by 34 times and the latter by 2.8 times. Without a command PerAct performs at chance level. Three precise tasks, putting a ring on a peg, stacking cups inside each other and hanging cups on a rack, are solved by almost nobody: an error of a few centimeters or degrees is irreversible there.

Share of successful episodes out of 25 per task. Image-BC is the better of the two variants, CNN and ViT. Data from Table 1 of the PerAct paper. Hover over a row to see the numbers.

On a real Franka Panda arm with a single Kinect-2 camera, one model learned 7 tasks with 18 variations from 53 demonstrations. The demonstrations were recorded with an HTC Vive controller: a human sets the target poses, and a planner drives the arm to them. The model pressed a hand sanitizer dispenser in 9 attempts out of 10, and swept beans into a dustpan in 1 out of 5. Its most common mistake was the gripper state.

Where are the limits of this approach?

6 Today

The limits of keyposes and what replaced them

The authors list the limits themselves. Between keyposes a motion planner drives the arm, and the path does not depend on the model. In their tasks that did not matter, but pouring water into a cup will not work this way: it needs a smooth tilting path. Dynamic tasks, where you have to steer continuously, and multi-fingered hands fit this scheme poorly too. The model only sees the current observation, so tasks that require remembering the past, like searching drawers for an object, are out of its reach.

The models of Part 3 went a different way. They output continuous action chunks straight from images and text, like π0, the 2024 model from Physical Intelligence: we saw its flow matching head with a chunk of 50 actions in lesson 1.8, and we take the whole model apart in lesson 3.6. They need neither keyposes nor a planner, but they also lack the hint a voxel grid provides.

The idea of an action as a value in a 3D grid stayed. In the next lesson (2.3) VoxPoser, a 2023 system from Stanford, builds value maps in a voxel grid without training on the task: they are assembled by code written by a large language model, and a planner searches for a path through them.

7 Check yourself

Eight common misconceptions

8 Takeaways

Lesson takeaways

A demonstration is compressed down to keyposes: frames where the joint velocities are near zero and the gripper is not changing state. The policy predicts the next keypose; the planner builds the path to it. A handful of decisions instead of hundreds.
The poses must land in the task's bottlenecks: grasp, release, going around obstacles. Poses at equal intervals give the same number of decisions, but zero success in the paper.
The action is a classification in a 100³ 3D grid: the most valuable voxel sets the gripper position, rotation is three axes of 72 bins each, plus the gripper and the collision flag. Classification keeps several correct answers; regression averages them.
The command in words, encoded by CLIP, chooses which of the correct answers is wanted. Without it PerAct performs at chance level.
Telling identical shelves and handles apart needs the whole scene. The Perceiver handles 8,000 patches and 77 text tokens through 2,048 latents, and the cost grows linearly in the number of tokens.
One model learned 18 tasks with 249 variations in simulation and 7 tasks on a real arm from 53 demonstrations. The limits are the planner, dynamics, multi-fingered hands and the lack of memory.

Open questions

Why are keyposes a poor fit for the task “pour water into the cup”?

There the path itself matters: the kettle is tilted smoothly, and the tilting speed decides where the water goes. Keyposes set only points, and the path between them is chosen by a planner that does not care how the kettle tilts. The authors call this a limitation and suggest combining learned and planned paths, or predicting a sequence of actions rather than a single pose.

PerAct sees only the current frame. Which tasks does that put out of reach, and how could it be fixed?

Tasks that require remembering the past: counting objects, pressing buttons in a given order when the scene does not show which are already pressed, finding an item by looking into drawers one by one. The authors suggest feeding in observations from past steps, keeping the Perceiver latents between steps, or adding a recurrent network on top of the latents.

Paper guide

M. Shridhar, L. Manuelli, D. Fox, “Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation”, CoRL 2022.

Read closelySection 3 and Figure 2: keyposes, voxelization and how the Perceiver encodes the input. Section 4.3 and Figure 4: the experiment with three drawers and the global field of view. Appendix B: all tensor sizes.
Safe to skipThe description of each of the 18 tasks in Appendix A and the augmentation details in Appendix E.
Question to keep in mindThe smallest voxel in C2FARM-BC is 0.47 cm, in PerAct 1 cm. Why does C2FARM-BC still lose on the drawer task?

S. James et al., “Coarse-to-Fine Q-attention: Efficient Learning for Visual Robotic Manipulation via Discretisation”, CVPR 2022 — C2FARM: keyposes, the action as a voxel and a coarse-to-fine grid.

A. Jaegle et al., “Perceiver IO: A General Architecture for Structured Inputs & Outputs”, ICLR 2022 — latents, cross-attention at the input and at the output.

Resources

PerActProject page: videos in simulation and on a real arm, links to the code
peract/peractPerAct code in PyTorch and trained models
RLBenchA benchmark of 100 tasks with demonstration generation; PerAct uses 18 of them
ARMThe C2FARM authors' repository, on which the PerAct code is built
perceiver-pytorchAn open PyTorch implementation of the Perceiver, on which PerAct's Perceiver is based

➡️

Next: LLM planners and VoxPoser. PerAct carries out one command: put, open, turn. A long task like “make coffee” it cannot break into steps. In lesson 2.3 large language models compose a plan out of the robot's skills: SayCan checks that every step is feasible, Code as Policies writes the policy code, and VoxPoser builds value maps in a voxel grid without training on the task — lesson 2.3.

Was the lesson clear?

Found a mistake or inaccuracy? Report it on GitHub

Found this lesson useful? Say thanks — that is how the author knows the course is being read.