2.1 Part 2 · Spatial representations, language and planning

Transporter and CLIPort: where and what

The policies of Part 1 look at an image and output numbers: coordinates or joint angles. They need dozens of demonstrations per skill, and most of them can't follow a command given in words. But in tabletop tasks an action is a point in the very image the robot sees: pick here, place there. Transporter turns an action into a pixel map and learns to pick up an object from a single demonstration. CLIP ties words to images, and CLIPort puts the two together: CLIP answers what to pick, Transporter answers where.

≈ 45 minutes3 labs, an interactive and a quizmissions with a goal

WORDSTrain a small CLIP right on the page and find an object that never appeared in training

MAPTeach the robot to pick up a cube from one demonstration and compare it with coordinate regression

PLACINGFind the rotation at which the part fits into the socket, using maps for 36 angles

COMMANDConnect words to maps and test a color that never appeared in the demonstrations

Start the lesson ↓

1 Words

Images and captions in one space

To carry out the command “put the blue cube in the bowl”, the robot has to connect the words with what it sees. The classical route is a detector with a fixed list of classes. But every new object or attribute calls for new labeled images.

CLIP (Radford et al., OpenAI, 2021) learns differently. Take 400 million image–caption pairs from the internet and two encoder networks: one turns an image into a vector, an embedding, and the other does the same for a caption. The training task is to guess which caption goes with which image. A batch of N pairs has N correct matches and N² − N wrong ones: the vectors of correct pairs are pulled together, and those of wrong pairs are pushed apart. This is contrastive learning.

After training, words and images live in one space. To find an object, it's enough to describe it in words: the image whose vector is closest to the text vector wins. This way CLIP, without a single labeled ImageNet image, classifies 76.2% of ImageNet images correctly — as well as a ResNet-50 trained on its 1.28 million labeled images.

Let's check this in miniature. The objects are four shapes in six colors, and the captions are “color + shape”. One combination is missing from training entirely: the model will never see a purple cube.

🔬 Lab 1

Rows are captions, columns are images. The brighter the cell, the closer the caption and image vectors. The diagonal is outlined: these are the correct pairs.

Search by description

The purple cube has a dashed outline: it was not in training.

Under the hood: the contrastive objective and our small model

Let a batch hold $N$ pairs, let $u_i$ be the normalized vector of the $i$ -th image, $v_j$ the normalized vector of the $j$ -th caption, and $\tau$ the temperature. Similarity is the dot product, and the objective is two cross-entropies: for each image, pick out its caption among $N$ , and for each caption, its image:

\begin{gathered} \ell_{ij} = \frac{u_i \cdot v_j}{\tau}, \\ \mathcal L = -\frac{1}{2N}\sum_{i}\Bigl(\log \frac{e^{\ell_{ii}}}{\sum_j e^{\ell_{ij}}} + \log \frac{e^{\ell_{ii}}}{\sum_j e^{\ell_{ji}}}\Bigr). \end{gathered}

In CLIP the image encoder is a ResNet or a ViT, and the text encoder is a transformer. A batch is 32,768 pairs; the temperature is learned, starting from 0.07.

Here an image is 16 × 16 pixels, and it's encoded by a network with one hidden layer: 768 → 48 → 16. A caption is the sum of the vectors of two words, a color and a shape, 16 numbers per word. The vectors are normalized, $\tau = 0.1$ . A batch is 16 different pairs, each image with a random shift, size and noise; 600 Adam steps. The matrix in the scene shows eight pairs the model hasn't seen in this exact form: the images are drawn anew.

The model never saw a purple cube, yet it found one. A caption vector is the sum of word vectors, and the network learned to encode the color and the shape of an image separately: it saw purple on the ball, the bowl and the can, and cubes in five other colors. For CLIP, scale does the same job: 400 million pairs contain almost any combination of attributes. That is exactly what a robot needs — to understand words that are not in its demonstrations.

But CLIP only answers the question “what”. It compares the whole image with the text and doesn't say where on the table to pick the object up or where to put it. How do you get the place of an action out of an image?

🔬 Lab 2

An action is a cell in the image

The policies of Part 1 output an action as numbers: from the whole image the network computes the object's coordinates. If the cube moves, the network has to notice the shift and recompute the numbers, and it learns this from hundreds of examples with the cube in different places.

Transporter (Zeng et al., Google, 2020) notices that in tabletop tasks the action already lies in the image. Depth cameras look at the table, and their images are fused into a top-down view that matches the table: every pixel is a spot where the robot can lower its gripper. So instead of computing coordinates, you can score every pixel: how good is it to pick here. The result is a map, and the robot picks where it is brightest.

The map is computed by a fully convolutional network: the same weights slide over the whole image. Move the cube, and the peak of the map moves with it. This is translation equivariance: there's no need to learn every position of the cube separately.

Ada is the home humanoid robot the course labs are built around. More about her in lesson 0.1.

Let's try it on Ada's table. A demonstration is a scene in which the operator picks up the red cube. We'll train two models: a map — a single 5 × 5 filter that slides over the 24 × 24 table grid — and coordinate regression — a network that outputs two numbers from the whole image, as in Part 1. The test is 30 new scenes in which the objects lie elsewhere.

Top-down view of the table: squares are cubes, the ring is a bowl, the disk is a ball.

The whole “where to pick” map. The same table from above, 24 × 24 cells: the darker the cell, the higher the “pick here” score. The large scene shows only its peak.

Test scene1

Under the hood: the map, softmax over cells and equivariance

The network outputs a score $Q(u, v)$ for every cell $(u, v)$ . The probability of picking in a cell is a softmax over all cells, and the training objective is the cross-entropy with the cell where the operator picked:

\begin{gathered} p(u, v) = \frac{e^{Q(u, v)}}{\sum_{u', v'} e^{Q(u', v')}}, \qquad \mathcal L = -\log p(u^*, v^*), \\ f(g \circ o) = g \circ f(o). \end{gathered}

The last equation is equivariance: if $g$ shifts the image $o$ , the map $f(o)$ shifts the same way. In Transporter the map is computed by a deep fully convolutional network over a top-down view fused from three RGB-D cameras. Here it's a single 5 × 5 filter over three colors: 75 weights and a bias. Coordinate regression is a 1728 → 32 → 2 network over the whole grid, about 55 thousand weights, trained on mean squared error. Both models train right here, with 120 and 300 Adam steps.

The map picked the red cube in new scenes after one demonstration, while regression kept missing even after twenty: it has to learn how every position of the cube turns into numbers. The Transporter paper shows the same at scale. On the block insertion task the map gets 100% from one demonstration, and a convolutional network with regression gets 8% even with a thousand demonstrations.

With two cubes the difference is even starker: the map has two peaks, and the robot picks one of them, while regression outputs the mean — a point between the cubes. This is the multimodality from lesson 1.3, only now you can see it in the image. Picking is only half the task. Where do you put the object down?

2 Placing

Where to place: template matching

Placing a part into a socket means finding the position and rotation at which the part matches the slot. Transporter does it the way you look for a familiar fragment in a photo: it crops the features around the picked object and lays the crop over every cell of the scene. Wherever the match is best, that's where it places. To find the rotation, the crop is rotated through 36 angles in 10° steps, and a map is built for each angle.

Transporter learns the features of the crop and of the scene from demonstrations. Here they are given: +1 in the socket's slot, −1 in the walls around it. The part is an L-shaped corner of five squares. The brightness of the map is the share of the part that lands in the slot minus the share that lands on the walls.

🧩 Interactive

Crop rotation0°

Under the hood: transport as feature correlation

Let $\psi$ be the features of the crop around the pick point and $\varphi$ the features of the whole scene. The score for placing at pose $\tau$ (a shift and a rotation) is their cross-correlation:

\begin{gathered} Q_{\text{place}}(\tau) = \psi\bigl(o[T_{\text{pick}}]\bigr) * \varphi(o)[\tau], \\ T_{\text{place}} = \arg\max_\tau Q_{\text{place}}(\tau). \end{gathered}

$\psi$ and $\varphi$ are two fully convolutional networks, trained with the same cross-entropy over the poses where the operator placed the object. Rotation is searched explicitly: the input is rotated through $k = 36$ angles, and the score becomes an H × W × 36 map. So the network doesn't learn rotation as a number; it compares rotated copies. In the scene the features are set by hand, the grid is 32 × 32, and the part covers 80 cells.

How many demonstrations you need

Maps and template matching give Transporter a kind of generalization that networks outputting numbers lack. Below is the success rate on tasks from the paper as a function of the number of demonstrations. GT-State MLP gets the exact poses of all objects as input — what perception provides to a classical stack — and still learns more slowly.

Horizontal axis: the number of demonstrations; vertical axis: the share of successful attempts. Data: Table 2 of the Transporter paper, 2020.

Transporter picks and places by maps and learns from dozens of demonstrations. But it knows no words: if there are three cubes on the table, it will pick the one that was picked most often in the demonstrations. How do you add a command to the maps?

🔬 Lab 3

CLIPort: CLIP picks what to grasp, Transporter picks where

CLIP knows words but gives no location. Transporter gives a location but knows no words. CLIPort (Shridhar, Manuelli, Fox, 2021) joins them as two streams. The semantic stream is CLIP features tied to the text of the command: it answers what to pick. The spatial stream is the Transporter network: it answers where. The two streams merge into one map: where to pick and where to place.

The main thing CLIP brings is words that never appeared in the demonstrations. In the paper the colors are split: the robot is trained on yellow, brown, gray and cyan objects and tested on orange, purple, pink and white ones. One model for 10 tasks works no worse than a separate model for each, and on a real Franka arm a single model learned 9 tasks from just 179 image–action pairs.

Let's build CLIPort in miniature. “Where” is a pick filter that learns to pick up cubes. “What” is the similarity to the command text, computed by the mini-CLIP from Lab 1: for each cell it takes a window, a small square of the table around the cell, and compares it with the command. The result is their product. The robot's demonstrations only have commands with red, blue and green. For comparison, there's a model that learns words from these demonstrations alone: each word adds its own filter to the pick filter.

where

what

final

The three maps show the same table from above, 24 × 24 cells; darker means a higher score. “Where” marks the spots where the filter learned to pick up cubes, regardless of the command. “What” marks where a window of the table looks like the words of the command. “Final” is their product: the “where to pick” map, as in Lab 2.

Test scene1

Under the hood: how the real CLIPort works and what we simplified

In CLIPort the semantic stream is CLIP's frozen image encoder (ResNet-50) and its text encoder. The text features are multiplied with the image features at several decoder levels, and at each level the semantic stream's features are added to the spatial stream's. CLIP gets only color: it was trained without depth, while the spatial stream sees RGB-D. Training uses the same cross-entropy over pick and place poses: 200 thousand steps on a single GPU over two days.

Our fusion is simpler: the probabilities of the two maps are multiplied, $\log p = \log \operatorname{softmax}(Q_{\text{where}}) + \log \operatorname{softmax}(Q_{\text{what}} / \tau)$ , $\tau = 0.05$ . The “what” map is the mini-CLIP similarity between the text vector and the vector of the 16 × 16 window around each cell. The “where” filter learns from nine demonstrations to pick the named cube, but it never sees the command. The “words from demos” model adds a word filter to the shared filter; a word that never appeared in the demonstrations has no such filter.

The model that learned words from the robot's demonstrations knows three colors, and purple means nothing to it. CLIPort knows purple from CLIP, even though the robot never once picked up a purple cube. The demonstrations teach it the main thing — picking cubes up and putting them down — and the words come from a model trained on a huge set of pairs from the internet. This division of labor became the foundation of the models that followed.

3 Today

What remains of these ideas

Separate “what” and “where” streams are gone, but the ideas stayed. An encoder trained on image–text pairs sits in almost every vision-language-action model (VLA, Part 3). In OpenVLA, an open VLA from 2024 (lesson 3.5), the image is encoded jointly by DINOv2 and SigLIP, a successor of CLIP. π0 (Physical Intelligence policy, 2024; lesson 3.6) is built on Google's vision-language model PaliGemma with its own SigLIP encoder.

The action as a point in the image came back too. RoboPoint (2024), a vision-language model fine-tuned on points, marks on the frame where to reach and where to place, following a command. Gemini Robotics-ER, a 2025 model from Google DeepMind (lesson 3.7), points at objects and predicts grasps, trajectories and 3D bounding boxes. That's the Transporter map, only now it comes from a large model that understands language.

But a table map works when the task boils down to pick-and-place and the scene is visible from above. Turning a tap, putting a cup on a shelf 30 cm up or opening a drawer can't be described by a pixel map of the table: you need three dimensions and rotations about every axis. That's what the next lesson is about.

4 Check yourself

Eight common misconceptions

5 Takeaways

Lesson takeaways

CLIP learns from 400 million image–caption pairs to guess which caption goes with which image. Words and images end up in one space, and you can find an object by its description, even by a combination of words that never appeared in training.
Transporter outputs an action as a pixel map of the top-down view. The network is translation-equivariant, so one demonstration is enough for the map where even a thousand fall short for coordinate regression.
A map can have several peaks: two identical cubes are two ways to pick, not the average between them.
Placing is template matching: the crop around the picked object is compared with the scene at 36 rotations, and the object goes where the match is best.
CLIPort combines “what” from CLIP and “where” from Transporter. One model learns many tasks and, thanks to CLIP, understands words that never appeared in the robot's demonstrations.

Open questions

Why does Transporter work with a top-down view rather than the camera frame as is?

In a camera frame, perspective distorts sizes and distances: the same cube looks bigger closer to the camera, and a pixel doesn't correspond to a single spot on the table. A top-down view fused from depth camera images matches the table, so every pixel is an action, and moving an object is the same shift in the image. Without that, equivariance doesn't work.

The robot has to put a cup on a shelf 30 cm up. What is a table map missing?

Height and rotations about every axis: a pixel map of the top-down view only sets a spot on the plane and a rotation about the vertical. For such tasks Transporter adds regression of the remaining degrees of freedom, while PerAct from lesson 2.2 moves to a 3D grid and searches for the next pose among voxels.

Paper guide

M. Shridhar, L. Manuelli, D. Fox, “CLIPort: What and Where Pathways for Robotic Manipulation”, CoRL 2021.

Read closelySection 3 and Figure 2: the two streams and how the text is mixed with the image. Section 4: seen and unseen colors and objects, one model for 10 tasks.
Safe to skipThe description of each task in the appendix and the augmentation details.
Question to keep in mindWhy does the semantic stream get only color, while the spatial stream gets color and depth? What would change if CLIP saw depth?

A. Zeng et al., “Transporter Networks: Rearranging the Visual World for Robotic Manipulation”, CoRL 2020 — pick maps, transport as correlation and the Ravens benchmark.

A. Radford et al., “Learning Transferable Visual Models From Natural Language Supervision”, ICML 2021 — CLIP: contrastive learning and zero-shot transfer.

Resources

Transporter NetworksProject page: videos and the code of the Ravens benchmark
CLIPortProject page, videos with a real arm, and code
CLIP: Connecting text and imagesOpenAI's post with zero-shot transfer examples
OpenCLIPOpen CLIP models of different sizes and training code
RoboPointA vision-language model that marks action points on the frame following a command

➡️

Next: PerAct, 3D representations and language. A pixel map of the table sets a spot on the plane and a rotation about the vertical. To open a drawer or put a cup on a shelf, you have to choose a pose in the volume. PerAct splits space into a million voxels and looks for the next keypose as the most valuable voxel — lesson 2.2.

Was the lesson clear?

Found a mistake or inaccuracy? Report it on GitHub

Found this lesson useful? Say thanks — that is how the author knows the course is being read.