3.3 Part 3 · Foundation models for robots

RT-2: actions as text tokens

In lesson 3.1, RT-1 learned more than 700 tasks from 130,000 demonstrations, but an object or a word that wasn't in the demonstrations doesn't exist for it. In lesson 3.2, vision-language models read an image as tokens and answered questions about it in words: they know the world from billions of image–text pairs, but they can't act. RT-2 (Google DeepMind, 2023) brings the two together without a new architecture: it takes the ready-made VLMs PaLI-X, with 55 billion parameters, and PaLM-E, with 12 billion, and fine-tunes them to output a robot action as a string of eight integers, that is, as one more phrase. They are fine-tuned together with the original web data; otherwise the model forgets what it knew about the world. In generalization to new objects, backgrounds and environments, RT-2 is about twice as good as RT-1, and it carries out commands that weren't in the robot data: place a can on the number 3, move an apple to the cup of the same color, choose a drink for a tired person. The price: the 55-billion-parameter model runs on a TPU cluster at 1–3 Hz.

≈ 45 minutes1 lab, 2 interactives and a quizmissions with a goal

AN ACTION IN WORDSTurn a gripper motion into a string of eight numbers and see how two different language models read it

THREE WAYS TO TRAINTrain a small “image and command → action” model on the table in the paper's three ways and compare them side by side on a command about a color the robot has never seen

WHAT GETS LOSTMeasure how a model fine-tuned on actions alone forgets how to name the color of an object in an image

WHAT EMERGED ON ITS OWNSort the commands from the paper: what the robot was taught with demonstrations, what came from the internet and what it can't do

Start the lesson ↓

1 VLA

Knowledge from the internet and robot actions

RT-1 from lesson 3.1 learned hundreds of tasks from demonstrations, but only with the objects and words that appeared in them. The command “move apple to 3” means nothing to it: there were no digits in the data. Collecting demonstrations for every object and word in the world is impossible: the RT-2 authors write that a brute-force approach would mean millions of robot trials, while the most capable language and vision-language models are trained on billions of tokens and images from the internet, an amount unlikely to be matched with robot data in the near future.

Knowledge of the world already lives in the vision-language models (VLMs) of lesson 3.2. But their output is text, and a robot needs numbers: a gripper displacement and rotation angles. In lesson 2.3, language models got around this by working as planners: SayCan chose a skill from a list, and Code as Policies wrote code. The RT-2 authors describe this role as a state machine: the model parses a command into ready-made primitives, and the low-level controllers that execute them get nothing from the model's knowledge. The paper puts its question directly: “can large pretrained vision-language models be integrated directly into low-level robotic control to boost generalization and enable emergent semantic reasoning?”

RT-2's answer is simple. A VLM already outputs a sequence of tokens, usually words. If you write the robot's action as text, you can put it in the model's answer next to ordinary words and fine-tune the model on robot demonstrations just as on questions about images. The authors call a model that takes an image and a command and outputs an action a vision-language-action model (VLA). No new parameters are added for the actions: “pre-existing vision-language models… are trained without any new parameters to output text-encoded actions”. Two models were fine-tuned: PaLI-X in versions with 5 and 55 billion parameters, and PaLM-E with 12 billion. The robot data is the same as RT-1's: demonstrations collected by 13 robots over 17 months in an office kitchen, with commands made of a skill verb and object names.

So how exactly does an action become text: which numbers, how many of them, and how does the language model's tokenizer see them?

🧩 Interactive

An action in words

A language model outputs vocabulary tokens, while a robot action is a set of continuous quantities. To turn one into the other, the continuous values have to be split into bins, and each bin has to get a token. A token here is an element of the language model's vocabulary, one “word” it can output; a bin is one of the equal intervals the range of an axis is divided into, so that instead of the exact value you write down the number of the interval.

RT-2 takes the split from RT-1. An action is an episode termination flag, six displacements of the gripper pose (three for position and three for rotation) and how far the gripper is open: eight quantities. RT-1 in lesson 3.1 had 11: the same seven for the arm and gripper, three for moving the base, and a mode: move the arm, move the base or end the episode. The RT-2 paper describes the action without the base, and instead of the mode it has a single command, to terminate the episode. Each continuous quantity is split uniformly into 256 bins, and the action becomes eight integers separated by spaces; this conversion is called action discretization: “terminate Δpos_x Δpos_y Δpos_z Δrot_x Δrot_y Δrot_z gripper_extension”. From here on, everything depends on the tokenizer of the particular VLM, the rule that splits text into vocabulary tokens. In PaLI-X, integers up to 1000 have their own tokens, so the bin number is the token. PaLM-E has no such tokens, and the authors overwrote the 256 least frequently used tokens of its vocabulary to represent actions; they call this a form of symbol tuning. The model's prompt is formatted as an ordinary question about an image, “Q: what action should the robot take to [task instruction]? A:”, and the answer is a string of numbers. Converting the answer tokens back into numbers for the robot is called detokenization.

Below is a translator with no training. The sliders set the gripper action, the string under them is the same action as eight numbers, and the tiles are the tokens the model will see. You can edit the string by hand: a number in it moves the gripper. The paper doesn't give the axis ranges, so the scales here are fractions of the range from −1 to +1, without centimeters or degrees.

The robot gripper in two views: on the left, the top view (displacement along x and y, rotation around the vertical); on the right, the side view (lift along z, tilt, jaw opening; the arc at the top is roll). The scales are fractions of the axis range from −1 to +1.

Tokenizer

Q: what action should the robot take to pick the cup? A:

Action string

Under the hood: discretization, two vocabularies and the output constraint

For a continuous quantity $a$ with bounds $a_{\min}$ and $a_{\max}$ , the bin number and the conversion back to the bin center are:

\begin{aligned} b &= \operatorname{round}\!\Bigl( \frac{a - a_{\min}}{a_{\max} - a_{\min}} \cdot 255 \Bigr), \\ \hat a &= a_{\min} + \frac{b}{255}\,(a_{\max} - a_{\min}). \end{aligned}

The termination flag is a separate discrete command, 0 or 1; it isn't split into bins. The quantization error is at most half a bin width; with the same 256 bins, that was precise enough for RT-1 to control the robot at 3 Hz.

The string in the paper looks like this: “1 128 91 241 5 101 127”. The paper prints this example with seven numbers, even though there are eight fields. In PaLI-X, the number 128 is the token “128”, one of the thousand number tokens in the vocabulary; in PaLM-E, the number is replaced by one of the 256 rare tokens reassigned to actions. The paper doesn't say which tokens exactly were reassigned, so here they are shown symbolically as ⟨r128⟩. The model is trained to predict the next token, and the authors note that this objective corresponds to the behavior cloning loss from lesson 1.1.

When the model controls a robot, it can't be allowed to answer in words. So the output is constrained: when the prompt is a command to the robot, only action tokens are allowed during decoding; on ordinary vision-language tasks, the model still outputs any text. In the interactive, the constraint works the other way around: a string is accepted only if it holds eight integers within the allowed limits.

The direction of the gripper scale here is the lesson's assumption: 0 means the jaws are closed, 255 means they are fully open. The paper doesn't specify the direction.

The format is settled: the action has become a phrase made of tokens, and it can be given to the model in the same form as the answer to a question about an image. But fine-tuning changes the weights of the whole model. If you train it on action strings alone, won't it lose what it was chosen for in the first place, its knowledge of the world from web data?

2 Fine-tuning

Co-fine-tuning and what it brings

The risk is real. If you show the model only robot demonstrations, the weights responsible for knowledge of the world rearrange themselves for a single task, and the model forgets what it learned from web data. This is called catastrophic forgetting, or simply forgetting: new training erases old skills, because nobody checks them any more. The authors name co-fine-tuning as the key detail of the recipe: the robot data is given together with the VLM's original web data (visual question answering, captions, interleaved images and text) instead of “naïve” fine-tuning on actions alone. The paper's explanation: this way the policy is exposed to both abstract visual concepts from web-scale data and low-level robot actions, which leads to more generalizable policies. The robot data gets a higher sampling weight in every batch: it makes up about 50% of the mixture for RT-2-PaLI-X and about 66% for RT-2-PaLM-E. The web data is, for example, WebLI: around 10 billion image–text pairs in 109 languages, filtered to the top 10% by cross-modal similarity, which leaves about a billion examples.

Here is the path of the data: the camera image and the command go into the vision encoder and the language model, the answer is tokens, and they are detokenized into numbers and sent to the robot; during training, the same model also receives web pairs with answers in words.

A diagram, not a drawing of the robot: the path from the robot camera is one control step, and the path from the web pair is an example of web data. Orange marks the action tokens and the command to the robot, blue the image patches, purple the words of the answer. Click the diagram to pause it.

The authors measured what this gives in about 6,000 trials on a real robot. On the familiar tasks from the RT-1 set, more than 200 commands, RT-2 and RT-1 performed alike: 91–93% versus 92%. The difference is in generalization: new objects, new backgrounds and new environments, over 280 commands, each category in an easy and a hard version. Hover over a table cell to see the condition.

Table

Model	familiar	objects, easy	objects, hard	background, easy	background, hard	environment, easy	environment, hard	average
R3M	45	32	14	13	9	0	2	12
VC-1	63	34	10	13	3	0	0	10
RT-1	92	31	43	71	9	26	14	32
MOO	75	58	48	38	41	19	3	35
RT-2-PaLI-X-55B	91	70	62	96	48	63	35	62
RT-2-PaLM-E-12B	93	84	76	75	71	36	33	62

RT-2-PaLI-X	training	objects, easy	objects, hard	background, easy	background, hard	environment, easy	environment, hard	average
5B	from scratch	0	10	46	0	0	0	9
5B	fine-tuning	24	38	79	50	36	23	42
5B	co-fine-tuning	60	38	67	29	44	24	44
55B	fine-tuning	60	62	75	38	57	19	52
55B	co-fine-tuning	70	62	96	48	63	35	63

Success in percent: the share of trials in which the robot carried out the command, from Tables 4 and 6 of the paper. Underlined: the best in the column. Hover over a cell or tap it.

Averaged over generalization, both versions of RT-2 scored 62% versus 32% for RT-1 and 35% for MOO: about twice as good, and about six times better than R3M and VC-1. Table 6 answers where the gain comes from. The same 5-billion-parameter architecture trained from scratch on demonstrations alone generalizes in 9% of cases; fine-tuning the pretrained VLM on robot data alone raises that to 42%, and co-fine-tuning to 44%. At 55 billion the gap is more noticeable: 52% versus 63%. The authors attribute this to the original data kept next to the robot data, which lets the model not forget the concepts it learned during VLM training; a larger model also generalizes better, and the authors didn't even train the 55-billion model from scratch. So the main gain comes from VLM pre-training itself, and co-fine-tuning adds noticeably to it only in the large model: 2 percentage points at 5 billion and 11 at 55. We'll come back to this difference in lesson 3.4.

Such a model comes at a price. 55 billion parameters can't run on an onboard computer, so RT-2 runs in the cloud on a TPU cluster; the robot queries it over the network, and one cluster serves several robots. The largest model outputs actions at 1–3 Hz, the 5-billion version at about 5 Hz. The authors admit that in tasks that need high-frequency control this may become a major bottleneck.

In the paper, forgetting shows up as a single row of a table. Can you see it in a model you train yourself, and measure what exactly it forgets?

🔬 Lab

Co-fine-tuning on the table

The paper compares three ways of training one architecture, and the difference shows only in the numbers of a table. Let's repeat the three ways on the table from lesson 2.1 with a model that trains in your browser in seconds, and see what each of them can and can't do.

The mini-VLA follows the RT-2 recipe. The table image is cut into 8 × 8 patches, and a shared encoder gives each patch a vector; the command words add up to a query, a single cross-attention picks a patch, and three output slots produce tokens from a shared vocabulary: one list in which words and actions sit side by side, and the model chooses from it in the same way. It holds both words (six colors and four shapes) and action tokens: the x bin, the y bin and the gripper state. There are 24 bins per axis, as many as there are cells in the action grid of lesson 2.1; RT-2 has 256. The token vectors at the input and at the output are the same, so a word learned from captions also works in a command. There are two datasets. The first is 300 robot demonstrations: the command “pick up the … cube” with red, blue and green only, and action tokens as the answer. The second is 240 “web pairs”: an image with one object of any of the six colors and four shapes, and a two-word caption. As in RT-2, the vocabulary is constrained by the task in training and at inference: for a robot command, only action tokens; for “describe”, only words.

The test is 20 new scenes for each command; the robot never once picked up a purple cube in the demonstrations. Captions are checked on 48 new images, two for each combination of color and shape; a caption counts as correct if it names the color correctly: the small model is worse at telling shapes apart, and even the pretrained model gets only about half of its captions fully right. Training one model takes from a few seconds to just over twenty on a computer, and longer on a phone.

Top-down view of the table, as in lesson 2.1: three cubes and a bowl. Next to the scene are the cards of the trained models: what each model answered with tokens on this scene and how many of the 20 scenes it solved.

Legend: marks on the scene

The small image in a card is the same scene with the 8 × 8 patch grid; the patches the model's attention falls on are filled in, and brighter means stronger. In the VLM card, the attention is shown for the “describe” command.

Command

Web pairs

Seed

Scene2

Under the hood: the mini-VLA, the shared vocabulary and the three training modes

A patch $p$ gives a vector $h_p$ through a shared two-layer encoder; the patch position, the sum of a column vector and a row vector, is added to it. The query is the sum of the vectors of the command words, $q = \sum_{w} E_w$ . Attention and context:

\begin{gathered} a_p = \operatorname{softmax}_p\bigl(h_p \cdot q\bigr), \qquad c = \sum_p a_p \,(h_p + \text{pos}_p), \\ \text{logit}_{s,t} = (c + S_s)\cdot E_t + b_t . \end{gathered}

Here $S_s$ is the vector of slot $s$ , and $E_t$ is the vector of token $t$ from the shared vocabulary, the same one that goes into the query. For the command “pick up the purple cube”, slot 1 chooses among the x tokens, slot 2 among the y tokens, and slot 3 between “close” and “open”; for the command “describe”, slot 1 chooses a color and slot 2 a shape. The loss is cross-entropy within the task's vocabulary, the optimizer is Adam, and the batches hold 16 examples.

The three modes from Table 6. “From scratch”: random weights, 100 epochs on demonstrations only. “Fine-tuning”: first 60 epochs on web pairs, which gives the pretrained VLM, then 120 epochs on demonstrations only. “Co-fine-tuning”: the same VLM, then 120 epochs on demonstrations and web pairs together. The mix has 240 web pairs out of 540 examples, so the demonstrations make up about 56%, close to the 50% of RT-2-PaLI-X.

Transfer to purple is possible because the vector of the word “purple” is learned from captions and shared by the input and the output: the web pairs align it with the vectors of patches of purple objects, and a query with this word finds the purple patch, even though the robot has never picked up such a cube. When fine-tuning on actions alone, the encoder weights and the vectors of the command words drift toward the action task, while “describe” and the caption words stay as they were and stop agreeing with them: that's the forgetting. The “picked” criterion is the same as in lesson 2.1: the center of the chosen cell is closer than 0.85 of the radius to the center of the right cube. The model is small, so the results depend on the random seed; the spread over several seeds is given in the mission debriefs.

The same in every browser. The exponential, the logarithm, the hyperbolic tangent and the cosine in the engine are built from additions, multiplications, division and the square root: the versions of these functions built into browsers sometimes differ in the last digit, and fine-tuning inflates such a difference into different numbers in the lab. That's why you'll get the same numbers in Chrome, in Safari and on your phone as in the debriefs.

On the tabletop, Table 6 repeated only in part. The difference between training from scratch and fine-tuning is there: the model trained from scratch doesn't pick up the purple cube, while the pretrained VLM, after fine-tuning, carries the word “purple” over from the web pairs into the action. Forgetting is there too, on all three seeds: the model fine-tuned on actions alone forgets how to describe images, and the co-fine-tuned one doesn't. But the tabletop didn't show a gain from co-fine-tuning in the actions: both models pick up the purple cube about equally often, and on seed C the co-fine-tuned one even falls behind. All the mini-model knows about the world is six colors and four shapes, and the word “purple” never appears in the robot's commands, so fine-tuning on actions doesn't touch its vector, and forgetting hardly gets in the way of transfer. RT-2 has incomparably more concepts from web data, and the authors explain the gain from co-fine-tuning by the model not forgetting them. But even in the paper the gain is noticeable only in the large model: 63% versus 52% at 55 billion parameters and 44% versus 42% at 5 billion. What skills did the robot gain that nobody taught it?

🧩 Interactive

What emerged on its own

The robot data had commands like “pick 7up can” and “open drawer”. The web data had digits, logos, faces, words in other languages and reasoning about images. Which of this knowledge reaches the actions, and which doesn't?

The authors use the term emergent capabilities for what the model does without a single demonstration of that kind, by transferring knowledge from web data, and divide them into three groups: symbol understanding (“move apple to 3”, “push coke can on top of heart”), reasoning (“move the apple to cup with same color”, “move banana near the sum of two plus one”, “mueve la manzana al vaso verde”) and human recognition (“move the coke can to the person with glasses”). Each command was run 5 times on four models under identical conditions. On average, RT-2-PaLI-X-55B succeeded in 60% of cases, RT-1 in 17% and VC-1 in 11%: more than three times better than the closest rival. By group, RT-2-PaLI-X got 82% on symbols, 46% on reasoning and 53% on people. The smaller RT-2-PaLM-E-12B averaged 40%, but turned out to be stronger at math: 35% versus 25%, which the authors attribute to a different pre-training mixture.

The authors state the limit of transfer up front: don't expect new motions from web data; it's concepts that transfer: objects, relations, symbols. The model's physical skills are limited to the distribution of skills in the robot data, and it only applies them in new ways. The appendix lists what RT-2 did poorly: grasping an object by a specific part, such as the handle; motions that weren't in the data, such as wiping with a towel or using a tool; precise motions like folding a towel; and reasoning that requires several layers of indirection.

Sort the commands from the paper into three groups. Drag a card, or tap it and then a group.

Under the hood: chain-of-thought reasoning inside a VLA

Since the model outputs text, you can put reasoning into the answer before the action. The authors fine-tuned a variant of RT-2-PaLM-E for just a few hundred gradient steps on data with an added “Plan” step: first the purpose of the action is described in words, and then come the action tokens. An example from the paper: “Instruction: I’m hungry. Plan: pick rxbar chocolate. Action: 1 128 124 136 121 158 111 255”. The authors call this addition a bridge between VQA datasets, where models learn to reason, and manipulation datasets, where they learn to act.

With chain-of-thought reasoning, the model answered more sophisticated commands: “I need to hammer a nail, what object from the scene might be useful?” → “Rocks. Action: 1 129 138 122 132 135 106 127”; “Bring me a drink” → “Plan: pick 7up can”; “I am sleepy, bring me a drink to help” → “Plan: Pick redbull can”. The authors stress that these are qualitative observations and initial evidence that a language-model planner like the ones in lesson 2.3 and a low-level policy can live in a single VLA.

Web data gave the robot words, symbols and reasoning, but not a single new motion, and it costs 1–3 Hz on a TPU cluster. What of the RT-2 idea lives on in the models of 2025–2026?

3 Today

What's left of the idea

Actions as vocabulary tokens became the standard way to make a VLA out of a ready-made VLM. OpenVLA (2024), an open 7-billion-parameter model built on Llama 2, encodes actions almost the same way RT-2 does: each action dimension is split into 256 bins, and the 256 least used tokens of the Llama vocabulary, which are its last 256 tokens, are overwritten for them. Lesson 3.5 covers OpenVLA and how such models are fine-tuned for your own robot.

The weak spot of this recipe is the rate. In RT-2, every action costs eight tokens that the model outputs one at a time, and even on a TPU cluster that comes to 1–3 Hz. Careful two-handed motions need dozens of actions per second, and the authors of FAST (2025) show that simple per-dimension, per-timestep binning works poorly on high-frequency data: the model learns to predict nearly identical neighboring tokens. FAST compresses the action sequence with the discrete cosine transform and only then turns the compressed sequence into tokens; the universal tokenizer FAST+ is trained on a million real robot trajectories, and training a VLA with it is up to five times faster. The other way out is not to output actions as tokens at all, but to generate a chunk of continuous actions with a separate part of the model on top of the same VLM. That's how π0 is built; it's the subject of lesson 3.6.

Co-training with web data has stayed too, but not everywhere. π0.5 (Physical Intelligence, 2025; lesson 3.8) learns from a mixture of robot recordings and web data, while OpenVLA was fine-tuned on robot data only. How much such a mixture is needed depends on the size of the model and on how diverse the robot data is. What happens when the data comes not from one robot but from twenty-two is the subject of the next lesson.

4 Check yourself

Eight common misconceptions

5 Takeaways

Lesson takeaways

RT-2 is a ready-made VLM fine-tuned to output actions as text: eight integers, each one a bin out of 256. In PaLI-X, numbers up to 1000 are tokens of their own; in PaLM-E, the 256 least frequently used tokens are overwritten for actions. No new parameters are added.
Co-fine-tuning, robot data together with the original web data, is needed so that the model doesn't forget the concepts learned during VLM training. In the paper it clearly helped the large model, 63% versus 52% at 55B parameters, while at 5B it gave only 44% versus 42%. On the tabletop, forgetting shows in the captions: the model fine-tuned on actions alone forgets how to describe images, and the co-fine-tuned one doesn't; in the actions the difference is small.
On familiar tasks RT-2 and RT-1 are equal; in generalization to new objects, backgrounds and environments RT-2 is about twice as good: 62% versus 32%. Trained from scratch, the same architecture generalizes in 9% of cases.
Web data transfers concepts: symbols, relations, words in other languages, faces. No new motions appear: the physical skills are limited by the robot data.
Chain-of-thought reasoning fits into the same text: “Plan: …” before the action tokens, after a few hundred steps of fine-tuning.
The price is speed: 55 billion parameters in the cloud on TPUs give 1–3 Hz. Hence FAST, which compresses action tokens, and π0, with a separate part for continuous actions.

Open questions

RT-2's bins are uniform. What does that mean for precise motions?

A bin is 1/255 of the axis range wide, and that width is the same for a sweeping transfer and for bringing the gripper up to a handle. If the displacement range is wide, there isn't enough precision for small motions; if it's narrow, you can't move fast. OpenVLA sets the bin bounds by quantiles of the data so that rare large displacements don't stretch the scale, and FAST moves away from per-axis bins to compressing the whole action sequence. Whether bins can be made non-uniform while staying ordinary tokens is an open question.

Web data teaches the model words but not motions. Where do you get motions that aren't in the demonstrations?

The RT-2 authors point to videos of people: they have plenty of motion, but neither robot actions nor the robot's body. Another path is more data from different robots, the subject of lesson 3.4. A third is to have the model learn from its own experience once it can already act: that's how π*0.6 (Physical Intelligence, 2025) learns, and lesson 3.8 covers it. So far none of these has given motions what web data gave words.

Paper guide

A. Brohan et al., “RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control”, Google DeepMind, 2023.

Read closelySection 3.2 on action encoding, the two vocabularies and co-fine-tuning; Section 4.3 and Table 6: from scratch, fine-tuning and co-fine-tuning on the same architecture; Section 4.2 and Figure 8 with examples of emergent capabilities.
Safe to skipAppendix A with the authors' contributions, Appendix D on the PaLI-X and PaLM-E architectures if lesson 3.2 is still fresh in your mind, and the full command lists in Tables 2 and 3.
Question to keep in mindIn Table 6, co-fine-tuning beats plain fine-tuning 44% to 42% at 5 billion parameters and 63% to 52% at 55 billion. Why does the gap grow with model size, and what should happen to forgetting in an even larger model?

Resources

RT-2Project page: videos of the robot, examples of emergent capabilities and of chain-of-thought reasoning
RT-1The paper that the robot data and the split of actions into 256 bins come from
OpenVLAAn open 7B-parameter VLA with the same action encoding
FASTAction tokenization with the discrete cosine transform

➡️

Next: Open X-Embodiment. RT-2 learned from the data of one robot in one kitchen, and web data helped it generalize. In lesson 3.4 the data comes from 22 robots in different labs, in a common format, and the question is put differently: does one arm benefit from the experience of another? Lesson 3.4 →

Was the lesson clear?

Found a mistake or inaccuracy? Report it on GitHub

Found this lesson useful? Say thanks — that is how the author knows the course is being read.