1.1 Part 1 · Imitation learning

Behavior Cloning

You can't write rules for every apartment, so instead Ada is shown the way: an operator drives her to the kitchen, and a neural network learns to reproduce his controls. That is behavior cloning, and nearly every modern robot model starts from it, from ALVINN in 1989 to large models that take a camera image and a command in plain words. The method's central difficulty: the trained network makes small mistakes, ends up where the operator never drove, and makes bigger mistakes there. In this lesson we'll see why that happens and find the data that keeps Ada from drifting off the route.

Ada is the home humanoid robot that the course labs are built around. More about her in lesson 0.1.

≈ 45 minutesa lab and 4 interactives

1989Train a mini-ALVINN and find a road where it steers wrong

LABDrive Ada to the kitchen, train a network on your drives, find where it fails, and fix that with data

FRAME BY FRAMEScrub through a crash and find the frame where the error began

SQUAREDLearn why errors pile up as the square of the trip length, and find the accuracy at which behavior cloning gets through a trip without a mistake

EXERCISESort the tricks: what will help Ada and what will hurt

Start the lesson ↓

1 Idea

Learning from demonstrations

Ada has to get from her charger to the kitchen. You could write rules: where to turn, how to pass through doorways, what to do near the sofa. But in another apartment you would have to write all of it again.

There is a simpler path: an operator drives Ada to the kitchen a few times, and a neural network learns to reproduce his actions. We record pairs of "what the robot sees → what the operator did" and train the network on them, the same way you train any classifier or regression: the observation is the input, the operator's action is the correct answer. This is behavior cloning. The trained network that picks an action from an observation is called a policy.

💡

How this differs from ordinary supervised learning. A cat classifier has no effect on which photos it will see tomorrow. A policy does: turn the wheel a little harder than the operator did, and the next frame no longer looks like the data. This whole lesson is about the consequences of that difference.

Under the hood: behavior cloning as regression

We have a dataset of demonstrations: pairs of "the state the operator saw" and "the action he chose". The policy is a network $\pi_\theta$ with parameters $\theta$ . Training is ordinary error minimization on these pairs. For continuous actions that is the mean squared error (MSE):

\theta^* = \argmin_\theta \frac{1}{N}\sum_i \bigl\lVert \pi_\theta(s_i) - a_i \bigr\rVert^2,

for discrete ones, like ALVINN's 45 outputs, the negative log-likelihood:

\theta^* = \argmin_\theta \Bigl(-\frac{1}{N}\sum_i \log \pi_\theta(a_i \mid s_i)\Bigr).

The trouble is in the distributions. The network learns on states from the operator's distribution $d_{\pi^*}$ but acts on its own, $d_{\pi_\theta}$ . Learning theory promises a small error only on $d_{\pi^*}$ . As soon as the policy differs from the operator even slightly, the distribution of its states shifts, and the guarantees stop applying.

Can a network that has only seen examples drive a car on its own?

2 History

1989: ALVINN

In 1989 Dean Pomerleau at Carnegie Mellon put this to the test: he taught a neural network to drive the NavLab van. The network took a 30×32-pixel image of the road plus rangefinder data, had 29 hidden neurons, and 45 outputs, each one standing for its own steering direction. The correct answer was encoded as a "hump" of activation around the right direction. It was trained on 1,200 synthetic road images, and the van drove at half a meter per second.

Below is the same idea in miniature. The network looks at a 16×15-pixel road and outputs 15 steering choices. The tasks are on the right: train the network, find a road where it gets things wrong, and see what it does at a fork.

What the camera sees

Network output: 15 steering notches

The whole network: 240 pixels → 12 hidden neurons → 15 steering notches

The brightness of the circles and bars shows activations on the current road. Click a hidden neuron: its weights appear as an image at the bottom left. Blue pixels switch it on, orange ones suppress it. Before training the weights are random; after training they look like pieces of road. The lines from hidden neurons to outputs are weights too: blue ones strengthen an output, orange ones suppress it, and the thicker the line, the larger the weight.

Road curve0.3 Car offset0 Fork

network outputcorrect answer▼ peak of the hump: the network's steering

The camera frame is 16 × 15 pixels: the light stripe is the road, dark is the shoulder. Curve and offset are in arbitrary units from −1 to 1: a curve of 1 is a sharp right turn, an offset of ±1 puts the car at the edge of the road.

The network is not trained yet: its outputs are random.

Under the hood: the ALVINN network and our mini version

ALVINN of 1989: 1,217 input neurons, a 30×32-pixel retina from the camera's blue channel, 8×32 values from a laser rangefinder, and one feedback neuron for road brightness. The hidden layer had 29 neurons; the output had 45 neurons for turn curvature plus one more for feedback, about 36 thousand weights in all. The correct answer was a Gaussian "hump" around the right turn rather than a single number, so the output slides smoothly as the road moves. It was trained by backpropagation on 1,200 synthetic road images for 40 epochs.

Our mini version: a 16×15 = 240-pixel input, a hidden layer of 12 sigmoid neurons, and 15 output neurons. The hump is a Gaussian 1.3 neurons wide around the correct steering. Training: 1,200 synthetic roads, 40 epochs of stochastic gradient descent with momentum 0.85 and step 0.35, MSE loss. In the training roads the curve ranges from −1 to 1 and the car offset from −0.3 to 0.3, like a driver who keeps to the middle. We read the steering from the peak of the hump and count the error in notches between the peak and the correct answer.

The correct steering in the synthetic data and the network output:

\begin{gathered} \text{steering} = 0.75\cdot\text{curve} - 0.85\cdot\text{offset},\\ \text{output}_i = \sigma\Bigl(\sum_j w_{ij}\,\sigma\Bigl(\sum_k v_{jk}\,\text{pixel}_k\Bigr)\Bigr). \end{gathered}

What the mini-ALVINN shows: the network steers correctly on roads that look like the training ones and fails when the car sits farther from the middle than it did in the data. A good driver keeps to the middle, so such frames almost never appear in his drives. What will happen to Ada if we train her on an operator's tidy drives?

🔬 Lab

You are the operator

Let's test this on Ada. You will drive her from the charger to the kitchen, train a policy on your drives, and let Ada drive on her own. The policy sees what Ada sees, 9 rangefinder beams and the direction to the kitchen, and outputs a steering rate. You can add drives from a program that knows the route and always drives neatly down the middle; from here on we call it the teacher. Then you will hunt for the policy's failures and compare which data keeps it from drifting off the route. The tasks are on the right, one at a time.

Top-down view of the apartment. Ada is the circle with eyes, the blue square is the charger, the dashed circle in the kitchen is the goal. While Ada drives, the blue lines are her 9 rangefinder beams. Gray lines are the drives the policy learns from. At the end of a drive, a check mark means Ada made it, a red cross means a crash.

training

Under the hood: what's inside the lab

World. A 6 × 4 m floor plan, one unit is a centimeter. Ada is a circle of radius 18 cm driving at a constant 60 cm/s. The control is a steering rate of up to 2.2 rad/s, with a 50 ms step.
Observation. 9 rangefinder beams in a ±120° sector up to 2.5 m, the cosine and sine of the angle to the kitchen in Ada's frame, and the distance to it. Everything is normalized.
Policy. A fully connected 13 → 48 → 48 → 1 network with leaky ReLU and a tanh output multiplied by 2.2 rad/s. Training: MSE and Adam, step 0.004, 900 iterations of 96 examples each, with the step 3 times smaller over the last 30% of iterations.
Teacher. A pure-pursuit program: it looks at a point on the route 42 cm ahead and turns toward it with gain 2.4. Each of its drives is offset slightly from the center, with a 3 cm spread.
Evaluation. Each policy drives 10 times from the same set of starts: a spread of about 7 cm and 0.1 rad, with Gaussian noise of 0.2 rad/s added to the steering, like imperfect motors. A run's start and noise depend only on its number, so policies are compared on equal terms.
Failure hunt. A drive from a chosen start runs without noise: from the same spot the outcome is always the same. "Near the data" means within 20 cm of the closest point of the drives the policy was trained on.
Noisy operator. While recording, noise of 0.5 rad/s is added to the steering, but the data records the teacher's clean action.
Side shifts. Every moment of a drive gets two extra copies shifted 15 cm left and right with a small rotation. The teacher computes the steering for them.

The lab's result: a policy trained on 10 tidy teacher drives makes it 6 times out of 10, and three times as many such drives don't help, 5 out of 10. Policies trained on a noisy operator's drives or on drives with side shifts make it 10 out of 10. Why does a policy trained on tidy drives crash so often?

3 Distribution shift

Why the policy drifts off

The policy crashes because its small error changes the next frame. The network errs only slightly, and Ada ends up closer to the wall than the operator ever was. That frame isn't in the data, so the network errs more. The frame becomes even less familiar, and so on until the collision.

In machine learning this is called distribution shift (covariate shift): the model was trained on the operator's distribution of states but runs on its own. That is also why new tidy drives don't help: no matter how many you add, they contain no frames away from the route.

Below is one of the crashes of the policy trained on 10 tidy teacher drives. Scrub through it frame by frame and find where the steering first went wrong and where Ada was at that moment.

Top-down view. The gray bundle is the 10 training drives of the teacher. The orange line is the policy's drive up to the current frame, the pale dashed line is its path beyond it, the red cross is the crash site. The blue beams are the 9 rangefinder beams, which is what Ada sees. The arcs at Ada show the steering: orange is the policy's, blue is the teacher's. The dashed line from Ada leads to the closest point of the training drives.

Frame0.0 s

policy steering and its errorteacher steeringdistance to the closest point of the training drives

Training the policy…

🎯

This is why validation accuracy misleads. Validation examples come from the operator's drives, the very distribution the network was trained on. But the policy drives through its own states. A low validation error says nothing about whether Ada will make it.

Every error drags the next ones along. How fast do they pile up as the trip gets longer?

4 Theory

The error grows as the square of the trip length

How fast errors pile up was worked out by Stéphane Ross and Drew Bagnell in 2010. Suppose the network errs with probability $\varepsilon$ on the operator's states, and an episode lasts $T$ steps. If the errors were independent, an episode would collect about $\varepsilon T$ of them.

But in behavior cloning the first error leads to places the operator never visited, and from there the policy may err on every step. So in the worst case the number of errors is on the order of $\varepsilon T^2$ .

On the chart, the gray dashed line is independent errors; the solid orange curve is behavior cloning with no ability to recover. The orange dashed line is the parabola $\varepsilon T^2/2$ : while errors are few, the orange curve follows it. The solid gray curve is a policy that can get back onto the route within k steps. The horizontal dotted line is one error per episode, the vertical one is Ada's trip: about 200 steps of 50 ms. Both axes are logarithmic, so the square looks like a straight line, just twice as steep as the gray dashed one: when T doubles, independent errors double, while behavior cloning's quadruple. Switch the axes to linear, and the curve becomes a parabola. Tasks on the right: find the accuracy at which behavior cloning almost never errs, then see what the ability to recover buys you.

Error probability ε0.1% Recovers within k steps10

independent errors,

\varepsilon T

behavior cloningparabola

\varepsilon T^2/2

recovery within

k

steps

Under the hood: where

\varepsilon T^2

comes from

Suppose that in a familiar state the policy errs with probability $\varepsilon$ , and after the first error it lands where the operator never was and, in the worst case, errs on every step from then on. The probability of erring for the first time on step $t$ is at most $\varepsilon$ , and then $T - t + 1$ error-filled steps remain until the end of the episode:

\begin{aligned} \E[\text{errors}] &\le \sum_{t=1}^{T} \varepsilon\,(T - t + 1) \\ &= \varepsilon\,\frac{T(T + 1)}{2} \approx \frac{\varepsilon T^2}{2}. \end{aligned}

Ross and Bagnell showed that the bound $J(\hat\pi) \le J(\pi^*) + T^2\varepsilon$ is tight: there are problems where it is attained. If the policy can get back onto the route within $k$ steps, each error costs at most $k$ steps and the sum grows linearly, roughly as $kT\varepsilon$ . On the chart that is the solid gray curve.

The orange curve is computed exactly for this model: a step counts as an error if the first error has already happened.

\begin{aligned} \E[\text{errors}] &= \sum_{t=1}^{T} \bigl(1 - (1 - \varepsilon)^t\bigr) \\ &= T - \frac{(1 - \varepsilon)\bigl(1 - (1 - \varepsilon)^T\bigr)}{\varepsilon}. \end{aligned}

The takeaway: the accuracy you would need is nearly impossible to reach on real frames. But if the policy can get back onto the route, each error costs it only a few steps. Where can the policy learn that skill, if the operator never leaves the route?

5 The fix

How to get recoveries into the data

The policy learns to recover from the data: the data must contain recoveries, states away from the route together with the correct steering from them.

A tidy operator has no recoveries: he never leaves the route. That is why three times as many of his drives didn't help in the lab, 5 out of 10 versus 6 out of 10. Meanwhile the policies trained on a noisy operator's drives and on side shifts made it 10 out of 10.

Here are three well-known ways to add recoveries, and two of them are in the lab.

Noisy operator (DART, 2017).While recording, noise is mixed into the steering. Ada gets pushed aside, the operator corrects, and his corrective action goes into the data. In the original paper this gave +62% over plain behavior cloning on grasping objects from a clutter pile.
Side shifts (Pomerleau, 1991).Every frame from the driver was shifted and rotated 14 times, by up to 0.6 m and 6°, and for each copy the correct steering was computed with a simple geometric model. The "side shifts" option in the lab does the same. By 1994 ALVINN was driving up to 90 miles without intervention.
Side cameras (PilotNet, NVIDIA, 2016).Two extra cameras look sideways, and their frames are labeled with the steering that brings the car back to the center within 2 seconds. The authors say it outright: the driver's data alone is not enough, otherwise the car slowly drifts off the road.

Under the hood: DART formally

While recording, the robot executes the operator's action with noise, but the data records the clean action:

\begin{gathered} a = \pi^*(s) + \eta, \qquad \eta \sim \mathcal{N}(0, \Sigma),\\ \text{into the data: } \bigl(s,\ \pi^*(s)\bigr). \end{gathered}

The noise pushes the robot sideways, and the operator shows how to come back from there. The DART authors choose $\Sigma$ so that the noisy states cover the future policy's errors: they train a policy, measure how far it diverges from the operator, and set noise of the same scale. Unlike DAgger, the operator stays in control the whole time and never labels someone else's drives.

🧭

What these tricks lack. All three add recoveries ahead of time: we guess where the policy will drift and put examples there. If it drifts somewhere we didn't think of, there will again be no examples. How to collect examples exactly where the policy actually goes is the subject of lesson 1.2.

Recoveries decide which states get into the data. But a policy can be spoiled from the other side too: through what the network sees at its input. It seems that the more it sees, the better it drives. Is that so?

6 Pitfalls

When more information is worse

Not always. The network looks for input features that coincide with the operator's action in the data, and it has no idea whether a feature causes that action. So an extra input sometimes makes the policy worse. There are two well-known pitfalls.

Spurious cause

The network sees the dashboard, where the brake light comes on. In the data the light is on every time the driver brakes, and the network learns "brake when the light is on". But the light is a consequence of braking, not its cause. This is a thought experiment from the paper by de Haan, Jayaraman and Levine (2019) on causal confusion.

Copying itself

If you feed the network its action history, it starts repeating its own previous action: in the data the steering changes smoothly, and "same as last time" is almost always right. Such a policy is right everywhere except the moment it has to start acting: at a traffic light it will never pull away. This problem is called copycat.

The Waymo team, training ChauffeurNet, wrote: "30 million examples are still not enough". They had to drop the motion history in half the examples so the network would look at the road rather than its own past.

Exercise: what will help Ada

Let's put together everything we learned about data and inputs. Sort the tricks into bins: will each one help the policy stay on the route? Drag a card, or click it and then click a bin.

Drift, spurious causes, copying itself: behavior cloning has plenty of weaknesses. Does anyone still use it?

7 What survived to 2026

Behavior cloning in today's systems

Yes, and widely. The architecture, the volume and the variety of data change, but at the core of nearly every robot model there is still learning from demonstrations.

1989ALVINN: pixels → steering on a real van, trained on synthetic roads.
1991–1994Learning from a live driver with shifted frames, up to 55 miles per hour.
2005DAVE: a convolutional network on a small truck, "stereo pair → steering".
2010–2011The $\varepsilon T^2$ bound and the DAgger algorithm (lesson 1.2).
2016PilotNet from NVIDIA: 72 hours of driving, three cameras, about 98% of the time without intervention.
2023ACT and Diffusion Policy: generative policies and action chunks. 50 demonstrations per task, 10–20 minutes of data (lessons 1.3–1.7).
2024–2026π0, Large Behavior Models, Atlas: behavior cloning on thousands of hours of data. Reinforcement learning is added on top (π*0.6, part 5).

8 Check yourself

Eight common misconceptions

9 Wrap-up

Lesson takeaways

Behavior cloning is supervised learning on pairs of "observation → operator's action". It is simple, fast, and underlies nearly every robot model.
The policy chooses which states it will see. A small error leads to places with no data: that is distribution shift.
So the error per episode grows in the worst case as $\varepsilon T^2$ , and validation accuracy guarantees nothing.
Drift is reduced by data with recoveries, states away from the route together with the correct action from them: a noisy operator, shifted frames, side cameras. The volume of tidy drives helps little.
More input information is not always better: spurious causes and copying itself.

Open questions

You are collecting demonstrations for a robot that hands over a cup. How do you plan the collection so the policy can correct its misses?

You need states to recover from: start some demonstrations from poor arm positions, mix a little noise into the recording, and record how the operator corrects a miss. It also helps to vary the starting poses and the placement of objects. The volume of identical tidy demonstrations helps little.

Why is "validation error" a poor predictor of a policy's success on the robot?

Validation examples come from the operator's distribution, while a running policy ends up in its own states. Validation error measures accuracy where the policy almost never is at the moment of failure. Success has to be measured with closed-loop rollouts, on the robot or in a simulator.

Paper guide

Pomerleau, "ALVINN: An Autonomous Land Vehicle in a Neural Network", NIPS 1989. A short paper, half an hour's read.

Read closelyThe network architecture and input, how the training examples were produced, and why the output is 45 neurons with a "hump" rather than a single number.
Watch forWhat the author writes about testing "under limited conditions", about forks, and about the need to teach the network to recover from errors.
Question along the wayWhy were random values fed to the feedback input during training? Hint: recall the section on copying itself.