1.2 Part 1 · Imitation learning

DAgger and interventions

In lesson 1.1 the policy drifted off the route: it ended up where the operator never drove, and there were no examples there. We added such examples by guessing where it would drift: with a noisy operator and shifted frames. In this lesson we'll do without guesses: Ada drives on her own, and an expert supplies the correct action wherever she ends up. That is DAgger. Then we'll look at how to save a human expert's time: let him take over control himself when he sees danger, or let the robot call him when it is unsure.

Ada is the home humanoid robot that the course labs are built around. More about her in lesson 0.1.

≈ 40 minutesa lab and 3 interactives

TWO ADASRun a DAgger iteration and compare with the policy from lesson 1.1 on the same starts

40 HINTSDecide where to ask the teacher when his time is limited

EXPERTTake over control, as in HG-DAgger, until Ada can drive without you

CURVEPlot it point by point: DAgger versus "even more demonstrations" at equal labeling

BUDGETFind the threshold at which Ada asks for help rarely and in time

Start the lesson ↓

1 Problem

Where to get the right data

In lesson 1.1 the policy crashed for one reason: it ended up where the operator had never driven. The operator drove Ada down the center of the corridor, while Ada, through small errors, drifted toward the wall. The data had no examples of "what to do by the wall", so that is where she erred, and every error took her even farther from the examples.

In the second half of lesson 1.1 we added states near the route to the data: a noisy operator and shifted frames. It helped, but it was a guess: we assumed in advance where the policy would drift. If it drifts somewhere we didn't think of, there will again be no examples.

It's more reliable to ask than to guess. Let Ada drive on her own, and at every point she reaches, let an expert say the correct action. In the papers, the expert is whoever knows the correct action: a human or a program. We add these examples and retrain, and the data appears exactly where the policy errs. That is DAgger. How does it work, and why?

2 Idea

DAgger: the policy drives, the expert labels

DAgger (Dataset Aggregation) is a loop of four steps. It was proposed in 2011 by Stéphane Ross, Geoffrey Gordon and Drew Bagnell:

Train a policy on the expert's demonstrations.
Let the policy drive on its own and record every state it visits.
Ask the expert to label these states with the correct actions.
Add the new examples to all the old ones and retrain the policy. Repeat from step 2.

The loop removes the cause of the drift. In behavior cloning the first error led the policy to places with no examples, and in the worst case errors piled up as $\varepsilon T^2$ . In DAgger, after a few rounds, examples appear exactly where the policy goes. The authors proved that the error then grows linearly with the episode length rather than quadratically.

There is a caveat: if a single error is irreversible, say Ada falls down the stairs, the gain disappears.

Under the hood: the algorithm and the guarantee

\mathcal{D} \leftarrow \varnothing

;

\hat\pi_1 \leftarrow

any policy

for

i = 1, \ldots, N

\pi_i \leftarrow \beta_i\,\pi^* + (1 - \beta_i)\,\hat\pi_i

— with probability

\beta_i

the expert acts on a step

collect trajectories of

\pi_i

of length

T

\mathcal{D}_i \leftarrow \{(s, \pi^*(s))\}

for all visited

s

\mathcal{D} \leftarrow \mathcal{D} \cup \mathcal{D}_i

\hat\pi_{i+1} \leftarrow

train with supervision on

\mathcal{D}

return the best

\hat\pi_i

on validation

In practice $\beta_1 = 1$ , and on later iterations the expert doesn't intervene: the paper says this variant "often works best". The best policy on validation is returned, not the last one.

The guarantee: with enough iterations

J(\hat\pi) \le J(\pi^*) + u\,T\,\varepsilon_N + O(1)

Here $\varepsilon_N$ is the best achievable error on the aggregated data, and $u$ is how much one error raises the cost of the episode. If an error can be corrected quickly, $u$ is bounded and the growth is linear. If an error is irreversible, $u$ grows with $T$ , and the advantage over behavior cloning disappears. The proof needs a no-regret learning algorithm: for example, training on all the accumulated data with a convex loss.

🎮

How it was tested. The original paper used video games. In Super Tux Kart the expert was a human with a joystick, and plain behavior cloning didn't improve with new demonstrations, while DAgger almost stopped flying off the track after five iterations. In Super Mario Bros the expert was a planner with access to the game's internal state. The lab below is set up the same way: the expert is the teacher from lesson 1.1, a planning program that knows the exact map of the house and outputs the correct steering. In task 3 you become the expert.

How many such rounds will Ada need to stop crashing?

🔬 Lab

Ada learns from her own mistakes

Let's test it on the policy from lesson 1.1: it was trained on 10 tidy teacher drives and makes it 6 times out of 10. First we'll run one DAgger iteration and compare the result with the original policy on the same starts. Then we'll limit the teacher's time: it will label only 40 frames, and where to ask is up to you. In the last task the expert is you: you take over control when Ada starts to drift. The tasks are on the right, one at a time.

Top-down view of the apartment: Ada drives from the charger (blue square) to the kitchen, the dashed circle is the goal. At the end of each drive, a check mark means Ada made it, a red cross means a crash.

policy from lesson 1.1the policy we are trainingteacher's hints

Under the hood: what's inside the lab

World and policy are the same as in lesson 1.1. The starting policy is trained on 10 tidy teacher drives.
Evaluation. Both Adas drive 10 times from the same starts with the same motor noise, as in the lesson 1.1 lab.
DAgger iteration. The policy drives the route on its own with 0.2 rad/s noise, and the teacher labels every visited state. The new examples are added to all the old ones, and the network is fine-tuned for 500 iterations with step 0.003.
40 hints. Each tap selects the 10 closest frames from the 10 drives of the lesson 1.1 policy within a 35 cm radius. The teacher labels only those, and the network is fine-tuned the same way as in DAgger, each time from the original policy.
You're the expert. The start and noise come from the test run where the lesson 1.1 policy crashes. While the button is held, the cursor steers, and these "observation → your steering" pairs go into the data. After the drive the network is fine-tuned.

The lab's result: one DAgger iteration is enough for Ada to make it 10 out of 10 instead of 6 out of 10. But the comparison is unfair: the policy after DAgger got new labeled examples, and the policy from lesson 1.1 got none. Maybe DAgger wins only because there is more data?

3 Comparison

DAgger versus "even more demonstrations"

To check this, let's compare the two approaches at the same number of labeled examples. The gray curve is behavior cloning on 5, 10, 20 and 40 tidy demonstrations. The orange one is the same 5 demonstrations plus DAgger iterations, one labeled drive per iteration. Each point is training from scratch plus 20 test runs, a second or two per point. You add the points; the task is on the right.

behavior cloning

demonstrations

DAgger

iterations after 5 demonstrations

behavior cloning: more tidy demonstrationsDAgger: 5 demonstrations plus labeled policy drives

An example is one labeled frame: an observation and the correct steering; one drive gives about 200 examples. The horizontal scale is logarithmic: each tick is twice the previous one. The numbers at the gray points are the number of demonstrations; at the orange ones, how many DAgger iterations were added to the 5 demonstrations. One test run is 5%, so neighboring points can differ by 5–10% by chance.

Press any button to add a point.

At equal labeling DAgger wins: what matters is not how many examples, but where they are. But in these comparisons the frames were labeled by the teacher, a program: it answers instantly, without errors, and never tires. Will it work the same way if the expert is a human?

4 Humans

Why DAgger is awkward for humans

With a human, DAgger works worse. A program doesn't care who is driving the car; a human does. Try saying how to turn the wheel correctly while watching a video of someone else driving: the reactions are off, there is no feedback, and everything feels delayed. The authors of HG-DAgger (2019) wrote exactly about this: when labeling "from the side", the quality of human labels drops.

Their solution: the human doesn't label everything in a row, but takes over control when he sees danger and hands it back to the policy himself. Only the segments where the human drove go into the data. In the lab this is the "You're the expert" mode.

In an experiment on a real car with virtual obstacles, HG-DAgger had no collisions and no departures from the road, though the sample was small.

Under the hood: HG-DAgger

The policy is an ensemble of networks. The human decides himself when to take over control and when to hand it back. Only "observation → human action" pairs from his segments go into the data; no labels are collected on the policy's segments.

Along the way, the authors estimate where it is dangerous for the policy to drive alone. The ensemble's "doubt" is the norm of the diagonal of the covariance of its outputs. The threshold $\tau$ is the mean of the top 25% doubt values at the moments when the human took over. In simulation, starting inside the region "safe" by $\tau$ gave 12 times fewer collisions.

🚗

Interventions in industry. Self-driving companies have counted disengagements for years: cases where a human has to take control. At Waymo, remote operators today don't drive the car but advise the system, and it can reject the advice. For the 1X NEO home humanoid, unfamiliar tasks are performed by a remote operator, and those sessions become training data.

Exercise: who decides when the expert is needed

HG-DAgger still has a weak spot: to take over in time, the human has to watch the robot all the time. Methods of this family collect data where the policy goes, and they differ in who decides when to call the expert. Sort the cards into bins: drag a card, or click it and then click a bin.

Can the robot work out on its own when it needs help?

5 Attention budget

The robot asks for help when it is unsure

It can, if it can estimate its own uncertainty. A human can't watch the robot all day, so the policy must recognize on its own when it needs help and call the expert only then.

A simple way to estimate uncertainty is to train three networks on the same data with different initializations. Where there is plenty of data, they answer alike. Where there is no data, they start to disagree. The spread of their answers is the uncertainty. Such a group of networks is called an ensemble.

Below, Ada drives ten times. If the ensemble's spread exceeds the threshold, she asks for help, and the teacher drives that step. The task on the right: find the threshold at which Ada almost never crashes and doesn't call the human without need.

This is how ThriftyDAgger (2021) works. There, a robot that calls the human by threshold beat HG-DAgger, where the human decides: 73 successful peg insertions out of 100 versus 57.

Ada's drivessteps where she asked for help and the teacher drove

Top-down view of the same apartment. The threshold τ is compared with the spread of the three networks' answers. The spread is measured in fractions of the maximum steering rate of 2.2 rad/s: τ = 0.1 means the networks disagree by about 0.22 rad/s.

Uncertainty threshold τ0.158

—steps with a call for help

—share of such steps

—crashes out of 10

—made it out of 10

Train the ensemble first.

Under the hood: ensemble uncertainty

Three networks $\pi_1, \pi_2, \pi_3$ learn on the same data and differ only in their initial weights. The policy drives by the mean answer, and the uncertainty is the spread of the answers:

\begin{aligned} \pi(s) &= \frac13 \sum_{k=1}^{3} \pi_k(s), \\ \sigma(s) &= \sqrt{\frac13 \sum_{k=1}^{3} \bigl(\pi_k(s) - \pi(s)\bigr)^2}. \end{aligned}

If $\sigma(s) > \tau$ , the robot asks for help.

Where there is plenty of data, the networks converge to one answer. Where there is none, each extrapolates its own way, and $\sigma$ grows. ThriftyDAgger adds a second signal, risk: a learned estimate of the probability of not reaching the goal. It picks the thresholds so that the human intervenes on a given share of steps, say 1%.

A suitable threshold exists, but the window is narrow: below it Ada calls the human too often; above it she stops asking for help and crashes.

So far we took only one thing from each intervention: the correct action at that spot. Is there anything else useful in it?

6 What's next

From interventions to reinforcement learning

There is. The intervention itself says the policy got into a bad spot: otherwise the human wouldn't have taken control. This "something went wrong here" signal is used as a reward in reinforcement learning. In that kind of learning, the robot tries acting on its own and learns from a numeric score of its attempts, the reward: the higher it is, the better the attempt. Here are systems that learn from interventions this way:

RLIF (2023)The step before an intervention gets a reward of −1, all others get 0, and the policy is trained with reinforcement learning. This can beat even an imperfect expert: only the fact of the intervention is needed.
HIL-SERL (Science Robotics, 2025)20–30 demonstrations, human interventions and reinforcement learning on a real robot: nearly 100% success on a task within 1–2.5 hours.
π*0.6 and RECAP (November 2025)The Physical Intelligence model learns from demonstrations, operator corrections and its own attempts. On folding laundry and making espresso, throughput more than doubled. The authors note that even a single iteration often improves the result noticeably, just as in our lab.

We'll cover reinforcement learning in detail in part 5, and π*0.6 in lesson 3.8.

7 Check yourself

Seven common misconceptions

8 Wrap-up

Lesson takeaways

DAgger cures distribution shift: the policy drives on its own, the expert labels its states, the data accumulates.
At equal labeling DAgger beats "even more demonstrations": what matters is not how much data, but where it is.
Humans find it hard to label from the side. HG-DAgger: the human takes over control himself, and his segments go into the data.
Uncertainty can be measured by the spread of an ensemble, so the robot asks for help only when the network is unsure.
An intervention is also a "this is bad" signal. Reinforcement learning in the systems of 2025 is built on it.

Open questions

You are launching a pilot of delivery robots with remote operators. How do you set up the process so that every intervention improves the policy?

Record every intervention together with the observations before it and the operator's actions; regularly add this data to the whole dataset and retrain; track the "interventions per hour" metric across policy versions; set the threshold at which the robot calls the operator itself, and check that interventions decrease without crashes increasing.

Why is it important in DAgger to aggregate all the data rather than train only on the latest iteration?

Each iteration covers the states the current policy got into. If you train only on the latest one, the new policy forgets how to behave where the previous one erred and starts erring there again. Aggregation preserves the whole distribution of states the policy has visited so far.

Paper guide

Ross, Gordon, Bagnell, "A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning", AISTATS 2011.

Read closelyThe DAgger algorithm (section 3) and the experiments: Super Tux Kart, Super Mario Bros, handwriting recognition.
Safe to skipThe proofs about no-regret online learning, if online learning theory is unfamiliar.
Question along the wayWhy does the algorithm return the best policy on validation rather than the last one?