1.7 Part 1 · Imitation learning

ACT: action chunking

A policy that decides afresh at every step makes 50 decisions a second at 50 Hz. It jitters, compounds errors and gets stuck wherever the person recording the demonstration paused to think. ACT, the policy of the bimanual robot ALOHA, answers with chunks: from a single frame it predicts a hundred actions at once and executes them in a row. We'll work out why a chunk helps, how many actions to put in it, what it costs — late reactions and jerks at the seams — and how large models smooth those seams. This lesson closes the main line of Part 1.

≈ 40 minutesa lab and 5 interactives17 missions with goals

ERRORRun 200 episodes and find when a chunk hurts

CHUNKSCompare chunks of 1, 10 and 100 actions on one scene and pick the length

ARCHITECTUREAssemble ACT from blocks so that the dimensions line up

CVAE AND BLENDINGSettle the argument about “style” and scrub through chunk blending step by step

SEAMSSketch the velocity curve and find the mode that survives 200 ms of latency

Start the lesson ↓

1 ALOHA

Cheap arms and 50 demonstrations

To teach a robot from demonstrations you need arms on which a person can comfortably show bimanual work, and enough data to train on. Precise bimanual systems are expensive, and fine work is hard to do on cheap ones.

In 2023 Tony Zhao, Vikash Kumar, Sergey Levine and Chelsea Finn built a teleoperation rig from inexpensive arms. The operator moves two small leader arms, WidowX, and two large follower arms, ViperX, mirror their angles joint for joint, so no inverse kinematics is needed. A non-expert assembles the rig in under 2 hours. The figure in the paper lists the price as “<$20k”: the base kit costs $18k, with options such as cameras $20k; a ViperX follower arm is about $5,600, a WidowX leader about $3,300. Four Logitech C922x cameras look from the top, the front and the two wrists, shooting 480×640 at 30 frames per second, while teleoperation and recording run at 50 Hz. The rate matters: at 5 Hz operators spend 62% more time on a task.

The follower arm repeats the leader's angles, so the arms share one kinematics but differ in size. A trigger on the leader's handle closes the follower's gripper. Click the diagram to pause it.

The hardware is imprecise: ViperX is off by 5–8 mm, and its payload is 750 g. If you simply replay the recorded angles, that error is enough to miss a small part.

The authors make up the missing precision with learning: the ACT policy (Action Chunking with Transformers) watches the four cameras and corrects the motion frame by frame. It was trained separately for each task on 50 demonstrations, 10–20 minutes of data, and each task was tested 25 times.

ACT success on real tasks

For Thread Velcro, 100 demonstrations were recorded. The best baseline, BeT, scored 0%.

Videos by the authors

The video loads from the authors' site when you click the cover. Clips of ACT reacting to perturbations are on the project page.

On cheap arms ACT handled fine tasks in 64–96% of attempts, except Thread Velcro, where success was 20%. The authors name the action chunk as the key ingredient. What's wrong with an ordinary policy that outputs one action per frame?

2 Compounding error

Fifty decisions a second

What's wrong with an ordinary policy? It outputs one action for every frame. At 50 Hz that's 50 decisions a second and 500 in ten seconds, and every decision is slightly off.

The error pushes the arm to where there are fewer demonstrations, and there the next decision errs more. Errors add up: in lesson 1.1 their growth was estimated as $\varepsilon T^2$ , that is, with the square of the episode length.

DAgger from lesson 1.2 added examples wherever the policy drifts. That needs an expert who labels every new state, and ALOHA only has 50 recorded demonstrations.

The other way out is to decide less often: from one frame predict a chunk of $k$ actions and execute it whole. There will be $k$ times fewer decisions, but between decisions the arm drives blind. Let's check on 200 episodes of a simple model when this is a win. Ada's gripper carries a mug across the table, and we watch how far the mug has strayed from the planned path. Every step this deviation is multiplied by $\lambda$ — the stability of the system without control. At $\lambda < 1$ the arm returns to the target by itself, like a position controller. At $\lambda > 1$ the deviation grows on its own, like a robot that is balancing.

📊 200 runs

Horizontal axis: the mug's deviation from the path at the end of the episode, cm. Bar height: how many of the 200 runs ended with that deviation. Arrows at the edges: runs that went beyond the plot.

Chunk k10 Stability λ0.97

Under the hood: the model and what theory says

\begin{aligned} &\text{within a chunk:} && x \leftarrow \lambda\,x,\\ &\text{at a decision:} && x \leftarrow (\lambda - c + \delta)\,x + \varepsilon\,\xi. \end{aligned}

Here $x$ is the mug's deviation from the path in centimeters and $\xi$ is standard noise. An episode is 200 steps of 20 ms, that is 4 s; there are 200 runs, each with its own fixed seed, so the picture repeats exactly. $\varepsilon = 0.3$ cm is the error at each decision. $\delta = 0.01$ is the error that grows with the deviation: far from the data the policy errs more. $c = \max(0, \lambda - 1)$ is the correction the policy learned from the expert: on an unstable system the expert keeps damping the growth of the deviation, and that is in the data. The last decision of any policy falls on the last step, so the comparison is fair. The spread $\sigma$ is the standard deviation of the final $x$ .

Theory. Simchowitz, Pfrommer and Jadbabaie (COLT 2025) showed that a smooth deterministic Markov policy errs exponentially more in the horizon at execution time than on the data, even with stable dynamics. Non-Markov, non-smooth and stochastic policies get around this, and the authors explicitly name action chunks and diffusion. Zhang et al. (ICLR 2026) proved that with a long enough chunk the compounding of error is bounded if the system is stable without feedback. The mechanism is stability in the control-theory sense. For unstable systems you need noise during demonstration collection so that the data contains corrections.

The argument isn't over. In a preprint by Lazzati et al. (August 2026) shortening the horizon doesn't explain the success: what works is non-Markov expressiveness and “implicit ensembling”. Zeng et al. from Tedrake's group (August 2026) consider the non-Markov nature of demonstrations the main factor, and with long context closed-loop policies work best for them.

A chunk narrows the spread if the system damps deviations by itself, and hurts if it amplifies them. But the per-step policy has a second problem that this model lacks: it gets stuck where the operator paused to think. Will a chunk handle that, and how many actions should go into it?

🔬 Lab

Smoothness versus reactivity

Will a chunk handle pauses? The operator moves the arm unevenly: halfway along the path they stop to think, and the per-step policy gets stuck right there.

At the pause point every demonstration contributes many “stay” frames and one “go” frame. The policy sees only the current frame and doesn't know how long it has been standing, so again and again it chooses “stay”.

More demonstrations won't help: the operator pauses in each one, and the number of “stay” frames grows just as much.

A chunk carries the continuation of the motion: it contains both the pause and what the operator did after it. Ada's gripper carries a mug across the table to a shelf, top view. The policy is trained on ten demonstrations and in a similar position takes the median of what the operator did — that's how a network trained with L1 behaves. Let's compare chunks of 1, 10 and 100 actions on one scene, push the mug mid-chunk and pick the length.

Ada is the home humanoid robot the course's labs are built around. More about her in lesson 0.1.

show the chunk Push moment1.0 s Chunk k10 Execute100 temporal ensembling m0.01 chunks of 1, 10 and 100 side by side

Under the hood: how the lab is built

Demonstrations. 10 runs over 0.8 m at about 0.25 m/s, recorded at 50 Hz. The operator stops at the 34 cm mark. Short pauses are 12 to 19 steps, 0.31 s on average; long ones 22 to 35 steps, 0.58 s on average.
Policy. Given the mug's position, we take nearby demonstration frames with Gaussian weights ( $\sigma = 8$ mm). For each step $j$ of the chunk — the weighted median of the positions those frames reached j steps later. These are absolute target positions, as in ACT. A shift of about 3 mm in a random direction is added to the chunk; away from the data it grows: 3 cm away it doubles. The gripper chases the target with a lag, covering 80% of the distance per step.
Why $k = 1$ gets stuck. At the pause point the data has ten runs of 12–19 “stay” frames and one “go” frame each. The median next position is “stay”, and the gripper stays no matter how much time has passed. A chunk gets through the pause if more than half of the frames are already moving by its end: then the median moves too.
The push shifts the mug in the gripper 6 cm sideways. The gripper follows the old plan, and the mug travels with the offset until a new chunk accounts for it. In missions 3 and 4 the push comes after the pause, right after a new chunk starts, that is, at the worst moment. Reaction time runs from the push to the first chunk that sees it. With temporal ensembling we count when the mug is at least halfway back on the path.
Outcomes. The mug is on the shelf if it was placed no farther than 3 cm from the mark. The trip is allowed 8 s. A jerk is a step where the gripper's acceleration exceeds 4 m/s².

A chunk gets through the pause, but while it's executing the arm doesn't see the push. The working window in the toy is 14 to 25 actions, and large systems also execute only part of the chunk. ACT's chunk is 100 actions. How does one network output a hundred actions at once?

3 Architecture

A hundred actions in one pass

How does one network output a hundred actions at once? It has to turn four frames and the joint positions into 100 actions of 14 numbers, and do it fast: ACT is queried at every step, 50 times a second.

The network from lesson 1.1 is built for one action: a frame in, an action out.

You could emit actions one at a time, the way a language model emits words. But then a chunk takes a hundred network passes in a row, and there's no time for that at every step.

ACT takes the transformer from lesson 1.6. The frame from each camera goes through a ResNet18 convolutional network and becomes a 15×20×512 feature map. The map is unrolled into 300 tokens, and 2D sinusoidal positions are added so the network knows which part of the frame each token comes from. Four cameras give 1200 tokens, to which a joint token and a “style” token $z$ are added — 1202 in all. Here $z$ is 32 numbers about the operator's manner: during training they are computed by the CVAE encoder, and when the robot runs, $z = 0$ is fed in. Why that's needed is in the next section.

The encoder mixes the tokens, and the decoder outputs all $k$ actions at once. It works like the attention lab in lesson 1.6. The decoder has $k$ learned queries, 100 in ACT, and each asks “what is the action at step $i$ ?”. A query compares itself with the keys of the camera, joint and $z$ tokens and assembles its action from their values, and all queries are computed simultaneously, in one pass. Attention where queries come from one token sequence and keys and values from another is called cross-attention. Each action is 14 absolute target positions of the leader arms, 6 joints and a gripper for each; with deltas the result was worse. Assemble this scheme from blocks: the dimension is shown at every junction.

🧱 Sandbox

Click a block, then a slot in the scheme, or drag the block. A block in the scheme can be returned with a click. Some blocks are decoys.

Under the hood: ACT by the numbers

The encoder has 4 layers, d = 512, 8 heads. The paper says 7 decoder layers, but in the original code only the first one feeds the output (issue #25 in the ACT repository; LeRobot uses 1). ResNet18 starts from ImageNet weights. About 80M parameters in total, training takes about 5 hours on an RTX 2080 Ti, AdamW, learning rate 1e-5, batch 8, a separate model per task. Inference takes about 10 ms.

The target positions are tracked by the PID controller in the Dynamixel motors. The leader arms' positions are predicted, not the followers': the gap between them, through the PID, implicitly sets the force. Why deltas are worse the paper doesn't explain. A possible explanation comes from lesson 0.3: position targets together with a controller give a system that is stable without feedback, and that is the condition under which chunks help.

1202 tokens in, 100×14 out, about 80M parameters and about 10 ms of inference. The scheme has a block that does nothing at inference — the CVAE encoder. What is it for during training?

4 CVAE

The latent “style” variable

Why does ACT need a CVAE encoder? People show the same task differently: one goes around the vase on the left, another on the right.

Regression with mean squared error averages the manners: the mean of two detours leads straight into the vase. That was the conclusion of lesson 1.3.

ACT is trained with L1 instead of MSE. L1 doesn't average, it takes the median — the side with more demonstrations. The policy doesn't drive into the vase, but the second manner is lost.

To keep the manners, ACT is trained as a conditional variational autoencoder, a CVAE. During training a separate encoder looks at the joints and at k actions from the demonstration, without images, and compresses the manner into a vector $z$ of 32 numbers. The policy gets $z$ together with the images and reconstructs the actions. The loss is $L_1 + \beta\,\mathrm{KL}$ with $\beta = 10$ : the $\mathrm{KL}$ term pulls $z$ toward the standard normal distribution. At inference there are no demonstration actions, so the encoder is discarded and $z = 0$ is fed in. What this gives is explained in two ways. Pick one explanation and test it by experiment: the operator went around the vase 26 times on the left and 24 on the right; in the diagrams the robot drives bottom to top.

⚖️ A dispute between two explanations

Explanation A

z is style. At inference the policy picks a manner and goes around the vase now left, now right, like the operator.

Explanation B

z is needed only in training. At inference z = 0, and the policy does the same thing every time, like L1 regression.

L1 regressionno latent variable

ACT, z = 0as at inference

Random zsame decoder, z ~ N(0, I)

Left out of 5026

Under the hood: the toy CVAE

The detour side is the only thing that can be encoded. The encoder maps “left” to $z \sim \mathcal N(+\mu, \sigma^2)$ and “right” to $z \sim \mathcal N(-\mu, \sigma^2)$ . The decoder with $L_1$ outputs the median, that is, the side that is more likely given $z$ . $\mu$ and $\sigma$ are found by brute force to minimize the sum of two terms: the share of wrong sides (an error costs 1, in the same units as $L_1$ ) and $\beta\,\mathrm{KL}\bigl(\mathcal N(\pm\mu, \sigma^2)\,\|\,\mathcal N(0, 1)\bigr)$ .

\begin{aligned} \beta = 10:&\quad \mu = 0,\ \text{side error } 48\,\%,\\ \beta = 1:&\quad \mu = 0.39,\ \sigma = 0.92,\ \text{error } 34\,\%. \end{aligned}

At $\beta = 10$ the encoder encodes nothing: $z$ knows nothing about the side. At $\beta = 1$ it tells the sides apart. At $\beta = 10$ the encoder is better off encoding nothing: every bit about the side costs more than it saves in L1. $L_1$ regression without $z$ outputs the median of all demonstrations — the side with more of them. If there are equally many demonstrations on the left and on the right, the median is undefined: L1 is equally happy with any trajectory between the sides, including one straight into the vase. Random z are ten numbers from a generator with seed 21.

At inference ACT is deterministic and behaves like L1 regression. For the authors, success on human data dropped from 35.3% to 2% without the CVAE, but an independent rerun — a 2026 preprint with the telling title “The Latent That Never Was” — didn't reproduce that drop. And once there was a lot of data, the ALOHA authors themselves switched to diffusion: in ALOHA Unleashed, on the ShirtMessy task, a diffusion loss gave 70% and an ACT variant with L1 gave 25%. The problem from the lab remains: a chunk of 100 actions sees the world once every 2 seconds. How does ACT react faster without shortening the chunk?

5 Temporal ensembling

Blending overlapping chunks

How to react faster without shortening the chunk? A chunk of 100 actions executes for 2 seconds, and all that time the arm sees no new frames.

A new frame is taken into account only in the next chunk.

In the lab we executed only part of the chunk and composed a new one more often. The reaction got faster, but each new chunk starts slightly differently from how the old one was going, and jerks appear at the seams.

ACT queries the model at every step, and for the current step it averages all the predictions that have accumulated for it — at k = 100, up to a hundred of them. This is temporal ensembling (TE), averaging over time. The weights are $w_i = e^{-m i}$ , where $i = 0$ is the oldest prediction: at $m > 0$ each next, fresher prediction weighs slightly less. In the ACT code $m = 0.01$ . Let's look inside and scrub through the process step by step. At the top is a tape: steps run horizontally, the gripper's lateral offset vertically; thin lines are the chunks that predicted the selected step, the thick one is what the gripper executed. Below is the weight of each prediction, from the oldest on the left to the freshest on the right.

🧠 The model's thinking, step by step

Time runs left to right: 20 ms steps horizontally, the gripper's lateral offset in centimeters vertically. The hatched rectangle is the obstacle; the gripper goes around it. The bars below are the weight of each prediction in the action at the selected step.

Step150 m0.010 two detour strategies

Under the hood: the formula and the tape

a(t) = \frac{\sum_i w_i\,\hat a_i(t)}{\sum_i w_i}, \qquad w_i = e^{-m i},

where $i = 0$ is the oldest chunk. In the ACT code the predictions for the current step are stored in the order the chunks appeared, and the weights $e^{-0.01\,i}$ run from the oldest. At $k = 100$ the weights sum to ≈ 63.5, so the oldest prediction contributes ≈ 1.6% of the action and the freshest ≈ 0.6%. The smaller $m$ , the faster new observations are taken into account.

In the tape a new chunk starts at every step from where the gripper is now and converges to the chosen detour in 12 steps. In two-strategy mode, chunks before step 100 choose the detour above, and from step 100 on, below. The obstacle occupies steps 132–168 and ±4 cm in height.

Blending smooths the motion but trusts old predictions more, and the average of two different strategies can lead into the obstacle. In practice, temporal ensembling added 3.3 percentage points to ACT's success, in π0 (the large policy from lesson 1.6) it made things worse, and in ALOHA Unleashed, the next policy for the same platform, it wasn't needed: a chunk of 50 actions is executed whole there, and the authors call replanning once a second a limitation. Querying the model at every step is possible because ACT is fast: inference takes about 10 ms. What if the model thinks tens of times longer?

6 Latency and seams

While the model thinks, the robot moves

What if the model thinks for a long time? π0 from lesson 1.6 produces a chunk in 73 ms on an RTX 4090; for Gemini Robotics (Google DeepMind, 2025; lesson 3.7) the path from observation to chunk is about 250 ms. All that time the robot has to do something.

The new chunk arrives late and was computed from a frame that is already stale: the robot has moved on in the meantime.

Wait for the new chunk standing still — the robot freezes. Switch to it as soon as it arrives — a jump at the seam. Blend chunks, as temporal ensembling does — different strategies get averaged.

Real-Time Chunking (RTC; Black, Galliker, Levine, NeurIPS 2025) generates the next chunk while the current one is executing. The actions that will have been executed by the time inference finishes are frozen, and the rest are inpainted so as to continue the old chunk; no retraining is needed. Let's test it on a robot that goes around an obstacle above or below. The policy is generative, like π0: a chunk of 50 actions, and it picks the detour side at random until the robot has moved off the axis. First sketch the velocity curve for synchronous execution, then compare four modes at 200 ms of latency.

✏️ Sketch the curve and set the conditions

Top view: the robot (orange circle) drives left to right from start to goal and goes around the obstacle — the hatched circle — above or below. Below are three plots over time: lateral position, velocity, and the velocity jump per 20 ms step. The red dashed line is the 0.15 m/s threshold: a jump above it is a jerk. Modes: “sync” — stand still while the chunk is computed; “naive” — switch to the new chunk as soon as it arrives; TE — blending from section 5; RTC — freeze and inpaint.

Latency200 ms obstacle

Under the hood: the modes and other solutions

Sync. Execute 25 actions, stop, wait for inference, start the new chunk from where we stand.
Naive async. Inference of the new chunk starts one latency before the end of the 25 actions, from the frame at that moment. When the chunk arrives, the robot switches to it immediately at the right step, without smoothing the seam.
TE. A new chunk arrives at every step, computed from the frame $d$ steps ago. The action is the average of all chunks with weights $e^{-0.01\,i}$ , as in ACT.
RTC (simplified). The first $d$ actions of the new chunk are the same as the old one's. After that the new chunk is blended with the old one with the old one's weight $e^{-j/6}$ ; the last 25 actions are new only. The new chunk continues the same detour side: in real RTC that is done by guidance during generation.
Criteria. A pause is any step where the robot moves slower than 5 cm/s; the trip must be made with less than 0.1 s of pauses in total. The velocity jump is the largest change in velocity between adjacent 20 ms steps, threshold 0.15 m/s. The obstacle is a circle of radius 7 cm, the robot 2 cm.

There are other solutions. BID (ICLR 2025) samples several chunks at every step and picks one by consistency with the previous decision; the relative gain is 32%, with EMA 46%, at roughly twice the compute. Asynchronous inference in SmolVLA (a small open policy from Hugging Face, 2025; lesson 3.5) cut the time per task from 13.75 to 9.7 s.

In the toy only RTC survives 200 ms of latency. In the RTC paper, on π0.5 (the next version of π0, 2025; lesson 3.8), with artificial latency of +100 and +200 ms RTC doesn't degrade, the synchronous mode degrades linearly, and temporal ensembling rocks the robot until a protective stop. RTC performs the same motion 20% faster than the synchronous mode and lights a match at latency above 300 ms. In the trained version of RTC (December 2025) the latency is simulated right during training, so there's no extra compute at inference; on π0.6 (lesson 3.8) the result is on par with the original RTC. What chunks do large systems take in the end, and how much of them do they execute?

7 Chunks in large systems

How many actions per chunk in 2023–2025

What chunks do large systems take? Almost all modern policies output chunks and most often don't execute them whole: a long chunk gives a coherent continuation, frequent inference gives reaction. Most of the systems in the table are covered in later lessons; the lesson number is in parentheses.

System	Chunk	Executed before the next inference	Rate
ACT, 2023	100 (90 on the website)	with TE — queried at every step; without TE — all k	50 Hz, chunk — 2 s
Diffusion Policy, 2023	16	8	10 Hz
Mobile ALOHA (ACT), 2024	45	n/a	50 Hz
ALOHA Unleashed, 2024	50 (1 s)	50, no TE	50 Hz
π0, 2024 — Physical Intelligence (lesson 3.6)	50	25 (0.5 s) on 50 Hz robots; 16 (0.8 s) on UR5e and Franka	50 / 20 Hz
π0.5 + RTC, 2025 — the next version of π0 (lesson 3.8)	50	sync 25; with RTC — asynchronous	50 Hz
GR00T N1, 2025 — NVIDIA, a model for humanoids (lesson 3.7)	16 (63.9 ms per chunk)	n/a	actions 120 Hz, vision-language model 10 Hz
TRI LBM, 2025 — Toyota Research Institute's large behavior model	16 (1.6 s)	8 (0.8 s)	10 Hz
Atlas LBM, 2025 — Boston Dynamics and TRI, see below	48 (1.6 s)	24 (0.8 s)	30 Hz
SmolVLA, 2025 — a small open policy from Hugging Face (lesson 3.5)	50	asynchronous, by queue threshold	n/a
Gemini Robotics, 2025 — Google DeepMind (lesson 3.7)	n/a	n/a	“effectively 50 Hz”

What grew out of ALOHA

Mobile ALOHA (2024) — ALOHA on a mobile base for $32k, 3 cameras, ACT chunk of 45. Co-training with static ALOHA demonstrations (825 of them) raises success “up to 90%” according to the abstract. In the table for ACT: Call Elevator from 0 to 95%, Rinse Pan from 0 to 80%, Wipe Wine from 50 to 95%.
ALOHA 2 (2024) — the leader grippers need about 10 times less force, passive gravity compensation, 4 RealSense D405 cameras, a MuJoCo model.
ALOHA Unleashed (2024) — over 26,000 demonstrations for 5 tasks, a transformer with a diffusion loss and 217M parameters, inference in 0.043 s on an RTX 4090.
Atlas LBM (Boston Dynamics and TRI, 2025) — 450M parameters. Execution can be sped up 1.5–2× with no noticeable loss of quality.

Chunks, seams and latency are covered. Which of these ideas are easiest to get wrong? Let's check.

8 Check yourself

Eight common misconceptions

9 Takeaways

Lesson takeaways

Action chunking: from one frame predict $k$ actions. There are $k$ times fewer decisions, and the chunk gets through the pauses where a per-step policy gets stuck.
Chunk length has a window: short chunks get stuck, long ones react late to the unexpected. Large systems execute part of the chunk: π0 — 25 of 50.
ACT: ResNet18 → 1200 tokens, plus joints and z, encoder-decoder, 100 queries → 100×14 absolute positions in one pass.
The CVAE is needed only in training. At inference $z = 0$ , and ACT is deterministic.
Temporal ensembling trusts old predictions more ( $w_0$ is the oldest). It smooths, but averages different strategies and copes poorly with latency.
RTC stitches chunks under inference latency: it freezes what will have been executed and inpaints the rest.
A chunk helps on a system that is stable without feedback, and can hurt on an unstable one.

ACT is trained separately for each task: 50 demonstrations per skill, one policy — one task. The policy repeats the operator's motions, but knows neither the geometry of the scene nor words, so it can't figure out what exactly to do in a new situation. Part 2 starts from that problem.

Open questions

Inference latency is 150 ms, the robot runs at 50 Hz. What chunk length and execution scheme would you choose, and why?

150 ms is 7–8 steps. The chunk should cover the latency with margin, and the executed part should fit within the time in which you need to be able to react. For example, a chunk of 50 actions (1 s), executing about 25 (0.5 s), as in π0. The synchronous mode would give a 150 ms pause for every half-second of motion, naive switching — jumps at the seams. Better asynchronously with RTC: freeze the 8 actions that will execute during inference and inpaint the rest. Temporal ensembling is risky if the policy is generative and neighboring chunks may pick different strategies.

Why does an action chunk help on a stable system but can hurt on an unstable one, for instance when balancing?

A chunk is executed without feedback. If the system is stable on its own — say, a position controller holds the target — deviations between decisions die out, and there are fewer decisions and fewer errors. If the system is unstable, over the chunk the deviation grows by a factor of $\lambda^k$ , and there's no one to correct it until the next decision. According to ICLR 2026, such systems need noise during demonstration collection so the policy learns corrections; the chunks then have to be short.

Paper guide

Zhao, Kumar, Levine, Finn, “Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware”, RSS 2023.

Read closelyThe description of ACT: action chunks, temporal ensembling, the CVAE and the architecture diagram. The ablations: chunk length, temporal ensembling, the CVAE and teleoperation rate.
Safe to skipThe ALOHA construction details, unless you plan to build one.
Question to keep in mindWhy does ACT predict the leader arms' positions rather than the followers'? The hint is in the architecture section.

Resources

ALOHA and ACTProject site: autonomous rollouts and reactions to perturbations
ACT codeTemporal ensembling is in imitate_episodes.py
LeRobot: ACTHow to train ACT yourself
LeRobot: asynchronous inferenceChunks under latency in practice
Real-Time ChunkingPhysical Intelligence blog: the match and the comparison with temporal ensembling
Mobile ALOHAZ. Fu, T. Zhao, C. Finn, CoRL 2024
ALOHA UnleashedShoelaces and a shirt: diffusion on 26,000 demonstrations
Bidirectional DecodingPicking a chunk by consistency with the previous one, ICLR 2025

➡️

Next: Part 2. How can a robot work out what to do and where, if all it can do is repeat motions? In Part 2 it gets scene geometry and language: Transporter and CLIPort, PerAct and planners on large language models. RTC was invented for flow policies like π0: flow matching is covered in the supplementary lesson 1.8.

Was the lesson clear?

Found a mistake or inaccuracy? Report it on GitHub

Found this lesson useful? Say thanks — that is how the author knows the course is being read.