PPO, visualized inside Isaac Lab

PPO can look like one large optimization algorithm from the outside. Inside an Isaac Lab run, it is easier to understand as two alternating phases: collect a rectangle of experience from many robot worlds, then revisit that fixed rectangle in shuffled mini-batches to improve the policy.

The shape to remember

24 environment steps × 4,096 parallel environments = 98,304 transitions. That rectangle is one rollout. PPO computes targets over it, learns from it, then starts a fresh one.

This article follows Isaac Lab develop on September 1, 2026, which pins RSL-RL 5.4.1. Its Unitree Go2 rough-terrain configuration uses 4,096 environments, 24 rollout steps, four mini-batches, and five learning epochs. The dimensions are representative, not requirements. Change the two collection settings below to see how the batch grows.

One PPO iteration, step by step

Collect01Observe[N, O]
Collect02Act + stepa, V, r, d
Collect03Store[T, N, …]
Targets04GAEA, R
Learn05Sample[M, …]
Learn06Optimizeθ → θ′
Repeat07Fresh rolloutcursor → 0

Weights stay fixed while collecting. They change after every mini-batch while learning.

Only stage 02 advances physics. Stages 01–03 collect transition fields under fixed weights; stage 04 writes advantages and returns once; stages 05–06 revisit that complete frozen batch. Stage 07 keeps the updated networks and replaces the rollout.

Interactive walkthrough · trace every tensor

Open one stage at a time

01 / 07

Parallel worlds · N 4,096 Rollout horizon · T 24

Rollout · B = N × T 98,304 transitions

One mini-batch · M = B ÷ 4 24,576 samples · 20 optimizer steps

01 · ObservationManager

Turn N worlds into model-ready rows

Isaac Lab computes every configured observation term across all environments at once, processes each term, and concatenates terms along the feature axis. One row belongs to one environment slot.

Input: simulator state + commands
Transform: compute → modify → noise → clip → scale
Output: TensorDict · batch [N]
Invariant: one shared network, N rows

Isaac Lab · vectorized scene 4,096 synchronized worlds

base velocityjoint statecommandsprevious action

read all N slots

ObservationManager Process each named term

compute›modifiers›noise›clip›scale

configured terms concatenate on the last dimension

group by consumer

Actor input · Go2 default policy group [N, O_policy]

Critic input Go2 reuses policy [N, O_policy] optional asymmetric setups use a separate privileged critic group

Shape rule: the leading dimension is always the environment batch N; observation terms only change the feature width O.

02 · Actor, critic, environment

Cache first, then advance physics

The actor samples one action row per environment and the critic predicts one value. PPO freezes that pre-step snapshot before Isaac Lab transforms actions, runs four physics ticks, computes feedback, and resets only finished slots.

Input: obs_t · [N, O]
Policy: a_t · [N, A]
Critic: V_t · [N, 1]
Weights: fixed for all T collection steps

Observation at t policy / critic rows [N, O]

one batched forward per model

Actor · π_θ sample Gaussian action a_t [N,A] also μ, σ and summed log π_old

Critic · V_φ estimate state value V_t [N,1]

cache before physics

Pending transition obs, a, V, log π_old, μ, σ the behavior-policy snapshot stays frozen

Original sample a_t this is what storage keeps

optional wrapper clamp

ActionManager transform action

decimation = 4

Simulator physics × 4 terminate → reward → reset done slots

return

Feedback r_t, done_t, obs_t+1, extras done = terminated OR truncated

Boundary rule: for a done slot, obs_t+1 is already the reset-state observation. The stored action remains the actor’s original sample, not the optional action-clamped tensor.

03 · RolloutStorage

Complete one time row, then move the cursor

process_env_step() attaches post-step reward and done to the cached snapshot. After 24 rows, storage contains 98,304 transitions arranged time-major.

Before step: obs, action, value, old policy stats
After step: reward + done
One row: [N, …]
Full rollout: [T, N, …]

Cached by act() pre-step record

obs [N,O]action [N,A]V [N,1]log π [N]μ, σ [N,A]

Attached by process_env_step() post-step feedback

reward [N]done [N]

timeout first adds γ times cached pre-step V_t to reward; done stays 1

copy row t

Cursor t → t + 1 scalar fields reshape to [N,1]

time ↓ / env →n = 0n = 1…n = N−1 t = 0[0,0][0,1]…[0,N−1] t = 1[1,0][1,1]…[1,N−1] ⋮⋮⋮⋱⋮ t = T−1[T−1,0][T−1,1]…[T−1,N−1]

Storage rule: fields are separate tensors that share the same [T,N] leading axes. RSL-RL does not allocate a separate next_obs tensor.

04 · Bootstrap + GAE

Walk backward without crossing resets

The critic evaluates the final observation once. GAE then scans from T−1 to 0, carrying later evidence backward until a done mask cuts the chain.

Bootstrap: V(obs_T) · [N,1]
Direction: t = T−1 … 0
Actor target: A · [T,N,1]
Critic target: R · [T,N,1]

Final observationobs_T[N,O]

critic only

BootstrapV_T = V(obs_T)[N,1]

computed right → left

t = 0t = 1done = 1mask = 0 · stop…t = T−2t = T−1

1 · TD residualδ_t = r_t + γ(1−d_t)V_t+1 − V_t

2 · AdvantageA_t = δ_t + γλ(1−d_t)A_t+1

3 · Return targetR_t = A_t + V_t

Actor signalnormalize A across T × N[T,N,1]

Critic targetkeep return R unnormalized[T,N,1]

Boundary rule: V_t+1 comes from the next stored value, except at the last row where V_T is used. Timeout rewards are already corrected, but the recursion is still masked.

05 · Flatten + shuffle

Change the view, not the data

For a feed-forward policy, every field needed by the PPO update is flattened in the same order, indexed by one random permutation, and split into four equal mini-batches. RSL-RL reuses that partition for five learning epochs.

Before: [T, N, …]
Flatten: [B, …] · B = T × N
Mini-batch: [M, …] · M = B ÷ 4
Reuse: 4 batches × 5 epochs = 20 updates

1 · time-major indices [T,N]

0,00,10,20,3 1,01,11,21,3 2,02,12,22,3

flatten update fields

2 · flat positions [B]

0123NN+1…B−1

randperm once

3 · shuffled indices perm[B]

70N+1B−1312…2

Mini-batch 1perm[0 : M][24,576, …]

Mini-batch 2perm[M : 2M][24,576, …]

Mini-batch 3perm[2M : 3M][24,576, …]

Mini-batch 4perm[3M : 4M][24,576, …]

Same frozen partition

epoch 1
epoch 2
epoch 3
epoch 4
epoch 5

Each epoch visits all 4 mini-batches → 20 optimizer steps

Each sampled index selects every field together: obs [M,O]action [M,A]old V / log π / A / R [M,1]old μ, σ [M,A]

Frozen-data rule: targets and old-policy statistics never change during the five epochs. If B is not divisible by four, the remainder is omitted; this Go2 batch divides exactly.

06 · PPO update

Re-score stored actions, then update once

The current networks evaluate the exact actions collected earlier. PPO compares current and frozen behavior probabilities, combines actor, critic, and entropy terms, clips gradients, and runs one optimizer step per mini-batch.

Frozen: action, old log π, old V, A, R, μ, σ
Recomputed: new log π, value, entropy, μ, σ
Clip: objective incentive at 1 ± ε
Output: updated actor θ′ + critic φ′

Frozen mini-batch behavior data + targets

obsstored actionlog π_oldV_oldAR

re-score stored action

Current actor + critic new forward pass

log π_θV_φentropyμ_θ, σ_θ

later mini-batches see the latest weights

compose loss

Total loss surrogate + c_v value − c_e entropy Go2 also clips the value prediction around V_old

backward

One optimizer step zero grad → backward → clip → step clip actor and critic gradient norms to 1.0

Clipped surrogate · example with A > 0 r = exp(log π_θ − log π_old) = 1.28

0.8 1.0 1.2 1.28

ratio remains 1.28 objective uses min(1.28A, 1.20A) = 1.20A

Adaptive KL: analytic KL(old ‖ current) adjusts the learning rate before the step; it does not end the epoch. 20 total steps: 4 mini-batches × 5 epochs, each compared with the same frozen behavior statistics.

Clip rule: PPO clips the improvement incentive in the surrogate objective. It does not clamp the actual policy ratio and is not a hard trust region.

07 · Fresh on-policy data

Keep the weights, overwrite the rollout

The actor and critic were updated in place. Storage resets only its write cursor, unfinished episodes continue from obs_T, and the next act() starts a fresh rollout with the new policy.

Keep: θ′, φ′, obs_T, allocated tensors
Reset: storage write cursor → 0
Do not reset: all environments globally
Replace: old rollout with fresh on-policy data

Keep across iterations updated actor θ′ + critic φ′ current obs_T unfinished episodes continue; only done slots reset inside env.step()

Reset logically RolloutStorage cursor → 0 allocated tensors stay in memory and are overwritten

1act with θ′one batched policy
2collect T new steps4,096 continuing world slots
3fill fresh rollout98,304 transitions
4run 20 updatesθ′ → θ″, φ′ → φ″

No replay database · no separate old-policy network · no policy copy per robot

On-policy rule: once the update finishes, the old rollout is logically discarded. The next optimization phase learns only from experience collected by the policy that now exists.

Observe · use the stage tabs or arrow keys

The important separation is temporal. During collection, RSL-RL runs the actor and critic under inference mode while Isaac Lab advances the world. During learning, the simulator stops advancing while the optimizer revisits the fixed rollout. Network weights do not change halfway through a rollout.

The collection phase in code

The current on-policy runner reduces to this shape:

obs = env.get_observations()

for t in range(num_steps_per_env):
    actions = ppo.act(obs)                # caches obs, action, V, old log π
    next_obs, reward, done, extras = env.step(actions)
    ppo.process_env_step(next_obs, reward, done, extras)
    obs = next_obs

ppo.compute_returns(obs)                  # bootstrap + backward GAE
ppo.update()                              # shuffled mini-batches, then clear cursor

There is a subtle but useful detail in that ordering. The transition is assembled across two calls. act() records what was known before physics: obs_t, a_t, V_t, the old action log-probability, and the action distribution parameters. After Isaac Lab steps, process_env_step() attaches r_t and d_t and copies the finished record into time row t.

Isaac Lab resets ended environment slots before returning the next observation. For a slot whose done flag is true, obs_t+1 is already the initial state of its next episode. The done mask is therefore essential when GAE runs backward.

What is actually in the rollout buffer?

With N environments, T rollout steps, action width A, and observation-group width O_g, the core tensors are:

Field	Shape	Why PPO keeps it
Observation group `g`	`[T, N, O_g]`	Re-run actor and critic during learning
Sampled action	`[T, N, A]`	Score the exact behavior action again
Reward, done	`[T, N, 1]`	Build return and stop credit at episode boundaries
Old value	`[T, N, 1]`	Compute advantage and optionally clip value updates
Old action log-probability	`[T, N, 1]`	Form the PPO probability ratio
Old Gaussian mean and standard deviation	`[T, N, A]`	Measure analytic KL for adaptive learning rate
Return, advantage	`[T, N, 1]`	Critic target and actor learning signal

This structure is usually called RolloutStorage rather than a replay buffer for a reason. It holds one fixed-horizon, on-policy batch. After the PPO update, only the write cursor is reset, and the next rollout overwrites the old one.

GAE turns rewards into a learning signal

Rewards alone do not say whether one action was surprisingly good given the state. The critic supplies a baseline. RSL-RL computes a temporal-difference residual and then accumulates it backward:

TD residualδ_t = r_t + γ(1 − d_t)V_t+1 − V_t

AdvantageA_t = δ_t + γλ(1 − d_t)A_t+1

Return targetR_t = A_t + V_t

The γ term discounts distant outcomes. λ controls how much later evidence flows backward: lower values lean toward low-variance one-step estimates; higher values use longer credit chains. Before optimization, advantages are normalized over the rollout by default.

Timeouts need separate care. For a slot marked time_outs, RSL-RL v5.4.1 applies r_t ← r_t + γV_t before storage, using the value cached before the step—not a value of a terminal observation. The done mask remains set, so the backward GAE recursion still cannot cross the reset boundary.

Sampling changes the view, not the data

The rollout rectangle is time-major because that makes collection and GAE natural. A feed-forward policy does not need that structure during the gradient update, so RSL-RL flattens [T, N, …] to [B, …], where B = T × N, then selects a random permutation.

For the Go2 example:

Rollout: 24 × 4,096 = 98,304 transitions
Mini-batch: 98,304 ÷ 4 = 24,576 transitions
Optimizer steps: 4 mini-batches × 5 epochs = 20
Reuse: every transition participates once per epoch, then is discarded after the update

The current feed-forward generator creates one permutation and reuses that partition across the learning epochs. If the batch is not divisible by the mini-batch count, the remainder is omitted; common Isaac Lab configurations choose divisible sizes. Recurrent policies cannot freely flatten across resets, so their generator preserves padded trajectory fragments and accompanying masks.

What PPO clips—and what it does not

For every mini-batch, the actor calculates a new log-probability for the action that the old policy actually sampled. Their difference becomes a ratio:

probability ratio r_t(θ) = exp(log π_θ(a_t|s_t) − log π_old(a_t|s_t))

r = 1 means the action is just as likely now. PPO limits the benefit of moving this ratio beyond 1 ± ε; it does not simply clamp every gradient or replace the sampled action.

The minimized training loss combines three signals:

Clipped surrogate loss: improve probability for positive-advantage actions and reduce it for negative-advantage actions, without earning extra objective improvement for moving too far.
Value loss: move the critic toward R_t; the Go2 configuration used here enables clipped value loss.
Entropy bonus: resist collapsing the Gaussian action distribution too quickly.

After backpropagation, actor and critic gradients are norm-clipped and the optimizer updates both networks. An optional adaptive schedule compares the old and current distributions with analytic KL and changes the learning rate; it does not stop the epoch early.

Three details that make the mental model click

The old policy is stored as numbers

Thousands of Isaac Lab environments do not each own a policy copy. One actor handles the whole [N, …] batch. PPO’s behavior-policy reference comes from the stored old log-probabilities and distribution parameters. The same actor module is updated in place before the next rollout.

“Done” belongs to the transition that just ended

Isaac Lab steps, computes reward and termination, resets ended slots, and only then computes the returned observation. Storage keeps the pre-step observation and the post-step reward/done pair; it does not need a second full observation tensor for every transition.

Action clipping is a different clip

PPO’s ratio clip constrains the learning objective. If the Isaac Lab RSL-RL wrapper is also configured to clip actions, that clamp happens on the tensor sent to the environment. The stored action and its log-probability still describe the actor’s original sample.

Takeaway

One PPO iteration in Isaac Lab is a disciplined change of tensor layout:

Isaac Lab produces observation groups shaped by parallel environments.
The actor and critic add action, value, and old-policy statistics.
Environment feedback completes one row in [T, N, …] rollout storage.
A final value bootstrap and backward GAE create returns and advantages.
Feed-forward training flattens to [T × N, …] and selects mini-batches.
PPO revisits the stored actions, clips the actor’s optimization incentive, updates actor and critic, and discards the rollout.

The simulator makes experience wide; time makes it tall; GAE makes it useful; mini-batch sampling makes it trainable. Then the updated policy goes back to the worlds and earns the next rectangle.