How RL Training Actually Works in Robotics

The Anatomy of a Robot Brain

In reinforcement learning, the robot's decision-making strategy is called its policy ($\pi$). At a conceptual level, a policy is just a mathematical function that answers a simple question: "Given what the robot senses right now, what action should it take?"

Because a physical robot moves through continuous 3D space with infinite possible joint positions, velocities, and sensor readings, we cannot store decisions in a discrete lookup table or hardcode them with rules. Instead, modern robotics RL uses deep neural networks as function approximators. Their adjustable numerical weights: The internal mathematical parameters of a neural network that scale incoming signals. During training, optimization algorithms adjust these values to minimize errors and improve task performance. allow the robot to smoothly generalize its motor skills across states it has never seen before.

"Neural network" is a broad umbrella term that encompasses everything from massive Vision Transformers to convolutional image networks. If you don't know what those are right now, that's fine, just know they are variants. I am going to keep it simple in this write-up and focus on a single variant to illustrate the core principles of robotics RL.

The Dual-MLP Architecture

An MLP (Multi-Layer Perceptron) is the workhorse of robotics RL. It is a fully-connected feedforward network. This means all the nodes in each layer are all connected to the nodes in the next layer. Actions happen by the network from propagating calculations from left to right (feeding the input forward through the network). The linear node layers are separated by non-linear activation functions (like Tanh or ELU) so the network is able to learn non-linear properties from its inputs. If textual explanations aren't helpful for you (they're not for me), check the figure below.

This neural network isn't as much of a mysterious black box as many people make it out to be. It's just a structured pipeline of floating-point numbers:

• Inputs: At any instant $t$, the network receives a clean snapshot of the robot's physical body. For example, node 1 might get the joint 1 rotational velocity, node 2 might get the gripper position, and so on. Each node has a fixed input assignment across all timestamps.
• Weights and Biases: A weight is a floating point number that scales the signal traveling along a connection between two neurons. As such, the weights are assigned to the edges of the network graph. In a 30x512 MLP all 30 input nodes connect to all 512 layer 2 neurons, so these connections form a 2D table (a $30\times 512$ weight matrix) where each row describes a node and each column is used to store the values for its edges that connect to another node. A bias on the other hand is a floating point number which represents a baseline offset attached to the neuron itself. Biases are not on input nodes but they are included on all layers after that, including the output layer. To store this we can just simply use a 1D vector whose size is equal to the number of nodes in the corresponding layer.
• The Squashing Functions: If you just multiply inputs by weights and add biases, stacking 100 linear layers is mathematically identical to 1 layer ($W_3\cdot W_2\cdot W_1=W_{\text{combined}}$) so adding new layers adds no benefit and your network is not able to learn complex non-linear behavior. Non-linear activation functions like Tanh (which is really just an s-curve) are used to squish values into some range non-linearly. For Tanh this is $[-1,+1]$. The simple fact that the tanh is an s-curve instead of a straight line gives the network the mathematical curvature needed to learn complex behaviors. Its such a simple concept but it is one of the most impactful discoveries in Machine Learning.
• Neuron Calculations & Activations: Every hidden neuron calculates its output in two simple steps. First, it computes an internal "pre-activation" $z$ by multiplying incoming inputs by their edge weights and adding its bias ($z=w_1{x}_1+\cdots +w_n{x}_n+b$). Second, right before that value leaves the neuron, its passed to the squashing function ($h=\tanh (z)$). That final squashed output $h$ is what we call the neuron's activation: The final firing value sent along outgoing edges to the next layer.

Proximal Policy Optimization (PPO) is what I will mainly be discussing in this article. It is a reinforcement learning algorithm which uses two completely separate MLPs. One is used as an "Actor" and one is a "Critic". The Actor drives the robot and decides actions to take while the Critic evaluates how good the Actor's decisions turned out.

In this setup, the Actor has its own weights $\theta$ to command joint movements, and the Critic has its own weights $\varphi$ to predict expected rewards.

Keeping the two networks completely separate is standard practice in robotics because they are optimizing for two different things. The Critic tries to accurately predict reward scores, while the Actor learns to control the robot smoothly. If they shared layers, large error corrections from a mispredicted score could easily drown out the subtle shifts the Actor needs for precise motor control.

The Multi-Rate Execution Loop

In robot reinforcement learning, control is almost always split across two distinct frequencies: a low-frequency policy loop (typically 20 to 50 Hz) where the neural network evaluates sensor data and makes decisions, and a high-frequency physics and control loop (500 to 1000 Hz) where the motors and contact dynamics are managed.

While a single clock would be simpler, it isn't usually a viable option. The system has incompatible timing requirements:

• The physics and motors must run fast (500+ Hz): Resolving stiff contacts, friction transitions, and joint damping requires sub-millisecond numerical integration. If motor feedback is evaluated too slowly, the discrete solver overshoots the contact constraint, corrects, overshoots again, and the joint chatters.
• The neural network cannot run that fast (20–50 Hz max): Polling hardware buses, normalizing observation vectors, synchronizing camera frames, and computing forward passes through deep networks takes time. A 500 Hz loop imposes a strict 2-millisecond hard deadline. Even with a high-end GPU, most networks cannot reliably finish in that time window.

Because the AI brain updates at 20 Hz while the physical motors require updates at 500 Hz, the trained policy cannot be used to command raw motor torques directly. Doing so would mean holding a static torque for 50 milliseconds, leaving the robot uncontrolled between decisions. On top of that, asking a neural network to calculate raw torques forces it to learn the robot's mass matrix, gravity compensation, and joint inertia from scratch which complicates the training.

Instead, the policy outputs target positions (or velocity deltas), and a deterministic proportional-derivative (PD) controller: A standard feedback loop generating motor torque proportional to position error (P) and velocity error (D). running at 500 Hz works to get the joint to that target until the next policy decision arrives. With this setup the PD controller acts as a sort of compliant: The capacity of a robot joint or controller to yield flexibly in response to external contact forces, absorbing impacts rather than acting as an unyielding, rigid body. virtual spring-damper, pulling the joints to where they need to be while (ideally) not overshooting and causing jitter.

Collecting Experience in the Simulation

So now you know about the actor and the critic networks, and you know their operating frequency. We can now look at the broader training loop. RL training alternates between two distinct phases: rollout collection: Executing the active policy in simulation to gather experience and optimization: Updating the neural network weights to improve behavior .

During simulation training, each step of this rollout: The process of executing an active policy in an environment to collect a trajectory of states, actions, rewards, and transitions before calculating policy gradients. collection cycle follows a fixed sequence:

1. Initialize: The environment initializes the robot to a valid starting state and randomizes physical parameters like payload masses, joint friction, surface contact properties, and visual lighting. (We explored tuning these parameter distributions to fix visual overfitting in Part II: Hardening the Robot Brain). This forces the policy to learn motor behaviors that work beyond idealized simulation.
2. Observe & Normalize: The policy receives raw proprioception: Internal sensory data measured from the robot's own body, including joint angles from encoders, joint velocities, and IMU readings. data along with target task goals. Because angular velocities ($10$ rad/s) and sub-millimeter position offsets ($0.001$ m) operate on vastly different numerical scales, a running filter normalizes each observation channel to zero mean and unit variance: Shifting data so its average is 0 (μ = 0) and scaling it so its spread is 1 (σ = 1). This ensures all sensor channels have equal numerical weight in the neural network. before network evaluation.
3. Action Decision & Baseline Evaluation: The actor network uses the sensor data and chooses a target movement (the setpoint $\mu (s_t)$), with an extra bit of random exploration noise to help it discover new strategies. At the same time, the critic network generates a baseline score guess $V(s_t)$ representing how much total reward the robot is expected to earn from this state. The Actor's action $a_t$ is then dispatched to the low-level controller.
4. Physics Sub-Stepping & Environment Step: The robot executes the movement. The low-level PD controller holds the target setpoint while stepping the physics engine (for example, 25 micro-steps at 500 Hz to fill one 20 Hz decision window, as described above). The simulator then returns the robot's new state $s_{t+1}$, an immediate reward score $r_t$ for the action, and a flag $d_t$ indicating if the task completed or failed. The reward is something that you setup manually (more on that later) and is calculated based on the state of the simulation and how close it is to what you want it to be.
5. Saving the Experience: The information about what the robot saw $s_t$, what action it took $a_t$, the reward it earned $r_t$, and the critic's baseline prediction $V(s_t)$ is saved into a rollout buffer: A contiguous memory buffer holding collected experiences (states, actions, rewards) from simulation before calculating weight updates. in GPU memory.

How to Train 4,096 Robots in a Single Matrix Operation

RL training takes a lot of episodes to get the robot to do what you want it to do. You can just train it sequentially on your CPU if you want but its so slow that its impractical. Most serious modern RL training tends to use parallelization on GPU(s). This allows you to go from training 1 robot at a time to training 4,096 robots all at the same time. If you're training 4,096 robots at the same time in simulation, you might wonder: do we need 4096 copies of the neural network or do we loop through the robots one by one to find out what the neural net says that robot should be doing?

Neither. In GPU memory, there is only one single set of network weights ($W_1$). Instead of a slow iterative loop, the simulator stacks all 4,096 robot state vectors into a single 2D table called a Tensor (specifically, an $X$ matrix with dimensions $4096\times 30$). Assuming you have 30 input items e.g. joint velocity, position, etc. It uses this single shared "brain" to control all robots simultaneously.

The GPU then computes a single matrix multiplication:

When it's time to update the network, the backward pass does the exact same thing in reverse: multiplying the transposed inputs $X^{\top }(30\times 4096)$ against the error tensor $d{Z}_1(4096\times 512)$ combines all 4,096 robot error vectors into one single consensus gradient matrix ($30\times 512$) without ever looping row by row or making copies of the network.

A Deeper Look Into Learning

Once the parallel simulators finish gathering data from running their episodes, data collection stops and the optimization phase begins. At this point the GPU now holds a full rollout buffer containing tens of thousands of recorded state tuples: what the robot saw $s_t$, what action it took $a_t$, the immediate reward it received $r_t$, the critic's baseline prediction $V_{\varphi }(s_t)$, and whether the step terminated $d_t$.

That buffer cannot go straight into the optimization process just yet though. Immediate rewards say almost nothing about whether an action was good in the big picture. A gripper that opens at step 10 may not collect any reward on that tick, but that open gripper is the only reason another grasping action 30 steps later is even possible.

Before updating any weights in the network, we need to pass the entire rollout buffer through a two-stage optimization workflow:

1. Compute Advantages: We use a process known as Generalized Advantage Estimation (GAE) to give us some additional data to help optimize our network. GAE computes a refined advantage: A positive or negative number which tells us if this specific movement turned out better or worse than what the critic expected from this exact position. score ${\hat{A}}_t$ (for the Actor) and a value target: The actual total cumulative reward the robot earns from that moment forward. While the robot was moving, the Critic made a guess: "I predict this state is worth 4.0 points." Once the simulation finishes and GAE runs, we now have the data to calculate what the state was actually worth (e.g. 7.2 points) score ${\hat{R}}_t$ (as the answer key for the Critic) for every recorded step. Each of these values are saved directly into the rollout buffer. The advantage score is used to tell the Actor which actions actually worked, and the value target is to teach the Critic what total score it should have predicted.
2. Optimize Networks (PPO): Once the rollout buffer is fully enriched with advantage scores and value targets, the training loop enters the optimization phase. We can't push hundreds of thousands of recorded steps through backpropagation in a single giant push, so the optimizer shuffles the entire buffer randomly and slices it into smaller chunks called mini-batches (typically 4,096 steps each). Shuffling ensures each mini-batch contains a diverse mix of experiences to keep the robot "well-rounded." Without it, the network would train on thousands of nearly identical consecutive frames, becoming an expert at one tiny slice of the simulation while failing the rest of it. For each mini-batch, the GPU updates both the actor and critic networks in parallel. The Actor consumes the advantage scores (${\hat{A}}_t$) to nudge its movement targets toward winning movements, while the Critic consumes the value targets (${\hat{R}}_t$) to minimize its prediction error. Because simulating physics is computationally expensive, the optimizer makes 3 to 5 full passes (epochs) over the mini-batches to extract maximum learning before clearing the buffer and sending the robots back into simulation.

When computing advantages with GAE, you have two primary dials to tune.

• The Overall Task Horizon ($\gamma$): Dictates how far into the future the robot cares about rewards. At $\gamma =0.99$ and a 20 Hz policy loop, the robot effectively plans roughly 5 seconds ($1/(1-\gamma )=100$ steps) ahead. Enough time to reach, grasp, and lift. If you set $\gamma =0.90$, the robot only looks 0.5 seconds ahead and becomes completely short-sighted. Set it to 1 and you essentially tell the robot time doesn't matter and you end up with a procrastinating robot that takes ages to do anything.
• The Sub-Action Credit Window ($\lambda$): Dictates how quickly credit decays backward between individual movements. In robotics, $\lambda \approx 0.95$ creates an effective credit window of about 1 to 2 seconds. Just long enough for an early wrist alignment to get credit for a successful grasp. If you set $\lambda =0.5$, credit dies in 250 ms which makes it hard to give credit to net good actions even if they were not recent. If you set $\lambda =1.0$, an early wrist rotation gets unfairly blamed for a random collision 400 steps later.

The Horizon Bug: Why Your Robot Suddenly Drops Everything

There is one notorious gotcha in GAE that trips up almost everyone who builds a custom simulation environment. In robotics, a failing simulation episode ends for two main reasons:

• Timeout (truncated): The episode reached its time limit. The robot didn't do anything wrong per say, the clock just ran out.
• Crash / Failure (terminated): The robot dropped the object, collided with a table, threatened humanity, or did anything else you told it it can't do.

If your simulation environment code accidentally merges both into a single done = True flag, the math tells the robot that whatever it was doing when the timer expires has zero future value. It could have been doing a perfectly acceptable, even desirable action, just ran out of time. But the advantage that GAE gives you in this scenario is sharply negative since it "failed", and GAE propagates that penalty backward into the past. From this your robot develops late-episode panic, learning that whatever it was doing before when it ran out of time was a terrible idea and this can frighten it into never taking the beneficial action you want it to take since it learned it will get penalized heavily for it. The simple way out of this is to handle simulation success, failures, and timeouts all as their own independent end states.

What an Advantage Does to the Weights

In the previous section, we saw how GAE calculates an advantage score (${\hat{A}}_t$) for every single action recorded in our rollout buffer. But this brings up a fundamental question: how does a simple score actually reach inside the neural network and update its millions of internal weights?

To appreciate why this is tricky let's compare how standard supervised learning works (like training a computer vision model to spot cats). In supervised learning, you have an exact answer key. If the model predicts "dog" when the label says "cat," the error is straightforward: you subtract the prediction from the ground truth and tell the network exactly what it should have done instead.

In robotics reinforcement learning, there is no precise answer key. No supervisor is standing over the robot telling it: "at timestep 42, your elbow angle should have been precisely 47.3 degrees." All the robot gets is trial and error. To discover what works, the Actor network outputs a central baseline setpoint ($\mu (s_t)$). This is its current best guess for what the joint angles should be. It then adds a small amount of random exploration noise (a gentle "wobble") to pick the actual action ($a_t$) sent to the low-level controller.

Now suppose the robot wiggles its elbow a few degrees higher than its baseline and attempts a grasp. The simulation runs, and GAE evaluates the result, handing us an advantage score ${\hat{A}}_t$. If the grasp succeeded, ${\hat{A}}_t$ is positive; if the arm crashed, ${\hat{A}}_t$ is negative. How do we convert that signed scorecard into precise adjustments across all the layers of the network?

Diving Into Gradients and Backpropagation

This is where the concept of a gradient comes in. Earlier, we established that a neural network is a giant web of floating-point numbers called weights. You can think of each weight as a tiny adjustable knob. A gradient is simply a sensitivity measurement for one of those knobs: "If I turn this specific weight knob slightly to the right, how much does the network's output change, and in what direction?"

Calculating that sensitivity for a single knob connected to the output layer is easy. But modern robotics networks have hundreds of thousands, or millions, or even billions, of interconnected weight knobs spread across multiple deep layers. Adjusting a weight in the first layer ripples through every hidden neuron before finally affecting the joint motors at the output.

Figuring out how every single knob contributed to the final motor command is the job of backpropagation: An efficient algorithm that calculates the gradient (sensitivity) of every network parameter by propagating error signals backward through the layers using the calculus chain rule. . Because every neuron simply multiplies its inputs by weights and passes the sum through a smooth squashing function (like the Tanh activation we saw in Fig. 2), the entire network is mathematically differentiable. Backpropagation applies the calculus chain rule in a single, lightning-fast sweep backward from the output motors all the way to the input sensors. In that single backward pass, every single weight knob in the network receives its own gradient, showing precisely how much turning that knob will shift the output setpoint.

The Policy Gradient

Once backpropagation tells us how to steer the network's output, the policy gradient update ties the whole loop together. The update sent backward to adjust the Actor's baseline setpoint for a joint is remarkably intuitive:

Let's break down each piece of this relationship.

First, $\mu (s_t)$ is the Actor's baseline setpoint for the current sensory state $s_t$—what the network's current weights originally wanted to do. Second, $a_t$ is the actual exploratory action the robot executed after adding random noise. The difference between them, $(a_t-\mu (s_t))$, represents the exploration offset: the exact direction and magnitude the robot wiggled (for example, $+3^{\circ }$ higher on the elbow joint). Finally, ${\hat{A}}_t$ is the advantage score from GAE, and the proportionality symbol ($\propto$) indicates that the size and direction of our weight adjustment scales directly with this product.

Look at how this product behaves in both scenarios:

When an exploratory movement succeeds, GAE gives it a positive advantage (${\hat{A}}_t>0$). Multiplying a positive exploration offset by a positive advantage produces a positive update. Gradient descent turns the network's weight knobs so that next time the robot encounters this state, its baseline setpoint $\mu (s_t)$ naturally shifts toward that exploratory angle. The robot has reinforced a winning habit.

Conversely, when an exploratory movement leads to a dropped object or collision, GAE assigns a negative advantage (${\hat{A}}_t<0$). Multiplying that same positive offset by a negative number flips the sign of the update. The optimizer nudges the weights in the opposite direction, pushing the baseline setpoint $\mu (s_t)$ away from that failed movement. Without anyone ever telling the robot what the perfect motor command was, trial-and-error combined with a signed scorecard nudges millions of parameters in the right direction.

This is where the parallel matrix design we looked at in Figure 4 comes back into play. For training at scale, rather than updating weights one robot at a time, the GPU computes this backward pass across all 4,096 simulation environments simultaneously. By multiplying the transposed state matrix against the advantage-weighted error tensor we turn thousands of independent trial-and-error experiments into one single consensus gradient matrix in fractions of a millisecond. It still feels a bit like magic to me.

Why We Need a Leash During Training

While this policy gradient formulation is remarkably powerful, it has a vulnerability: it is very easy to overfit: When a trained policy performs exceptionally well in its training environment but fails to generalize to new, unseen environments. or over-react to a single batch of data.

If a robot gets lucky on a few rollouts and stumbles into a high reward by chance, an unconstrained gradient step might steer the network weights aggressively in that direction. Because neural network weights are interconnected, a large update intended to fix an elbow movement in one state can inadvertently destroy the shoulder stabilization skills the robot learned hours earlier. If you attempt to run multiple training passes over the same recorded batch, the policy rapidly overfits to that stale data, causing overall performance to crash catastrophically.

In early RL algorithms, the only solution was to take tiny, timid update steps and throw away expensive simulation data after a single pass. To make training fast, stable, and data-efficient, we need a way to apply these gradient updates while putting a strict mathematical leash on how far the policy is allowed to change. That brings us to the core engine of PPO.

Inside the PPO Optimization Engine

Earlier, when we outlined the two-stage optimization workflow, we noted that the optimizer makes 3 to 5 full passes (epochs) over the recorded simulation buffer before throwing it away. Let's dive into how we are able to do that without over-fitting or over-correcting.

In older reinforcement learning methods, you could only run one single gradient step (1 epoch) on your rollout buffer. The moment the network weights updated even once, the robot's brain changed, meaning the recorded experiences in the buffer were now "stale" (off-policy). If you ran a second gradient pass on that same data, the updated network would overfit to lucky anomalies, take an overly aggressive step, and catastrophically crash the robot's balance. Throwing away hundreds of thousands of expensive physics steps after just 1 gradient pass was slow and wasteful.

This is where Proximal Policy Optimization (PPO): An on-policy reinforcement learning algorithm using a clipped surrogate objective to constrain policy step sizes without second-order matrix solves. gets its name. In Latin, proximus means "nearest" or "in close proximity." PPO places a strict mathematical leash on every gradient update, guaranteeing that the updated policy remains in the immediate neighborhood of the policy that gathered the data. This proximal leash is the exact mechanism that makes it safe to run 3 to 5 epochs of gradient descent on the same simulation data without the robot's policy collapsing.

PPO enforces this proximal leash by tracking the action probability ratio $r_t(\theta )$, which compares the updated policy's decisions against the old policy that recorded the buffer:

If $r_t(\theta )=1.15$, the updated policy is 15% more likely to take that action. If $r_t(\theta )=0.85$, it's 15% less likely. (Note: this ratio $r_t(\theta )$ is completely separate from the environment reward $r_t$).

To keep the update within safe bounds, PPO clips this ratio inside a narrow trust window (typically $1\pm ϵ$, with $ϵ=0.20$):

If an action succeeded (${\hat{A}}_t>0$) and its probability ratio reaches $1.20$, the objective flattens: the gradient drops to zero and the optimizer stops pushing. This keeps the network from over-committing to what might have just been a lucky bounce. Meanwhile, the $\min$ operator ensures the clipping is one-sided: if an update accidentally makes a good action less likely, it is never clipped, allowing full gradient flow to correct the mistake.

In production, the actor policy loss, critic regression, and an exploration bonus are combined into a single master objective:

The entropy bonus: A mathematical measure of randomness in the policy's action distribution. Higher entropy prevents premature convergence to poor local minima. $S[{\pi }_{\theta }]$ pays the robot to stay curious. Without it, a robot that earns a small reward for standing still might collapse its action distribution to near-zero variance early in training, locking into a mediocre strategy and never learning to walk.

The Secret Sauce to Making PPO Work

In the original PPO papers they make it seem like if you simply implement the clipped objective formula $L^{\text{CLIP}}$ then badda-bing badda-boom you have a working robot policy. In practice, if you write a pure mathematical implementation of PPO and hit "train," your robot will almost certainly fail spectacularly.

As it turns out, a huge portion of PPO's real-world success and stability doesn't come from the loss equation alone. We also have a handful of critical low-level code implementation details that research studies have surfaced over the years. For our use-case, three specific engineering practices make a big difference here:

1. Orthogonal Initialization: Think of a deep neural network as a chain of 4 or 5 audio amplifiers in a row. If each amplifier multiplies the volume by $1.5\times$, after 5 layers the sound is screaming at $1.5^5\approx 7.6\times$ volume, causing severe clipping and distortion. If each amplifier multiplies by $0.5\times$, after 5 layers the volume collapses to $0.5^5\approx 0.03\times$, essentially total silence. In a neural network, "signal variance" is simply this spread (or volume) of numbers flowing through the layers.

   If you initialize the network with random weights, signal variance quickly explodes or dies. If variance explodes, numbers become huge ($\pm 50$) and get pinned to the flat edges of squashing functions like Tanh ($\pm 1.0$). Because the slope at those edges is basically zero, gradients completely die during backpropagation and the network goes deaf. If variance goes way down on the other hand, the neurons all output values near zero, meaning the network doesn't have enough signal to tell the difference between a robot arm at $0^{\circ }$ versus $90^{\circ }$.

   Orthogonal initialization solves this with good ol' geometry. An orthogonal matrix is a special matrix where every row is perpendicular to every other row and has a length of exactly 1. In multi-dimensional space, multiplying a vector by an orthogonal matrix is a pure rotation and reflection ($\Vert Wx\Vert =\Vert x\Vert$). It spins the vector to point in a new direction, but it never stretches and never shrinks the vector's length. Because lengths are preserved, the spread (variance) of information leaving each layer is identical to what entered. And it gets better, the transpose of an orthogonal matrix ($W^{\top }$) is also orthogonal, meaning error gradients flowing backward during backpropagation are equally preserved.

   There is one catch though. Non-linear activation functions (like ReLU or Tanh) naturally chop off or squash part of the distribution, eating away roughly half of the signal's energy at each step. To cancel this out, we multiply the orthogonal matrix by a small booster gain of $\text{Gain}=\sqrt{2}\approx 1.414$. Because signal variance scales with the square of the multiplier ($(\sqrt{2}{)}^2=2$), this doubles the signal energy right before it enters the activation function. This way the network still can learn complex non-linear traits and we don't have to sacrifice all of our signal energy to do so.

   Finally, the very last layer (the output layer commanding the motors) is treated as a special exception. While hidden layers use full-strength weights to think, we deliberately scale the output layer weights to almost zero ($0.01$). If the output layer had full-strength random weights on timestep 1, the network would immediately command extreme joint angles, causing the robot to violently spasm and collide on startup. Scaling output weights to $0.01$ ensures the initial commanded actions are near zero, letting the robot start in a calm, neutral posture and explore smoothly from a safe baseline.

2. Global Gradient Clipping: In physics simulation, contacts are stiff and unforgiving. When a robot gripper accidentally slams into a tabletop at high speed or bangs into a hard joint limit, the sudden impact creates a massive spike in simulation penalties and value errors. Without protection, that single collision sends a tidal wave gradient through backpropagation, violently overwriting millions of weights across all layers in a single backward pass. This can cause the robot to forget skills it spent hours learning. Global gradient clipping acts as a voice of reason for the network to keep it from panicking. It treats the gradients across all network layers as a single high-dimensional vector. If its total Euclidean length exceeds a threshold (typically max_grad_norm = 0.5), the optimizer scales the entire vector down proportionally. The direction of the correction is completely preserved, but the shock value of it is reduced. The robot still learns that colliding was a mistake, but it doesn't overreact to the mistake.
3. Advantage Normalization: As we established in the policy gradient update, the size of our weight adjustment scales directly with the advantage score ($\text{Weight Update}\propto {\hat{A}}_t\times (a_t-\mu )$). But raw advantage scores vary wildly across a training run. In early training, when the robot fails constantly, returns are tiny or negative (e.g. scores between $-50$ and $-40$); late in training, when the robot masters the task, returns can soar into the hundreds.

   Advantage normalization fixes this by standardizing the advantage scores in each mini-batch so their average is zero and their spread is one ($(\hat{A}-\mu )/(\sigma +10^{-8})$).

   This forces the optimizer to evaluate actions relative to the robot's current skill level. In early training, when almost every rollout is a failure (say, scores between $-50$ and $-40$), the attempts that were slightly closer to the goal ($-40$) get shifted into positive territory, giving the robot breadcrumbs to follow. In late training, when the robot succeeds constantly, only truly exceptional moves get rewarded. At the same time, scaling the spread to 1.0 guarantees that our gradient updates stay a predictable, healthy size from the first minute of training to the final hour.

Monitoring Training, Beyond Looking at Reward

When an RL training run isn't working like you want it to you have to fight the instinct to interrogate the reward curve like its withholding information you need. Staring at the reward curve to debug training is like staring at the scoreboard after a basketball game to figure out why your team lost. To know how to debug the failure you gotta know how all the pieces tie together. Here is a little refresher, bringing all the previous sections together now.

• The Critic tries to look ahead and guess what kind of score the robot will end up with from where it's sitting right now ($V(s_t)$).
• GAE subtracts that guess from what actually happened to create up the advantage scores (${\hat{A}}_t=R_t-V(s_t)$), deciding whether each action was good or a mistake.
• The PPO clipping leash checks those scores and makes sure that any updates stay within reason by squashing any extreme values. ($1\pm ϵ$).
• And finally, gradient descent updates the Actor weights so we can try the simulation again, just hopefully better than last time.

Because every piece feeds directly into the next, errors carry forward and destabilize the whole thing. If the Critic gets confused and starts throwing out random guesses, the advantage scores become gibberish. The Actor tries to learn from that nonsense, picks up bad habits, and 50 iterations later your reward curve drops like the 2008 stock market. And if the leash is pulled too tight the Actor takes timid baby steps and the reward never budges at all.

Checking The Dashboard

The metrics below are a great thing to be checking during a problematic training run.

Where to Look When a Run Goes Bad

When the training run goes wrong, work down this checklist in order.

1. Check the Critic first (Explained Variance). If explained variance is near zero or negative, your Critic's value predictions are failing. Before touching anything else, double-check that your observation vector isn't missing vital physics (did you forget angular velocities?) and that contact penalties aren't spiking into outer space.
2. Read KL divergence and clip fraction as a pair. If KL is spiking above 0.05 while clip fraction blows past 40%, the updates are way too violent. Dial back the learning rate or drop your PPO epochs per batch. If KL is basically zero and clip fraction never leaves the floor, your policy isn't learning at all. Bump the learning rate or make sure you turned on advantage normalization.
3. Look for an entropy cliff drop. If entropy craters in the first ten iterations while task reward is flat on the floor, your robot found the freeze trap. Bump up the entropy bonus $c_2$ or ease up on harsh movement penalties so the arm is willing to explore again.
4. Check for a setup bug. Inspect your environment code for episode termination flags. If timeouts and crashes share the same flag, GAE assumes running out of time is a catastrophic failure, zeroing out future value and giving your robot late-episode panic.
5. Only touch the reward function as a last resort. Rewriting reward functions is tempting, but it can be the biggest time sink and can create a brand new set of weird behaviors. In practice, most early rounds of RL training that fail are from broken telemetry, missing state inputs, or unstable update steps, not your reward math. You need to setup the params so that the training even has a chance to succeed. It can train to your poor rewards at first that is fine, just get it training then worry about tweaking the rewards.

Once you absorb all of the information in this article and start to make sense of the metrics available to you, RL stops feeling like an exercise in lighting incense and praying to the gradient gods. I hope this article at least helped to build some foundational intuition within you.

After learning how RL works you will have a much better time training your robot to do something cool like make your bed for you while you watch the training dashboard and identify if the policy is updating nominally. Of course, the confusion can quickly come back though as you read the latest research paper about how some lab of highly caffeinated Phd students used a new method of machine learning to solve the problem way better than before... but that is part of the game! This is a discipline where you need to always be learning.

The Robot RL Engineering Series

If you want to learn more, or see the things you just learned put into practice, check these out: