In Part I: Building a Robot Brain, I trained a 7-DOF Franka Panda arm using Mamba State Space Models and Liquid Neural Networks, reaching a 98.7% success rate at picking up colored cubes and sorting them onto target pads.
That 98.7% headline number looked great, but it's hiding a secret. That number was only achievable with ideal conditions and in very specific scenarios. As soon as you change the table, change the lighting, basically change anything at all, it fails.
This failure is because reinforcement learning algorithms do not learn general robotics principles on their own. They optimize ruthlessly against the exact world you put in front of them. In my initial setup, that world was a mostly static and highly idealized simulation.
When a policy is highly fragile like this, the root causes usually fall into two categories:
- Blatant Exclusions (Simulation Cheat Codes): Assuming motors respond with zero latency, have infinite acceleration, or the environment will always have perfect lighting. The neural network will optimize with these unrealistic behaviors since it has no way to know any better. It can only optimize based on what it is given.
- Accidental Exclusions (Unmodeled Blind Spots): Forgetting to model rubber fingertip friction, ignoring joint damping, or not accounting for camera sensor noise. Missing these can cause the policy to learn unnatural compensatory tricks or have unrealistic expectations that break when deployed in reality.
Closing this means tackling both sides of the problem. I had to model the physical constraints I knew, and randomize the visual variations I could not predict:
Modeling the Franka's real rubber fingertip pads so the fingers have authentic compliance, damping, and torsional friction.
Guiding arm joints toward a natural ready pose with joint-space potentials to eliminate unnatural wrist contortions and kinematic exploits.
Preventing visual overfitting by training across diverse 3D materials, lighting angles, and object poses.
Adding realistic actuator lag and gripper slew limits so the controller cannot rely on instantaneous acceleration or infinite torque.
Fixing the Contact Physics
In my Part I setup, the gripper fingertips used 's default contact settings. Under default settings, bodies interact as rigid surfaces with basic sliding friction and zero resistance to twisting.
This was an accidental exclusion that distorted the robot's learning. In the real world, a Franka Panda has soft silicone pads on its fingers. When the fingers close on a cube, the rubber compresses and conforms to the surface, generating additional friction that stops the object from twisting and keeps it from sliding out of the gripper.
Because the default simulation physics gave the gripper zero torsional friction, any off-center grasp made picking up the cube similar to trying to pick up a marble with chopsticks: the slightest misalignment and the thing twists and falls out of the grip. This made the robot learn some less than ideal gripping behavior and the performance of the held grips was overall quite poor. The problem was not the neural network on this one, this one was purely a modeling problem.
To restore realistic contact physics, I updated four key contact properties on the fingertip geoms:
| Contact Property | Default Simulation | Silicone Rubber Pad | What It Changes in Practice |
|---|---|---|---|
| Contact Friction Mode | Sliding only 3 dimensions | Sliding + Torsional 4 dimensions | Enables torsional friction around the contact normal, stopping off-center cubes from spinning out of the fingers. |
| Friction Coefficients | Slide: 1.0 Twist: 0.005 Roll: 0.0001 | Slide: 1.8 (+80%) Twist: 0.08 (16×) Roll: 0.001 (10×) | Increases sliding grip and vastly boosts torsional and rolling friction to match physical silicone rubber. |
| Contact Softness & Damping | Rigid / slow response 20 ms, critical damping (1.0) | Compliant cushion 8 ms, critical damping (1.0) | Creates a softer, critically damped contact spring that absorbs finger closure impacts without bouncing. |
| Material Override Priority | Averaged (0) Finger & cube properties are mixed | Override (1) Fingertip rubber dominates contact | Forces the fingertip's rubber properties to override the cube's defaults instead of averaging them together. |
Beyond static friction, physical rubber pads exhibit progressive stiffening. They deform easily on initial touch to conform around the object, then rapidly stiffen as they compress under load. In MuJoCo, this is modeled with a mathematical function called solver impedance (abbreviated in the simulator as solimp).
Rather than treating contact as an all-or-nothing rigid collision, the solimp function dynamically calculates contact stiffness as a function of penetration depth: solimp="dmin dmax width midpoint power". Tuning this to "0.9 0.99 0.003 0.5 2" creates a 3 mm compression cushion matching the physical thickness of the Franka's silicone pads. The function starts compliant (0.9) to absorb initial closure impacts without bouncing, then smoothly ramps stiffness up to near-rigid (0.99) under full squeeze so the cube stays locked in place without sinking through the finger mesh.
Fixing this exclusion gave our simulation robot a big upgrade. With proper rubber friction, the cube stopped slipping, allowing the policy to develop smooth, compliant grasping habits that translate to real hardware.
Scaling Training with Vectorized GPU Physics (MJX)
Training a RL model is a pretty computationally heavy task, even for a lightweight model like the one I've built. Hardening that policy comes with additional computational cost. When you add compliant contacts, motor delays, and wider physical variations, the policy needs tens of millions of environment steps to explore and converge.
On a laptop CPU, running vision-based reinforcement learning was painfully slow. Stepping the physics engine, generating camera frames, and transferring tensors back and forth between CPU memory and GPU memory meant that my training runs were taking around 4-8 hours, just to test a tiny incremental change. The worst part would be that I made a mistake somewhere along the way and that whole training run would just become throwaway work. I quickly realized that it would take too long to learn and adapt trying to do it the free way on my laptop.
To make large-scale hardening practical, I brought out my wallet and moved simulation entirely onto the GPU using .
Rather than running one simulation at a time on the CPU, MJX compiles the entire physics pipeline into fused GPU kernels with JAX. Paired with a hardware-accelerated GPU raytracer, MJX steps physics and renders cameras for 256 parallel environments simultaneously directly in GPU VRAM on an RTX 3090 whereas on my laptop I was only able to run 4 parallel envs at a time.
Because the simulation states, sensor observations, neural network weights, and PPO gradients all live on the same GPU, there is zero memory transfer bottleneck. A training run that once took 8 hours on a laptop to gather 9 million steps completes 15 million steps in under 45 minutes on a single cloud GPU. That's a 10x increase in training throughput for under a dollar per hour of compute.
Visual Robustness & Domain Randomization
Now that the physics and training throughput are fixed, the next point of failure is vision.
In Part I, my policy was trained on a single, uniform tabletop surface. I had originally messed up the texture of the table and the wood grain look was warping horribly around the edges of the table but I didn't care too much at first, I just wanted something that worked. The vision encoder learned to spot the colored cube in the scene with this broken texture and its movements relied on it.
Once it was working pretty well I wanted to record a better demo with a table that actually, ya know, looked like a table. So, I swapped the plain rectangular prism table with a wonky texture for a fixed woodgrain texture and gave the table legs. I checked the scene locally without running the sim just to make sure it all looked good, and it did. I was excited to record my new fancy demo.
This was my face when I watched the new recorded demo video because my robot which was previously working well dropped from 98.7% success rate down to 5.7%. The recording was terrible!
The baseline policy successfully identifying the cube, descending, and placing it when operating on the single table surface it saw during training.
This failure illustrates a fundamental rule of deep learning in robotics: neural networks will always exploit anything you let them exploit.
The vision encoder had not generically learned the spatial concept of "a cube resting on a surface." Instead, it had learned to exploit its visual data to identify the specific pixel arrangement of the scene it was trained on and its weights were set based off of that. It didn't need to work any harder to generalize, because I hadn't given it a reason to. When the new table altered the look of the pixels around the cube, the spatial feature maps became noise, and the policy freaked out.
Native 3D Domain Randomization
The standard remedy for visual brittleness is . If the policy sees hundreds of visual variations during training, it learns to ignore irrelevant background features and instead focus on what actually matters: physical object edges, depth cues, and relative spatial geometry.
Doing domain randomization correctly requires some forethought, though. You need to add randomization in a way that makes sense for your scene. Think about what the robot might actually encounter in its real world operating environment. Will it see different lighting conditions around it, changing weather conditions, or surface texture changes? You need to randomize those so it becomes robust against scenarios not explicitly covered in its training data. Also, gradually introduce randomization, as randomizing everything indiscriminately just drowns out the learning signal.
The key for me was to randomize a few things natively in the 3D scene. I assigned the table in each of the 256 parallel training environments one of eight real-world surface materials—including wood, concrete, linoleum, tile, rubber, and granite. At the same time, I modified lighting angles. Because the textures and lights exist natively in the 3D world, the vision encoder is forced to learn true geometric edges and spatial depth across physically plausible surfaces.
After training across this material distribution for 14.7 million steps, the vision encoder becomes indifferent to table appearance. It achieves a 99.9% success rate across all eight surface textures, including the woodgrain table that previously caused it to collapse to 5.7%.
The hardened policy operating across shifting table materials. By training over a diverse set of 3D materials, the vision encoder learns to track the cube independently of background appearance.
| Evaluation Environment | Home Success | Approach / Lift / Place |
|---|---|---|
| Woodgrain Table (Nominal) | 99.8% | 100% / 100% / 100% |
| Dark Table (Baseline Surface) | 100.0% | 100% / 100% / 100% |
| Random Multi-Texture (Active DR) | 99.9% | 100% / 100% / 100% |
Realistic Movements and Some Final Reward Tuning
The policy is now more robust against environment changes, yee haw! However, it can still fail on physical hardware if it commands unrealistic motor movements so our work is not done yet.
In the simulation, the neural network can output drastic target changes from one step to the next. A poorly simulated motor will obediently attempt to snap to that position in a single millisecond tick. In the real world though, electric motors have rotor inertia, winding inductance, and signal latency. Demanding instantaneous torque steps causes current spikes, gearbox chatter, and controller instability. It could even break your robot, which would make all stakeholders very sad.
Modeling Motor Delays and Slew Limits
To prevent the network from relying on impossible acceleration, I added two dynamic filters. First, a discrete first-order low-pass filter for the setpoints commanded by the policy. This added roughly a 20 ms physical response lag at 50 Hz so the policy has to plan its trajectories knowing the arm takes time to catch up. Second, I added slew-rate limiting to the actuators. Now the gripper fingers move at a realistic physical velocity limit, taking roughly 0.42 seconds to complete a full stroke. Same concept applies for the joint actuators.
Reward Tweaking Continues Until Morale Improves
When I inspected eval footage of the policy after sorting the cube, I noticed a few strange behaviors. Two main issues you probably saw in the Part I videos were the little flick or toss the robot did to get a cube in the landing zone, and the second was that its wrist twisted at weird angles after completing the task.
This was caused by reward under-specification. The original retreat reward only measured the Cartesian distance from the gripper tip to the center of the workspace (a 10 cm sphere in space). The reward did not care how the arm's 7 joints were oriented, as long as the fingers ended up inside that sphere. Because twisting the wrist was slightly faster than retracting the whole arm, the optimizer happily exploited that shortcut.
To fix the odd wrist behavior, I replaced the simple Cartesian distance reward function I had with a combined Cartesian and joint-space potential one:
In this formulation, measures the physical distance from the gripper to the home position, while calculates the root-mean-square error between the robot's current 7 joint angles () and their canonical ready pose (). This creates a smooth gradient pulling the entire arm back into a clean, natural posture.
The earlier changes to the fingertip rubber modeling helped the toss problem along with a fix for the unconstrained release behavior. To teach it to set the cube down gently rather than flinging it, I added a precision placement bonus (awarding up to +8.0 scaled by how close the cube lands to dead center) and a gentle touchdown bonus (up to +6.0, only paid out if vertical touchdown speed is under 5 cm/s). Paired with penalties for high descent speed and an instant failure if the cube bounces or slides away after release, the policy learned that a deliberate, cushioned touchdown was vastly more profitable than a lucky throw.
All of these changes come together to make the overall robot behavior so much smoother and better! In fact, it looked so good that I had to do the robot justice and snazz the recordings of it up. I recorded the trajectories of the robot in MuJoCo and imported them into Blender. That is why these final videos look so much nicer. The movement of the robot is still 100% coming from the trained policy, I just made it more visually appealing to look at.
The policy operating under realistic actuator lag, lighting randomization, and joint-space posture shaping. Motions are smooth and continuous, and the arm settles into a clean ready configuration after placement.
Key Takeaways and Sim-to-Real Principles
This project has been a painful reminder that reinforcement learning models are exceptional at finding shortcuts in simulation and reward policies. If you leave contacts frictionless, actuators instantaneous, backgrounds static, or rewards unconstrained, the policy will exploit those simplifications every time. It takes quite a bit of back and forth fiddling to get it right.
By systematically addressing each of these issues I faced, I transformed a fragile, half-decent demonstration into a much more robust and well-behaved policy.
| Domain Area | Default Simulation | Hardened Policy | Impact on Robustness |
|---|---|---|---|
| Contact Physics | Rigid geoms, sliding friction only | Compliant rubber pads, torsional friction | Prevents off-center grasp spin and contact bouncing |
| Simulation Speed | Single-instance CPU stepping | 256 parallel GPU environments (MJX) | Scales training throughput by over 10x |
| Vision Robustness | Single static table texture | 3D material and lighting randomization | Recovers success from 5.7% to 99.9% across surfaces |
| Actuator Realism | Instantaneous setpoint jumps | Low-pass motor lag and slew rate limits | Ensures commanded torques respect physical motor limits |
| Placement & Touchdown | Binary zone milestone, unconstrained release | Precision center & gentle touchdown bonuses | Stops ballistic flicking and ensures cushioned, centered placement |
| Arm Posture | Cartesian end-effector distance only | Joint-space ready pose potential | Eliminates unnatural wrist contortions at episode end |
Key Takeaways
- Model realistic constraints: Model the proper materials into the sim, ensure motor movement has non-idealized behavior, constrain it to your real hardware limits.
- Randomize details for robust policies: Expose the vision encoder to diverse materials, shadows, lighting, positions, etc. so it learns the key concepts not just specific setups.
- Inspect rollout behavior regularly: Good looking reward data during training can hide bizarre joint contortions and exploit loopholes that your metrics miss.
This was a massive step forward for this project. Now I can train on the cloud, I have policies that are becoming more robust, and I have a Blender pipeline setup to capture beautiful renders of the new and improved robot. So what is next? I could try to actually take it sim-to-real, but I don't have a spare $25,000-$40,000 USD lying around to buy this robot arm unfortunately. So, I will continue to make the things it does in the simulation more and more applicable to the real world! In the next writeup, I will cover a scene change and an increase in task difficulty. I will give it a moving conveyor belt with various objects being fed in continuously. It will need to pick the items off of a moving conveyor belt and stack them in 3D bins. I want to also train it to have a "turbo mode" where it can operate near max speeds for 10-15 minutes to deal with a simulated surge in factory throughput. This is a much harder task, so it will take some time to get this next one out. I will do my best to update you soon once I have some interesting findings. See you next time.
Mamba SSM + Liquid Neural Networks
Replacing heavy Transformer context windows with linear-time State Space Models for 50 Hz recurrent robot control.
How RL Training Works
An intuitive breakdown of policy gradients, actor-critic networks, PPO clipping, and diagnostic training curves.