In my previous write-ups about my Mamba 3 + Liquid Neural Net robot policy, I was showcasing the progress of training it on a simple pick-and-place color sorting task. It was useful to at least show and validate that the policy can work and is capable of doing some simple tasks well.
I got to thinking about it though, and the pick-and-place task I was doing was kinda dumb if I am being honest. It didn't highlight any of the unique strengths of the Mamba cortex nor the LNN brainstem. So, I got to thinking about how I could show off the strengths of those components a bit more.
My first thought was to replace the humans on a Bering Sea crab fishing boat with a sophisticated robot arm to get humans out of that miserable task, but further thinking led me to believe it would be miserable for the robots as well. We need some drastic redesigns of the whole crab fishing process to make that one make more sense. Skip.
I then remembered back to my college days where I designed a super simple computer vision based strawberry picking robot. Much to my surprise, when I looked up robots that do similar agricultural picking tasks today, there is still a shocking lack of progress in that area despite very strong economic and academic interest. Thus, my own interest piqued. I decided to take on another red-fruited challenge and try to use my Mamba 3 + LNN setup to create a smart apple picking robot that is capable of beating the leading apple picking research paper arms as of October 2026.
Why the Orchard?
Why do I think the apple picking problem showcases the skills of this particular trained policy, you might be wondering? Let's examine the main challenges with picking apples, both for humans and for machines, and we can talk about why I think I have a valid use-case here.
To understand why, we have to look at the real-world trade-offs between human pickers and traditional automation across the core challenges of an apple harvest:
| Picking Challenge | Human Harvester | Traditional Automation |
|---|---|---|
| Dexterity & Bruising | Incredible dexterity and sensitivity. Squishy human fingers twist stems cleanly without bruising apples. | Rigid and clumsy hard materials prone to running into things, tearing spurs, or squeezing fruit too hard. |
| Harvest Speed | Blistering pace: ~2 seconds per apple (~1,800 apples/hour). | Slow and fragmented: 7–10+ s per pick in ideal circumstances. |
| Canopy Occlusions | Specialized ability after millions of years of evolution. Effortless and efficient. | Frame-by-frame computer vision easily loses targets when branches sway or shadows shift and has poor spatial awareness and object permanence. |
| Terrain & Wind | Dynamic balance and maneuverability. Able to deal with rutted ground, and wind gusts with no problems. | Wobbly robot platforms, uneven terrain, and gusty winds disrupt slow-moving arm trajectories and require course corrections. |
| Operating Window & Cost | Daylight-limited, physical fatigue, and rising seasonal labor shortages. | Fatigue-free: runs 20+ hours a day, including at night under floodlights. |
The table lays out the fundamental problem: machines have the endurance and cost advantage, but collapse on the real-time dexterity, memory, and speed that humans take for granted. My setup is designed to tackle those exact machine weaknesses:
- Mamba 3 Memory: It has memory with Mamba 3 so it can remember where an apple is even if it's now occluded by another apple, a branch, or a leaf.
- Rotary Positional Embeddings (RoPE): It can deal with undulations from the wind or from the cart passing over uneven terrain, like when wheels roll over fallen apples in the orchard rows.
- Liquid Neural Network (LNN/CfC): It uses the LNN to make micro-adjustments as required for dexterous movements at the critical moment of grasping an apple.
- Harvest Speed: Lastly, a highly specialized RL model like this should be able to lead the field of research right now with the speed it's able to harvest (at least in the simulation).
The new challenge seemed exciting and I immediately got to pivoting.
What Apple Orchards Actually Look Like
When you think of an apple tree, you probably think of something like this:
Sprawling open canopy with crooked, irregular branches
Dense dome of deep foliage and deeply hidden fruit
In reality, that is not how most apple trees look for high-volume commercial apple orchards. At least not for the kinds of apples you find in the produce section of your grocery store. Juice farms might still use trees like this since they can get away with using a simple mechanical shaker to harvest them as bruises don't matter for that use case. But for apples you eat whole, modern apple farms use high-density plantings that look like these images below, creating near-planar 2D fruiting walls. The fact that apple orchards have taken this shape to help their human harvesters gives our robot harvester a massive advantage as well.
Small trees with short spurs & drip lines
Trellis with high-tensile wire supports
Another down the isle view. Human for scale.
Along with the flat trellised rows, modern orchards frequently use protective netting overhead to protect their crops from hail and severe weather, weed fabric along the ground, and dense fruit clusters that grow directly off the central leader trunk and short spurs.
Dense clusters touching wood & neighboring apples
Another human for scale here
Overhead netting & tarp ground cover
However, despite the modified shape of the trees and the easier-to-access fruit, industrial apple picking is still mostly done by hand as mechanical means simply aren't good enough yet. That yet is a key detail here because I think we are close, really close, to getting this to be a feasible robot task that makes economic sense for the farmers.
Building the Simulation Twin in MuJoCo
The first step for me was to create a reasonable twin of the orchard in my simulation environment, MuJoCo.
Rigid wood colliders & active fruit cluster physics
Platform camera (with scout bounding labels) + RealSense D405 wrist view
I chose to make my sim run at night with LED lights to minimize the environmental factors that trip up many computer vision systems. It's one less variable to worry about which often times saves a factor more headaches.
I gave it a depth camera to help it be more precise rather than a plain RGB camera. Optional but helpful nonetheless.
I offboarded identification to a separate, more purpose-built trained policy that runs independently on the platform and communicates that data to the Mamba arm.
Finally, I chose to have a single bin for placing picked apples, with the identification model data being used to control a flap in the bin for sorting the apple into the correct location. All of these are choices ones I would make if I turned this into a real product and they don't detract from showcasing the special trained networks I have. These choices simply give my robot the best chance to succeed.
The Training Strategy
With this setup, I got to work training. I chose to follow a similar pattern as before where I use a scripted expert that has access to privileged information to perform the apple picking task.
I use a form of Teacher-Student Imitation Learning (DAgger) followed by Residual Reinforcement Learning (PPO).
- Why a Privileged Scripted Expert First? Learning contact-rich picking directly from raw camera pixels with trial-and-error reinforcement learning from scratch requires millions of simulation steps and frequently gets trapped in bad local minima. The scripted expert has access to "god-view" privileged state: ground-truth 3D positions of every apple, spur, and branch capsule. It runs deterministic state logic (stand-off, insert, close, lift, carry, release, back off). This gives the student policy reliable training wheels to learn the basic mechanics of reaching and grasping.
- Why DAgger (Dataset Aggregation) Instead of Plain Behavior Cloning? Standard behavior cloning only trains on trajectories driven by the expert. The moment the student policy makes a slight mistake during rollout, it drifts into unvisited states and fails. DAgger runs the student policy in the sim to collect rollouts, but queries the expert at every step: "Given the exact state the student is currently in, what should the robot do?" This trains the student how to recover from its own errors.
- Dual-Rate Training Architecture: During training, the 10 Hz Mamba-3 Cortex is trained using a parallel associative scan across entire episode sequences for fast, long-horizon temporal credit assignment. Meanwhile, the 50 Hz Liquid CfC brainstem is unrolled with truncated backpropagation through time to learn reactive, continuous-time joint velocity control.
- Why Follow Up with PPO? Because if this is ever going to work in the real world, the robot has to actually learn how to do the task itself. The scripted expert is cheating. It uses privileged "god-view" simulation state with exact 3D millimeter coordinates for every apple and branch. PPO takes off the training wheels. The actor must execute the entire grasp and recover when things don't match the script's idealized world.
What Didn't Work (So Far)
I am writing this article before I have "succeeded" with this task. I could've waited, but I didn't want to ghost on posting for so long, so I thought it would be better to post this progress update.
So, I don't know what works best just yet, but I do know PLENTY of things that DON'T work:
The robot would line up on an apple and just hover, never taking the critical step to grasp. Sensor noise was diluting the signal “close” value below the motor threshold. Switching to a dedicated commit movement forced the gripper to actually close.
When dodging a branch, tilting 20 degrees left or 20 degrees right were both clear paths. The network took the average, diving head-on into the trunk. Teaching it to pick one side smoothly instead of averaging competing modes eliminated that mode of crashes.
Again with this setup I had to go through a new round of reward tuning pain. I had too many penalties relative to the behavior the robot was capable of. Progress rewards were being clawed back whenever a reach failed, so the math accidentally proved that doing nothing was the safest way to protect its score. Increasing strictness as behavior improves and relaxing strictness while the policy is young helped to get un-stuck.
I chose a gripper that would grasp apples better for this setup. However, it had flat rubber pads which squeezed round apples out of the fingers like wet melon seeds. They were also too bulky to slip into tight cluster and would frequently knock apples to the ground. Modifying them to be slimmer fingers lined with curved silicone retaining lips stopped the mid-air dropouts.
Apples kept mysteriously vanishing right after clean grasps. Watching replays revealed the arm took a straight-line path to the bin and slammed its own elbow into the apple in its hand punching the apple loose. Path tweaks, platform layout tweaks, as well as collision avoidance all helped to eliminate this failure mode.
My first pass at simulated apple stems detached at the slightest nudge, dropping apples before the fingers could close. There somehow is real research out there that tells us exactly how much force a real apple needs to be taken off the stem. Which, as it turns out, is 86-degree bend-and-twist to shear the joint or roughly 22 N on a straight pull. I updated that and any other values I could find to be more realistic.
Where I'm At Today
At the time of writing this my policy is doing decently well. I had it at one point able to harvest 90% of apples it was presented with an average time to harvest per apple of under 7 seconds which would make it the "best on the market" today (if it was on the market).
But that wasn't a very good benchmark to be honest. Reason is that was just a single simulated orchard row, no domain randomization, and no occlusions, no environmental factors, and no real-world noise. So, I have been working on making the simulation more robust in that sense and of course my policy took a hit in performance because of that. I've got it back up to 65% success rate but I'm struggling with speed and smoothness now. Its a bit of a slow clumsy mess at the moment.
So, while I continue to chip away at this task and make incremental progress, I will keep documenting what I am doing and learning so I can share here. Hopefully I will be back in less time than the last gap in articles was, but if I'm not, just know I'm still tinkering away at it!
See you in the next one.