Real-Time Nonlinear MPC in the Browser

The Pre-Computed Trap

Following a pre-computed trajectory works great until a big disturbance knocks the system off course. A tracking controller like TVLQR: Time-Varying LQR. The same optimal-feedback idea as LQR, except the gains change at every instant along a planned trajectory, because the machine's behavior near the plan changes as the plan moves. only knows how to pull the machine back toward its original stored path. If a hard bump pushes the pendulum far away, the controller tries to erase the whole error in a single instant, which saturates the motor and causes the swing to fail. What you really want is a controller that can abandon the old plan and calculate a fresh one on the fly.

A person knocked off course doesn't try to teleport back onto the old path. They look at where they are, look at where they need to be, and work out a new way to get there.

The Receding Horizon

Nonlinear Model Predictive Control (NMPC): Nonlinear Model Predictive Control. At every control tick, re-solve a short-horizon trajectory optimization from the current state and apply only the first step of the answer. 'Nonlinear' because it optimizes over the real curved physics rather than a linearized stand-in. solves this problem by re-planning constantly. Instead of tracking a pre-baked trajectory, the controller solves a brand new trajectory optimization at every control tick, starting from wherever the machine happens to be right now. It applies the first step of that new plan, throws the rest away, and repeats the whole process a few milliseconds later.

This uses a different philosophy than part one did. TVLQR solves the trajectory problem exactly once, offline, and spends the rest of its life correcting deviations from that single answer. NMPC is more akin to brute force. It solves the trajectory problem constantly, online, and never commits to an answer longer than one tick. The plan and the controller stop being two different things and merge into one.

People also reach for reinforcement learning to get a plan, similar to this, that adapts. NMPC does the same job in a completely different way. RL finds a controller by running a machine, real or simulated, millions of times and rewarding whatever works, then bakes the result into a neural network that maps state to force in one fast forward pass. NMPC never trains on anything and there is no neural net. It carries an explicit physics model and an explicit cost, and every tick it runs real optimization against both, from scratch, live.

The trade-off is almost the exact opposite of reinforcement learning. A trained policy is cheap to run and expensive to produce, often needing hours or days of simulated training before it ever touches a real machine. NMPC is the opposite. It is cheap to produce, as long as you can model your physics well mathematically, but it runs live on every tick, which makes it pretty CPU intensive. For it to perform well you need a very fast control loop. In this article I aimed for 200 Hz, but the faster you can run it, the better. MPC also gives you something RL cannot, which is an explainable solution. If the solver converges, you know exactly what problem it just solved and what assumptions went into it. A trained neural net can only tell you what worked during training, and everything else is a bit of a black box.

One detail worth mentioning is that MPC does not compute the entire four-second swing on every tick. It only plans for the next short window of time, called the horizon. It asks the same question Part I asked offline, just smaller and anchored to the present: what is the best sequence of forces over the next half-second? You take the very first force command in that answer, apply it to the motor, throw the rest of the plan away, and ask the exact same question again one tick later.

Throwing away almost all of the work you just did sounds wasteful, and it is. What you get in return is a plan that is never stale, and a machine that is always solving for the situation it's actually in, not the one someone predicted (or failed to predict, more critically).

Real-Time Iteration

Solving a trajectory optimization two hundred times a second is where the trouble lies. Part one's offline solve took about 2 seconds to compute. Doing that inside a 5-millisecond budget, in JavaScript no less, is no small ask.

But I've kinda been lying so far...You're never really solving it truly from scratch. The answer you computed one tick ago is very nearly the answer you want now. It's basically the same problem, shifted forward by 5 milliseconds. So, what can you do? Warm starts. Shift the previous solution left by one slot, put something plausible in the new empty slot at the end, and hand that to the solver as its starting guess. Start close enough to the answer and one iteration is enough to stay close enough. That's Real-Time Iteration: Real-Time Iteration. Rather than solving the optimization to convergence every tick, run only one iteration (or a few) starting from the previous tick's answer shifted forward, on the reasoning that a well warm-started problem barely needs to move. .

For the part one controller that was tracking the offline computed trajectory, that works exactly as advertised. One iteration, every tick, forever. The controller in this write up has no plan to warm-start from, and needed a very different amount of iteration budget to even discover what it should be doing. That turned out to be the harder half of this whole project.

Here's that controller live, tracking part one's stored plan right alongside TVLQR. Wait for the swing-up to get going and hit KICK, same disturbance, same instant, both controllers. TVLQR tries to erase the whole error immediately and saturates. The MPC plans a longer way back and never needs full force, it doesn't always catch either, but it wins more often than the pre-computed tracker does. The horizon and rate dials drive the MPC directly: lower the Hz and watch the zero-order hold: Holding the control output constant between updates, which is what any digital controller does. The slower the loop, the longer the machine is being driven by a decision made about a situation that has already changed. do its work. Somewhere around 30Hz the correction is answering a question the pendulum already stopped asking, and it comes down.

Swinging Up From Scratch

An interesting question is whether the controller needs part one's offline computed plan at all.

Let's take the reference away completely so we have nothing precomputed. All the controller is given is a description of what I want. The machine's total energy: Kinetic plus potential energy. For this machine, hanging at rest and standing upright at rest are two different energies, and the gap between them is exactly what the motor has to supply to get from one to the other. should equal the energy of standing upright.

That target may seem strange to aim at at first. It says nothing at all about where the pendulum is. Hanging at rest is one number while a standing equilibrium is another. The difference between them is the work the motor has to do. The optimizer is not specifically told it needs to swing. It works out on its own that pumping back and forth is the only thing available that raises energy, because it's searching over sequences of forces and that's the sequence that scores best.

This brings up what I call the short-sightedness trap. A controller with a plan can afford to be short-sighted, because the plan already did the long-range thinking. One without a plan cannot. With a short time horizon it simply commits to whatever move looks best inside its own tiny window. Experiment with the figure below: try giving it a short horizon and you'll see it just gets energy fast without knowing how to use it. Give it a full second of horizon and the identical math finds the real pump, back, forward, catch.

Getting It to Move at All

When I first tested the from-scratch controller, it refused to move at all. The optimizer was not throwing errors, but the cart just sat there hanging for the entire run. The problem came down to a subtle mathematical trap in how the angle was measured at the hanging starting position, where a small jump in angle caused a massive fake spike in the optimization curvature and froze the solver.

Frozen at the Start

The terminal cost has a catch bowl: A mathematical penalty zone shaped like a concave bowl (typically parabolic, created by a squared error term like θ²) in it, the term meant to lock the pendulum down once it's near the top, and I'd written it by wrapping: Ensures the target angle (the upright top position) maps directly to 0 radians, removing rotational discontinuity. the joint angle into (−π, π] and squaring the result. Perfectly ordinary. Except the simulation always starts hanging at exactly θ = π, which is precisely where the wrap folds over.

Nudge the angle up by ε and it wraps around to −π. Nudge it down by ε and it stays at π − ε. The solver builds its terminal cost's derivatives by finite differencing, so it takes both of those nudges and differences them straight across the seam. What comes back isn't a big gradient, which would've been easier to spot. It's an teeny tiny one, with a curvature of about −6.3×10⁷ where the true value is near −100. That number went straight into the value function on the first backward pass of every single episode and as it turns out, is quite important. This negative curvature value poisons the backward pass's own safety check. Because its negative it can't be certified as an improvement, as is expected in a positive-definite environment, and the pass aborts before it produces anything. The normal fix for that is regularization, add a damping term and try again, but the bad curvature is so far outside anything the damping schedule expects that it takes many doublings just to force the math through. By the time it finally does, that same damping dominates all gains so the effective result is nothing moves.

The fix is to use a different wrapping function. 1 − cos θ has no seam anywhere, and near θ = 0 it curves like θ²/2, the same bowl I wanted, without the discontinuity right on the critical area. Lesson learned. Picking a wrapping function with a break in it was the wrong call for something that touches every angle, since sooner or later the system lands exactly on the break, and here "sooner" was literally frame one.

Getting It Fast Enough

With the solver finally converging, the next problem was that it was slow. A from-scratch swing-up needs far more iterations than Real-Time Iteration normally gets away with, and at twenty iterations a tick the solve was blowing well past the 5-millisecond budget. Profiling made the cause obvious fast. Almost none of the time was arithmetic. It was garbage.

Every expensive piece of the backward pass, the Jacobian: The matrix of partial derivatives describing how each output of a function responds to a small change in each input. Here, how next tick's state responds to a nudge in this tick's state or force. iLQR needs it at every step to build its local model. , the matrices it builds, even a single physics step, was allocating fresh arrays and objects on every call, thousands of times a solve. JavaScript's garbage collector handles that fine at normal speeds. It does not handle it fine at 200 ticks a second on a 5-millisecond clock. The fix was the same one, over and over: stop littering on the hot path. The Jacobian went from finite differencing fourteen physics evaluations a step to an analytic formula. The backward pass's own matrices went from fresh 6×6 arrays every timestep to flat, preallocated Float64Array scratch buffers, reused every tick instead of rebuilt. The physics step itself got the same treatment.

Though I did consider throwing WASM at this problem, the truth is that I didn't needed a faster language, I just needed more optimized code. The physics step that used to hand back a fresh object every call now only costs 58 nanoseconds, and the whole solve fits comfortably inside the time budget. This control algorithm is very dependent on writing good fundamentally sound code.

Getting It to Actually Stay Up

With the solver now fast and correct, the pendulum could finally be caught. However, it still didn't know how to behave once it was up there.

The energy target on its own turned out to be a bad objective. It couldn't tell a pendulum spinning fast at some arbitrary angle from one standing calmly upright, both can carry the same total energy, so nothing in the cost encouraged a controlled catch. It also had nothing to control the cart, which wandered off the end of the rail while the links kept pumping. Tightening the effort and cart-centering weights, adding a penalty on the joint velocities, and raising the weight on catching near upright got it to a real, repeatable swing-up, though none of it moved in a straight line: push any one weight past its sweet spot and something else breaks.

Even tuned, it was still failing 25% of the time with a runaway cart. Counterintuitively, A harder rail limit made that worse. What fixed it was a longer horizon. At 0.8 seconds the optimizer can't see far enough ahead to know it'll need to come back, so it yeets itself. With a 1 second time horizon the identical system plans a pump that never leaves the rail on its own, no extra constraint required. Catch rate rose back to 100%.

One problem remained after all those fixes — the cart kept drifting after a catch, slowly and forever, even with the pendulum balanced perfectly. Energy is the right objective for swinging up but its a pretty useless one for balancing. The fix was to stop trying to use the swing-up cost to also cover balancing. Part one's Riccati solve already produces exactly the number needed here, $P$, the total future cost of being in any given state near upright.

Blending that in as a terminal cost once we near the linearization region gives the planner a reason to care what happens after it ends. With this the cart settles and the drift disappears.

The Result

Here's the best version of it, running on its own: the from-scratch controller, no precomputed anything, replanning its whole future from wherever it actually is. The faint links fanning out ahead of it aren't decoration, they're the solver's own prediction, the literal horizon it just solved, redrawn from scratch every tick. Kick it and watch that prediction reroute in real time, live, in your browser.

What I find funny about this write up is how little of the actual work for me in making this was control theory. The controller that finally behaves is about fifteen lines of cost function and it is basically straight out of a textbook. What took the time was a silly mistakes I kept making like having a wrapping function with a discontinuity sitting on the start state, lazy programming practices, and tweaking the terminal cost. As with most things though, the ordinary business of making something actually run as intended is most of the job.

One cool thing about debugging this controller though — all of these issues are findable because the controller carries an explicit model I could set a breakpoint in. A trained policy doesn't explain itself when it fails, so debugging it is a lot more like guessing. With NMPC, I could see exactly what the solver thought the machine was doing and why it was making the decisions it was, and that made it possible to find the problems and fix them.