Microduck, From Zero

0 · Start here

There are two repositories, and they split the work cleanly. Keep this split in your head and everything else has a place to go.

Facts to keep

• About 25 cm tall and 800 g. Fifteen Dynamixel XL330 servos and an IMU (an orientation sensor) share one serial bus.
• Policies control 14 joints: five per leg (hip yaw, hip roll, hip pitch, knee, ankle) and four for the neck and head. The fifteenth servo is the mouth, which the policy leaves out.
• Everything runs at 50 Hz, so the robot decides every 20 ms, in training and on the robot.
• Every policy reads the same 61 numbers and writes the same 14. That shared contract is what lets the robot swap from walking to standing up mid-fall.

Suggested schedule

1 · Foundations

Four ideas carry the whole project: reinforcement learning, how a servo moves a joint, simulation, and the gap between simulation and reality.

1.1 Reinforcement learning in one picture

Reinforcement learning (RL) trains a policy, a function from what the robot senses to what it does. Nobody tells it the right answer. It tries actions, gets a reward number that says how good things went, and slowly changes itself to collect more reward over time.

PPO (Proximal Policy Optimization) is the algorithm. In plain words: run the current policy in thousands of copies, see which actions turned out better than expected (the advantage), and nudge the policy toward them, but clip the nudge so one update can't change it too much. That clipping is why PPO is stable enough to be the default for legged robots.

# the heart of PPO, in pseudocode
ratio     = pi_new(action | obs) / pi_old(action | obs)
objective = min(ratio * advantage,
                clip(ratio, 1 - 0.2, 1 + 0.2) * advantage)   # don't move more than ~20%
maximise the average objective over a big batch

Why thousands of copies: RL needs a huge amount of experience. microduck_rl runs --env.scene.num-envs 4096 ducks in one GPU simulation, so an hour of training is years of practice. This "massively parallel" style is what made learned locomotion practical (Rudin et al. 2021, in Module 2).

1. L1, read now: OpenAI Spinning Up, "Introduction to RL", parts 1–3. spinningup.openai.com. About 45 minutes, and the clearest intro there is.
2. L2: Sutton & Barto, Reinforcement Learning: An Introduction, 2nd ed., chapters 1, 3 and 13. Free at incompleteideas.net/book.
3. L3: Schulman et al., "Proximal Policy Optimization Algorithms", 2017. arXiv 1707.06347. Schulman et al., "High-Dimensional Continuous Control Using Generalized Advantage Estimation", 2015. arXiv 1506.02438.

1.2 How a joint actually moves

The policy does not output torques. It outputs a target angle for each joint, and the servo's own controller pushes toward it. That controller is a PD controller:

torque = kp * (target - angle)  -  kd * velocity
         └─ spring toward target ┘   └─ damper ─┘

• kp (stiffness): high means it snaps to the target; low means it is soft and lets gravity pull it.
• kd (damping): stops it overshooting and wobbling.
• Tiny hobby servos like the XL330 are voltage controlled. They have friction that depends on load, and the battery voltage sags when they work hard. At this scale, the servo model is most of the difference between simulation and reality. That's why microduck_rl uses a detailed actuator model called BAM instead of an ideal PD.
• Backlash is play in the gears. The motor turns a degree before the joint moves. microduck_rl trains "Backlash" variants with ±1° of play per joint.

Kinematics is the geometry. Forward kinematics means joint angles in, positions of feet and head out. The kinematics crate in the robot repo does this, for example to know where the head's depth sensor is pointing.

1.3 Simulation

Training on a real 800 g robot would break it thousands of times. So training happens in MuJoCo, a physics engine designed for robots and contact. microduck_rl uses mjlab, which runs MuJoCo on the GPU (MuJoCo Warp) with the rsl_rl PPO trainer. The robot's shape comes from its CAD model in Onshape, exported to MuJoCo's XML format (MJCF) with onshape-to-robot.

A simulator is always wrong somewhere: friction, contact, motor response, mass distribution. A policy that exploits the simulator's mistakes falls over on the real robot. This is the sim2real gap.

1.4 Crossing the sim2real gap

Three tools, all used in microduck_rl:

1. Domain randomization. Don't train in one simulated world. Each of the 4096 copies gets slightly different battery voltage, voltage sag, command delay, friction and backlash. A policy that works across all of them treats the real robot as just one more variation.
2. Better models where it matters. Model the servo down to its voltage control law (BAM), and model backlash with an extra passive hinge per joint, read through the play just as the real encoder reads it.
3. Sensor realism. Add noise, encoder bias and IMU misalignment to the observations, so the policy never relies on perfect readings.

1.5 ONNX and deployment

ONNX is a file format for neural networks that any runtime can execute. Training happens in PyTorch on a big GPU; the robot runs the ONNX file on a small ARM board. One trap the repo calls out: observations are normalized during training, so the normalizer must be baked into the ONNX file. Run a raw checkpoint and the policy sees unscaled numbers and behaves randomly. That's why there is exactly one safe export path, scripts/export.py.

2 · Paper ladder

There's no single "microduck paper". The project stands on a short stack of papers. Read them in this order. For each, I've said what to take from it, so you can skim the rest.

Background, not required: Antoine Pirrone's community project Open Duck Mini is an open, small BDX-style duck that is worth comparing with microduck.

3 · The training repo, microduck_rl

3.1 Commands you'll actually type

# needs a CUDA GPU and uv. Smoke test first: catches most config errors for cents
uv run train Mjlab-Velocity-Flat-MicroDuck --env.scene.num-envs 64 --agent.max_iterations 5

# real training: about 1-2 h for a usable gait at 4096 envs
uv run train Mjlab-Velocity-Flat-MicroDuck --env.scene.num-envs 4096

# no GPU? the same command on Hugging Face Jobs
uv run train Mjlab-Velocity-Flat-MicroDuck --env.scene.num-envs 4096 --hf-jobs

uv run play Mjlab-Velocity-Flat-MicroDuck --wandb-run-path <entity/project/run_id>   # watch it
uv run scripts/export.py Mjlab-Velocity-Flat-MicroDuck --wandb-run-path <...>       # ONNX
uv run scripts/infer_policy.py --walking output.onnx    # drive it with the keyboard, CPU MuJoCo

3.2 The tasks (skills)

Each main task has a -Backlash- twin with ±1° of gear play per joint.

3.3 The contract that holds it together

Every policy, whatever its skill, reads 61 numbers: 48 of proprioception, then a 13-number command block of twist (3), head pose (4) and body pose (6), always in that order. A task that doesn't use a slot fills it with zeros rather than deleting it. Output is always 14. Because of this, the runtime can hand the robot from the walking policy to the stand-up policy at any instant. The publish tool refuses any ONNX that isn't [1,61] → [1,14].

3.4 Reward design: the hardest part, written down

The repo's AGENTS.md is a list of lessons that "were each learned the hard way". These are the ones that transfer to any RL project:

• RL optimizes the letter of the reward. Every gap gets exploited: a "ballistic whip" instead of a roll, a "head-tripod" instead of standing.
• Never pay for being in a bad state. The policy parks in the cheapest pose that qualifies and farms it. Pay for progress instead.
• No jackpots. If reaching a goal pays every step afterwards, arriving early with violence is the optimum. Track a slowly moving target instead.
• Introduce smoothness penalties after the skill exists. A tax on attempting, active while a hard skill is being learned, makes "do nothing" the winner.
• Train the zero command explicitly. Random sampling almost never produces exactly "stand still", which is the most common state on a real robot.
• Sign conventions bite. A negative weight on a penalty that is already negative turns it into a reward, and the policy farms the violation. Check: every penalty's episode reward in wandb must be ≤ 0.

Here is the real fix for the "parks in a bad pose" problem, from src/mjlab_microduck/tasks/mdp.py:

def upright_progress(env, asset_cfg=_DEFAULT_ASSET_CFG):
    """Potential-based upright shaping: Δcos(tilt) per step.
    Pays for PROGRESS toward upright, charges for progress toward fallen,
    and pays exactly ZERO for holding any pose, so no state can farm it."""
    asset = env.scene[asset_cfg.name]
    quat = asset.data.root_link_quat_w
    cos_tilt = torch.nan_to_num(1.0 - 2.0 * (quat[:, 1] ** 2 + quat[:, 2] ** 2), nan=1.0)
    if not hasattr(env, "_upright_potential_prev"):
        env._upright_potential_prev = cos_tilt.clone()
    fresh = env.episode_length_buf <= 1                    # no spurious delta after a reset
    env._upright_potential_prev[fresh] = cos_tilt[fresh]
    delta = cos_tilt - env._upright_potential_prev
    env._upright_potential_prev = cos_tilt.clone()
    return delta

Read it line by line. 1 − 2(qx² + qy²) is the cosine of the trunk's tilt from a quaternion: 1 upright, 0 lying flat. The reward is this step's value minus last step's. Rising pays, falling costs, holding still pays nothing.

3.5 Curricula and reading a run

• Training stages change reward weights and command ranges over time. If a metric drops exactly at a stage boundary, the pacing is wrong: stretch the stage, don't rush it.

• Reverse curriculum: start some episodes partway through a manoeuvre, even nearly finished. That's the fix for "learns the start, never the last mile".
• Warm start is not resume. Loading another task's checkpoint also restores its step counter, which jumps every curriculum to its last stage. The repo has an environment flag for exactly this.

3.6 Publishing a policy

uv run publish --onnx output.onnx --repo <user>/microduck-polite-bow --kind episodic --duration-s 4.0
# then, on a robot
sudo robotctl policy add polite-bow <user>/microduck-polite-bow
robotctl robot do polite-bow

--kind episodic means it runs for a fixed time and returns itself to standing (kicks, a bow). perpetual means it runs until told otherwise (a gait, a held pose).

4 · The robot repo, microduck

4.1 Seven small programs, not one big one

The four rules in docs/design/architecture.md are worth memorising, because they are good design for any system:

1. The recovery path (configd, updaterd, btd) survives a dead robotd.
2. robotd is the authority on safety. Clients send intents ("go this fast"); robotd decides what is executable.

3. The control loop never waits on another service. Everything it reads from elsewhere is a last-value cache.
4. Every piece of state has exactly one owner.

4.2 How programs talk: JSON-RPC over unix sockets

→ {"jsonrpc":"2.0","id":1,"method":"robot.move","params":{"vx":0.15,"vy":0,"vyaw":0}}
← {"jsonrpc":"2.0","id":1,"result":{ ... "requested":[0.15,0,0], "applied":[...], "limited_by":[...] }}

One JSON object per line, one socket per service.

Look at the reply to robot.move: it reports what you asked for, what was applied, and what limited it. That's rule 2 made visible.

4.3 The 50 Hz tick

Each tick, robotd reads all sixteen devices on the bus (15 servos and the IMU board) in one transaction, builds the 61-number observation, runs the ONNX policy, applies safety, and writes the joint targets back. Reading the full bus costs about 8 ms, 40% of the 20 ms budget.

4.4 Updates that can't brick a robot

A release is a whole directory. updaterd checks its signature, points the current symlink at it, restarts the services, and asks robot.health. If the answer isn't healthy, it puts the old release back by itself, and a boot counter catches crash loops that slip past. Your config lives outside the release directory, so it survives both an update and a rollback.

4.5 The simulated duck

scripts/duck-sim runs the real daemons against a MuJoCo body instead of servos. Everything above the bus (the loop, the policy, safety, IPC, robotctl) is the code a robot runs, unchanged. What it can't tell you: anything in a driver.

5 · Build things

Six projects, from "runs on any laptop" to "trains a real policy". Each has an answer or expected result you can check.

Project 1 · Watch a reward get farmed (20 min, any computer)

Save this as reward_farming.py and run python reward_farming.py. It scores three scripted behaviours under two reward designs.

import math

STEPS = 200   # 4 s at 50 Hz

def stand_up(t):      return max(0.0, 90.0 - 90.0 * t / 50)    # rises over 1 s, then stands
def park_halfway(t):  return max(45.0, 90.0 - 90.0 * t / 50)   # rises to 45 degrees and parks
def stay_down(t):     return 90.0                              # never tries

def gated(tilts):     # +1 every step the trunk is "on the way up"
    return sum(1.0 for a in tilts if 20.0 <= a <= 70.0)

def potential(tilts): # delta cos(tilt), like microduck's upright_progress
    cos = [math.cos(math.radians(a)) for a in tilts]
    return sum(b - a for a, b in zip(cos, cos[1:]))

print(f"{'behaviour':<14}{'gated':>8}{'potential':>11}")
for name, f in [("stand_up", stand_up), ("park_halfway", park_halfway), ("stay_down", stay_down)]:
    tilts = [f(t) for t in range(STEPS)]
    print(f"{name:<14}{gated(tilts):>8.0f}{potential(tilts):>11.3f}")

Task: before running it, predict which behaviour each reward ranks first.

behaviour        gated  potential
stand_up            27      1.000
park_halfway       188      0.707
stay_down            0      0.000

Project 2 · Domain randomization by hand (30 min, any computer)

Tune a simple joint controller two ways, then test both on a "real" leg that is heavier and laggier than the model. Save as domain_randomization.py:

import random
DT = 0.02  # 50 Hz

def run(kp, kd, mass, friction, delay, steps=150):
    """Step from 0 to 1 rad. Returns mean absolute tracking error (lower is better)."""
    q, v, err = 0.0, 0.0, 0.0
    queue = [0.0] * (delay + 1)                 # commands arrive `delay` ticks late
    for _ in range(steps):
        queue.append(kp * (1.0 - q) - kd * v)
        torque = queue.pop(0)
        v += (torque - friction * v) / mass * DT
        q += v * DT
        if abs(q) > 10: return 10.0              # unstable: worst score
        err += abs(1.0 - q)
    return err / steps

GAINS = [(kp, kd) for kp in range(2, 61, 2) for kd in (0.2, 0.5, 1.0, 1.5, 2.0, 3.0)]
nominal = min(GAINS, key=lambda g: run(*g, mass=1.0, friction=0.3, delay=0))
random.seed(0)
worlds = [(random.uniform(0.6, 2.5), random.uniform(0.1, 0.8), random.randint(0, 3)) for _ in range(60)]
robust = min(GAINS, key=lambda g: sum(run(*g, *w) for w in worlds) / len(worlds))

real = dict(mass=2.2, friction=0.15, delay=3)
for name, g in [("nominal-tuned", nominal), ("randomized", robust)]:
    print(f"{name:<14} kp={g[0]:>2} kd={g[1]:<4} sim {run(*g, 1.0, 0.3, 0):.3f}  real {run(*g, **real):.3f}")

nominal-tuned  kp=22 kd=3.0  sim 0.153  real 0.434
randomized     kp=12 kd=3.0  sim 0.162  real 0.360

Project 3 · Run the simulated duck (45 min, Linux, macOS or WSL2)

mkdir duck && cd duck
git clone https://github.com/pollen-robotics/microduck
git clone https://github.com/pollen-robotics/microduck_rl     # duck-sim looks for it beside microduck
cd microduck_rl && uv sync && cd ../microduck                 # needs Rust (rustup) and uv installed
scripts/duck-sim               # a MuJoCo window opens and the duck stands up
scripts/duck-sim status        # health, and whether it is standing
scripts/duck-sim drive 0.15 0  # walk forward for 8 s
scripts/duck-sim ctl health    # anything robotctl does
scripts/duck-sim monitor       # joints, IMU, sticks, live
scripts/duck-sim down

On Windows, use WSL2 (Ubuntu); the daemons target Linux. If something fails, scripts/duck-sim log and simlog show each half.

Project 4 · Talk to robotd yourself (30 min, needs Project 3)

With the simulated duck up, its robotd socket is ~/.cache/duck-sim/duck-a.sock. Write a JSON-RPC client in plain Python:

import json, os, socket

sock = socket.socket(socket.AF_UNIX)
sock.connect(os.path.expanduser("~/.cache/duck-sim/duck-a.sock"))
f = sock.makefile("rw")
next_id = 0

def call(method, params=None):
    global next_id
    next_id += 1
    f.write(json.dumps({"jsonrpc": "2.0", "id": next_id, "method": method, "params": params or {}}) + "\n")
    f.flush()
    while True:                                # skip notifications until our reply arrives
        msg = json.loads(f.readline())
        if msg.get("id") == next_id:
            return msg

print(call("robot.health"))
print(call("robot.move", {"vx": 0.15, "vy": 0.0, "vyaw": 0.0}))
print(call("robot.move", {"vx": 5.0,  "vy": 0.0, "vyaw": 0.0}))   # absurd speed: watch "limited_by"

Task: compare requested and applied in the two move replies. Then send {"vx": 0, "vy": 0, "vyaw": 0} to stop.

Project 5 · Train and deploy a real policy (2 h, CUDA GPU or Hugging Face Jobs)

1. cd microduck_rl, then the smoke test from 3.1 (64 envs, 5 iterations). It must finish without NaNs.
2. Train Mjlab-Velocity-Flat-MicroDuck at 4096 envs. Add --hf-jobs if you have no GPU.
3. While it trains, open wandb and check every Episode_Reward/<penalty> is ≤ 0 (the sign rule from 3.4).
4. uv run play … and watch. Write down one thing that looks unnatural.
5. Export with scripts/export.py (never a hand-converted checkpoint), then drive it with scripts/infer_policy.py --walking output.onnx.
6. Load it into your simulated duck: scripts/duck-sim ctl policy load walk <your HF repo> after uv run publish … --kind perpetual --slot walk.

Project 6 · Design a new trick (stretch, a weekend)

Make a "polite bow": from standing, lower the head, hold, and come back. Follow the repo's own workflow:

1. Copy the closest template. A move ending in a pose means the StandUp config.
2. Check in sim that the target pose is a stable equilibrium. Test tilt, not only height.
3. Reward: one fixed target pose from t = 0 (Gaussian on joints and height), an impact penalty, and an upright term. No waypoints, because the policy camps at waypoints.
4. Write a CPU config test that checks joint indices and reward signs. Then smoke test, then train.
5. Publish as --kind episodic --duration-s 4.0.

6 · Pros and cons

7 · What AI can't do here, so you can

The repo even ships AGENTS.md and CLAUDE.md files, because AI coding agents work in it every day. They are good at writing config boilerplate, reward functions, Rust IPC code and docs. These are the parts they still can't do, and where a person who understands this course is worth hiring:

8 · Quiz

Take it the day after reading. Answers are shuffled and of similar length, so you have to know them.

0 / 0 answered Reset quiz

Reference shelf