Reinforcement Learning for Robotics Part 4: Domain Randomization

2026-08-20 | By ShawnHymel

In the last episode, we deployed our trained agent to the real robot and ran into the sim-to-real (or “sim2real”) gap: the differences between simulation and reality that the agent was never trained to handle. We patched things up with firmware fixes like pitch offsets, motor boosts, and low-pass filters. It worked, but it required manual tuning and largely defeated the purpose of training a robust agent in the first place.

In this post, we examine domain randomization (DR). Rather than trying to perfectly model the robot, we deliberately introduce randomness into the simulation during training (noisy sensors, variable motor torque, random pushes, friction variation) so the agent learns to handle a wide range of conditions it might encounter in the real world. The result is a more robust policy that doesn't need firmware workarounds.

All project files are available at github.com/ShawnHymel/reinforcement-learning-for-robotics.

What Is Domain Randomization?

Image of Reinforcement Learning for Robotics Part 4: Domain Randomization

Domain randomization is currently the most popular technique for bridging the sim-to-real gap in robotics. Rather than investing enormous effort in perfectly modeling every physical property of the robot and its environment, DR takes the opposite approach: randomize those properties across a wide range during training and let the agent learn a policy that works across all of them. If the real world falls somewhere within that range, the policy should generalize.

For our balance bot, that means randomizing:

Sensor noise: Gaussian noise added to pitch, pitch rate, and wheel velocity readings, simulating a real IMU and noisy encoder signals
Action delay: randomly delaying the application of motor commands by 0–1 timesteps, simulating the I²C communication lag between the ESP32 and the STM32 motor controller
Motor noise and gain: adding noise to motor commands and randomizing how much torque the motors can actually produce (60–100% of nominal), modeling battery voltage sag and droop
Random pushes: applying random forces to the chassis in the X and Y direction with some probability each step, simulating bumps, nudges, or surface irregularities
Mass variation: randomizing the chassis mass each episode, accounting for our simplified model and non-uniform density
Friction variation: randomizing wheel-ground friction so the agent learns to handle different surfaces (rubber mat, hardwood, carpet, desk)
Axle torque noise: applying small random torques to the wheel joints each step to simulate the tire ridges that MuJoCo can't model directly with convex geometry

Each training episode, a new set of randomized parameters is sampled, and the agent must learn a policy that works across all of them.

Updating the Environment

The updated environment is in workspace/software/ep04/balance_bot_env_dr.py. The structure is the same as Episode 2's environment, with a new DomainRandomConfig dataclass added to hold all the randomization parameters:

Copy Code

@dataclass
class DomainRandomConfig:
    pitch_noise_std_dev: float = 0.0
    pitch_rate_noise_std_dev: float = 0.0
    wheel_vel_noise_std_dev: float = 0.0
    action_delay_steps: int = 0
    action_delay_random: bool = False
    motor_noise_scale: float = 0.0
    push_prob: float = 0.00
    push_force_max_n: float = 0.0
    mass_scale_range: tuple = (1.0, 1.0)
    friction_scale_range: tuple = (1.0, 1.0)
    motor_gain_range: tuple = (1.0, 1.0)
    ridge_prob: float = 0.0
    ridge_torque_max_nm: float = 0.0

The defaults effectively disable DR (passing in the default config gives you the same behavior as the Episode 2 environment). This makes it easy to enable randomization incrementally.

DR is applied in three places inside the environment:

In get_observation(): Gaussian noise is added to pitch, pitch rate, and wheel velocities before the observation vector is returned to the agent.
In reset(): at the start of each episode, new values are sampled for chassis mass, friction, and motor gain. The action delay buffer is also reset here.
In step(): the action delay buffer is updated each step, random pushes are applied to the chassis with the configured probability, and random axle torques are applied to simulate tire ridges.

The reward function stays exactly the same as in Episode 2. DR is an environment-level change: the PPO algorithm itself doesn't need to change at all.

Curriculum Learning with Domain Randomization

Adding domain randomization all at once tends to make training unstable. The agent has a hard time learning the basic balancing task if it's simultaneously dealing with noisy sensors, random pushes, and variable motor torque from the start. The solution is to continue using curriculum learning, but extend it to seven phases.

Phases 1 and 2 are identical to Episode 2: balance only, then balance with position and yaw penalties. The agent builds the core skill before any DR is introduced.

From Phase 3 onward, domain randomization is added gradually:

Phase 3: sensor noise (pitch, pitch rate, wheel velocity) and action delay
Phase 4: motor noise and gentle random pushes (0.5% probability per step)
Phase 5: mass and friction variation
Phase 6: motor gain randomization (60–100% of nominal torque)
Phase 7: axle torque noise to simulate tire ridges (5% probability per step)

The key principle here is to make sure the agent is performing well at each phase before introducing the next level of difficulty. If the robot isn't balancing reliably by the end of Phase 2, adding DR in Phase 3 won't fix it, as it'll just make training harder. Watch TensorBoard at the end of each phase and only move on when episodic returns are consistently high.

You can try combining some of these phases if you want to reduce total training time. It may work fine, but be prepared to split them back out if the agent struggles to converge.

Training

Open workspace/software/ep04/train_with_ppo_dr.ipynb in JupyterLab. The setup is similar to Episode 2, with a few important changes:

More environments and steps: Domain randomization requires more training data to converge. We're using 8 parallel environments and 500,000 steps per environment per phase. Note that the total training time is now a few hours.
Larger networks: 16 nodes per hidden layer wasn't enough capacity for the agent to learn across all the DR variations. Bumping up to 32 nodes per hidden layer (still well within the ESP32's 5ms budget) helped the actor and critic converge reliably. You can go up to 64 nodes without a meaningful performance hit on the hardware.
A load_agent helper: A new utility function lets you resume training from a saved checkpoint if a phase fails or something breaks mid-run. Useful for debugging without restarting from scratch.

Once all seven phases complete, run the evaluation cell to watch the agent in real time. With DR, you should see the robot handling those random pushes and recovering cleanly. The environment will nudge it during evaluation just as it did during training.

Check TensorBoard at localhost:6006 to review training across all phases. What to look for: episodic returns may dip at the start of each new phase as the agent encounters harder conditions, but should climb back toward the maximum by the end. Some forgetting is normal; just ensure that the best model from each phase is still solid.

Deploying to the Robot

Once training is complete, the final cell should export the actor as an ONNX file as well as convert it to C code using the scripts we saw in episode 3.

Copy actor.h into workspace/software/04-domain-randomization/balance_bot_fixes/ and open the sketch in Arduino IDE. This is the same firmware as episode 3's balance_bot_fixes sketch, but with all the post-processing disabled (e.g., no pitch offset, no motor boost, no deadband, no low-pass filter). The DR-trained agent should handle those conditions on its own.

The only meaningful change to the firmware is the larger actor network: 32 nodes per hidden layer instead of 16. The actor_forward() call is identical. Upload and test.

The difference should be noticeable. The robot balances more robustly without any firmware tuning, and when nudged, it recovers cleanly. It's not perfect (no sim-to-real transfer ever is), but it's a significant improvement over the firmware-patched version from episode 3, and it required zero manual tuning.

Image of Reinforcement Learning for Robotics Part 4: Domain Randomization

What's Next

We now have a balance bot that learned to handle real-world conditions through simulation alone. In the next episode, we'll extend the agent further by adding commands to the observation vector: teaching the robot to respond to directional inputs so it can be driven around like a remote-controlled vehicle. That will require another round of curriculum learning as the agent learns to balance and follow commands simultaneously.