ITRI Internship - Quadruped Vision Locomotion

A reinforcement-learning locomotion policy for the Unitree Go2 quadruped. From one small forward depth camera (16 × 16 pixels) and its own joint sensors, the robot learns to climb stairs, jump onto and off a 40 cm box, leap gaps up to 0.9 m and cross narrow beams. The goal was to improve the Go2’s real-world performance on tasks like stairs and obstacles, so every skill was trained in Isaac Gym, checked sim-to-sim in Gazebo through the robot’s C++ controller, and then run on the real robot.

Research internship at ITRI (Industrial Technology Research Institute), Taiwan, May to October 2026.

My role: I built the training environment (procedural terrains, reward functions, curriculum and domain randomization for every skill) and the GRU memory module that carries terrain across frames. I also reworked the deployment pipeline: the existing sim-to-sim and real-robot code was written for another model, so I adapted the export, depth preprocessing and controller to this policy.

Tools: PyTorch, Isaac Gym, PPO, NVIDIA Warp ray casting, TorchScript, ROS 2 Humble, Gazebo, rl_sar, Docker, TensorBoard

Videos

Approach

I started from MGDP, a research framework that compresses depth into a compact terrain code, and adapted it to the Go2. The policy combines a small visual token with its own body state. Signals that only exist in simulation shape training and are dropped at deployment.

Figure 1. Training in Isaac Gym (top) and the deployment path (bottom). The exported policy and depth encoder run in the rl_sar C++ controller, first in Gazebo and then on the real Go2.

• Vision: two 16 × 16 depth frames go through a CNN encoder that denoises and compresses them, then a GRU memory that keeps track of the terrain under the body, producing a 64-D visual token.
• Body: 45-D proprioception (joints, IMU, velocity command) goes through a state estimator over the last 4 observations, producing a 7-D body token.
• Action: an actor MLP (116 → 512 → 256 → 128) outputs 12 PD joint targets.
• Training only: a height-map encoder aligned with 32 of the visual latent dimensions through a contrastive (InfoNCE) loss, clean-depth and future-depth reconstruction, and a critic that also sees true velocity, foot heights, terrain heights and the randomized physics. Training runs in three phases: depth encoder, then GRU memory, then policy. The auxiliary visual-memory losses were designed by Shih-Ting Yang.

Training runs 4,096–6,144 robots in parallel, and the final policy comes from one checkpoint lineage of about 566k PPO iterations.

Process: one policy, skill by skill

Every stage adds one family of skills to the same policy without retraining from scratch. The hard part is keeping old skills while learning new ones.

The recovery policy is kept separate on purpose: locomotion training ends an episode on body contact, so the walking policy never learns to get up, and a separate policy keeps flips out of its depth and GRU memory. Control hands back to walking once the robot is upright and still for 0.5 s, and the visual memory is reset.

Sim-to-sim: what a skill depends on

The gap leap scored about 80% in Isaac Gym but fell short in Gazebo. To find out why, I re-ran the trained policy in Isaac with one condition changed at a time. Each panel compares the training condition against Gazebo-like ones.

Figure 2. Gap-crossing success by gap width, from fixed-condition probes of gap-leap checkpoints (485k–502k iterations). The physics-step panel uses a later checkpoint than the other two. Hover a column for exact values.

What I changed. The leap leaned on PhysX’s 5 ms contact step for extra push-off, so I moved training to a 2 ms step at 2.5× the physics cost. I also randomized actuation delay per robot from 0 to 40 ms and widened foot friction to 0.1–1.3. Checkpoints are gated on a 1 ms-physics probe before they leave Isaac.

Not everything I suspected was real. Capping the calf torque at Gazebo’s limit cut the leap to 33–57% in Isaac, but Gazebo’s controller never actually enforced that cap. I checked before shipping the fix.

Figure 3. Air-beam first-try foothold rate across training. Rewards changed after 459k iterations to stop paying for corrected steps.

Lessons: six bugs, with before and after numbers

Most of the work was finding out why a policy that looked good on paper didn’t do what I expected.

The box-jump level rose while nobody cleared it

0% clean clears while the curriculum level went from 7 to 10.2.

• Cause: four “process” rewards (pre-jump lift, forward leap and so on) were cheaper to farm than the skill, speed tracking still paid while ramming, and the curriculum promoted on distance alone.
• Fix: removed the process rewards, kept a clean-crossing reward, ended the episode on ramming, and required a clean crossing with no body contact for promotion.
• Lesson: reward the precise successful state. Anything easier gets farmed.

It was the approach angle, not the speed

13–19% → 79% front-paw mount rate, at ±0.2 rad yaw vs square-on.

• Cause: I had been tuning speed. Broken down by condition, speed helped (68% at 0.2 m/s → 83% at 0.8 m/s) and angle was the limiter. An apparent regression was a change in the yaw mix.
• Fix: a closed-loop alignment command that charges lateral offset and heading error on the approach.
• Lesson: break a metric down by condition before changing anything.

The gap nobody crossed

0–13% → ~80% crossing rate at every width, before and after the redesign.

• Cause: the gap showed a high curriculum level (13.2), but robots walked in and nose-dived. Base contact ended the episode before the fall penalty fired, and distance promotion rewarded the walk-in.
• Fix: redefined the skill as a leap, with one reward per phase (takeoff, flight, far touchdown, stable landing) and promotion only on a confirmed crossing.
• Lesson: write the success test first, then the rewards that lead to it.

Two simulators, two depth images

0.22 → 0.009 worst-row depth error at a sharp edge (normalized depth).

• Cause: a full parity audit showed the camera geometry matched exactly. But up to 35% of a Gazebo frame had no return, read as “near” where training used “far”, which showed up as a phantom wall across the top of the image. Resampling also differed.
• Fix: matched the no-return convention and added training’s noise and dropout to deployment (fixing the convention alone made the gait worse). A two-stage resample removes most of the remaining edge error.
• Lesson: match the whole input distribution, noise included, not just the geometry.

The recovery that only worked in Isaac

100% → 48% recovery success in Isaac once joint-velocity limits were lifted, as in Gazebo.

• Cause: in Gazebo the robot rocked on its back with legs flailing at 10–30 rad/s. Isaac’s physics clamps joint speed; Gazebo’s effort control doesn’t.
• Fix: retrained without the clamp, with a motor torque-speed curve on half the robots and an overspeed penalty.
• Lesson: reproduce the deployment failure inside the training simulator before retraining.

Export the pair, not the policy

Frozen → walking in Gazebo, with a stale vs matched depth encoder.

• Cause: the policy and its depth encoder are trained together, and the encoder keeps changing. The “best” checkpoints of the two came from different iterations.
• Fix: the export always pairs both networks from the same iteration, and a check compares the deployed token against training (cosine similarity 0.9995).
• Lesson: a checkpoint is every network the policy reads from, at one iteration.

Two smaller reward lessons came up along the way. A still robot escapes reward and penalty alike: the rear legs dragged down stairs because speed-scaled terms vanish when the body stops, so leg-lift credit had to be independent of speed. And a minimum can become the target: an anti-stall floor at 60% of the commanded speed became the cruising speed (0.80 → 0.58 of command) until I moved it to 85% on the approach only.

Takeaways

1. Define success strictly and measure it directly, separate from curriculum level and mean reward.
2. Probe fixed conditions (width, speed, angle, delay) before reading a dashboard average.
3. Give each phase of a skill one reward owner, so nothing pays twice.
4. Unit-test reward and termination logic before spending GPU time.
5. Reproduce a deployment failure in training before changing the policy.
6. Change one variable per run, and write down why in a decision log.

Slides

Credits

Built on MGDP (ArcLab, The University of Hong Kong) and legged_gym. The base perception method is theirs, and the visual-memory design apart from the GRU module is by Shih-Ting Yang. The curriculum, terrains, rewards, recovery policy and deployment work on this page are mine. Special thanks to Yueh Lin, Shih-Ting Yang and Isaac Lin.