PPO

year: 2017/08
paper: https://arxiv.org/pdf/1707.06347
website:
code:
connections: RL, TRPO, optimization

---

TLDR

PPO is a policy gradient method that optimizes the policy in a way that is less sensitive to the choice of hyperparameters. It does this by using a clipped objective function that prevents the policy from changing too much at each iteration. This makes PPO more stable and easier to tune than other policy gradient methods.

Motivation

Q-learning (with function approximation) fails on many simple problems and is poorly understood, vanilla policy gradient methods have poor data effiency and robustness; and trust region policy optimization (TRPO) is relatively complicated, and is not compatible with architectures that include noise (such as dropout) or parameter sharing (between the policy and value function, or with auxiliary tasks).

The vanilla policy gradient objective / loss:

$$L^{PG}(\theta )={\hat{\mathbb{E}}}_t[\log {\pi }_{\theta }(a_t\mid s_t){\hat{A}}_t(s_t,a_t)]$$

${\pi }_{\theta }$ … stochastic policy
${\hat{A}}_t$ … estimator of the advantage at time $t$
${\hat{\mathbb{E}}}_t$ … empirical average over a finite batch of samples (in an algorithm that alternates between sampling and optimization)

By differentiating it we get the policy gradient estimator:

$$\hat{g}={\hat{\mathbb{E}}}_t[{\nabla }_{\theta }\log {\pi }_{\theta }(a_{t\mid s_t}){\hat{A}}_t(s_t,a_t)]$$

But using this gradient directly is not a good idea (for reasons explained in TRPO).

The raw objective from TRPO (or originally conservative policy iteration) attempts to fix the issue of destructively large policy updates of the policy:

$$L^{\text{CPI}}(\theta )={\hat{\mathbb{E}}}_t\left[\frac{{\pi }_{\theta }(a_t\mid s_t)}{{\pi }_{{\theta }_{\text{old}}}(a_t\mid s_t)}{\hat{A}}_t\right]={\hat{\mathbb{E}}}_t[r_t(\theta ){\hat{A}}_t]$$

But it still needs some constraint to prevent excessively large policy updates (large advantages; risk of overfitting to current batch; learning instability):

PPO penalizes changes that move $r_t(\theta )$ away from $1$:

$$L^{\text{CLIP}}(\theta )={\hat{\mathbb{E}}}_t[\min (r_t(\theta ){\hat{A}}_t,\text{clip}(r_t(\theta ),1\pm ϵ){\hat{A}}_t)]$$

When $r_t(\theta )$ would make a large positive improvement (beyond $1+ϵ$), we ignore it and use the clipped value instead
When $r_t(\theta )$ would make things worse, we keep that negative signal.
→ We take a lower bound on the original objective $L^{\text{CLIP}}(\theta )\le L^{\text{CPI}}(\theta )$ with a penalty for having too large updates.

The final objective for PPO is usually something like:

$$L_t^{\text{CLIP}+\text{VF}+\text{S}}(\theta )={\hat{\mathbb{E}}}_t[L_t^{\text{CLIP}}(\theta )-c_1{L}_t^{VF}(\theta )+c_{2}S[{\pi }_{\theta }](s_t)]$$

$L^{VF}$ is a value function loss, e.g. mean squared error: $(V_{\theta }(s_t)-V_t^{\text{targ}}{)}^2$
$S$ is an entropy bonus to encourage exploration: $-{\sum }_i{\pi }_{\theta }(a_i|s_t)\cdot \log ({\pi }_{\theta }(a_i|s_t))$
Note: We maximize the above objective. So with standard optimizers, we would minimize the negative of this.

Implementation Details

https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/
https://github.com/vwxyzjn/ppo-implementation-details

Terminology / Flow:

• Collect trajectories using policy (in $N$ episodic or open-ended environments in parallel).
• Trajectory: [(s, a, r, s+1, ...), ...] of length $M$.
• PPO epoch: Policy optimization epoch over minibatches (slices) of a trajectory.
• Iteration: One or more PPO epochs + optimizing policy using PPO on averaged values.
• Repeat.

Don’t reset environment every time you collect experiences; only when it’s actually done. Use a cyclic buffer where the last obs and dones will become the first obs of the next iteration.
→ We collect fixed-length trajectory segments, which allows us to optimize long or even infinite horizon problems.

A shorter trajectory length $M$ means more emphasis on bootstrapping.

Shuffle trajectory to prevent bias from ordering of experiences / highly correlated consecutive timesteps.

Not sure about the blog’s claim on mini-batch level normalization of advantages. For what I can tell everyone normalizes on batch level.

Some evidence that clipping value loss hurts perf.

gradient norm (l2) is clipped to 0.5; some evidence that it boosts perf

References

Very clean jax impl: https://github.com/keraJLi/rejax/blob/main/src/rejax/algos/ppo.py
DPPO (parkour mujoco): https://arxiv.org/pdf/1707.02286
https://pettingzoo.farama.org/tutorials/cleanrl/advanced_PPO/