The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games

year: 2021
paper: https://arxiv.org/pdf/2103.01955
website: https://sites.google.com/view/mappo
code: https://github.com/marlbenchmark/on-policy
connections: PPO, MARL

---

MAPPO objective

The objective of MAPPO is equivalent to that of PPO, but equipping each agent with one shared set of parameters, i.e. ${\theta }^i={\theta }^j,\forall i,j\in \mathcal{N}=\{1,\dots n\}$. It uses the agent’s combined trajectories for the shared policy’s update.
So the objective for optimizing the policy parameters ${\theta }_{k+1}$ at iteration $k+1$ is:

$$\sum _{i=1}^n{\mathbb{E}}_{\mathbf{o}\sim {\pi }_{{\theta }_k},\mathbf{a}\sim {\pi }_{{\theta }_k}}\left[\min \left(\frac{{\pi }_{\theta }({\mathbf{a}}^i|\mathbf{o})}{{\pi }_{{\theta }_k}({\mathbf{a}}^i|\mathbf{o})}{A}_{{\pi }_{{\theta }_k}}(\mathbf{o},\mathbf{a}),\text{clip}\left(\frac{{\pi }_{\theta }({\mathbf{a}}^i|\mathbf{o})}{{\pi }_{{\theta }_k}({\mathbf{a}}^i|\mathbf{o})},1\pm ϵ\right)A_{{\pi }_{{\theta }_k}}(\mathbf{o},\mathbf{a})\right)\right]$$

The constraint on the joint policy space imposed by shared parameters can lead to an exponentially-worse sub-optimal outcome (details; page 4).