policy gradient theorem

Computing the gradient ${\nabla }_{\theta }J(\theta )$ is tricky because it depends on both the action selection (directly determined by ${\pi }_{\theta }$) and the stationary distribution of states $d^{\pi }$ following the target selection behavior (indirectly determined by ). Given that the environment is generally unknown (not differentiable), it is difficult to estimate the effect on the state distribution by a policy update.

Policy gradient theorem to the rescue:

Policy Gradient Theorem

$$\begin{array}{rl}{\nabla }_{\theta }J(\theta )& ={\nabla }_{\theta }\sum _{s\in \mathcal{S}}{d}^{\pi }(s)\sum _{a\in \mathcal{A}}{Q}^{\pi }(s,a){\pi }_{\theta }(a\mid s)\\ & \propto \sum _{s\in \mathcal{S}}{d}^{\pi }(s)\sum _{a\in \mathcal{A}}{Q}^{\pi }(s,a){\nabla }_{\theta }{\pi }_{\theta }(a\mid s)\end{array}$$

https://lilianweng.github.io/posts/2018-02-19-rl-overview/#policy-gradient

https://lilianweng.github.io/posts/2018-04-08-policy-gradient/

References

https://lilianweng.github.io/posts/2018-04-08-policy-gradient/